关于后端:大数据专题spark-mysql-python爬虫携程景点爬取含虚拟机镜像

44次阅读

共计 9995 个字符，预计需要花费 25 分钟才能阅读完成。

​
  博主介绍：✌退职 Java 研发工程师、专一于程序设计、源码分享、技术交换、专一于 Java 技术畛域✌

项目名称

大数据专题 -spark mysql python 爬虫携程景点爬取（含虚拟机镜像）

视频成果

大数据专题 -spark mysql python 爬虫携程景点爬取（含虚拟机镜像）_哔哩哔哩_bilibili 

零碎阐明

一．问题定义解决

1.1 须要解决的问题

本次大作业报告次要解决的问题：

 应用 spark 读取 csv 文件数据并剖析数据。应用 spark 将 csv 文件数据导入到 mysql 中。应用 sql 查找数据库数据。构建 echarts 框架的图表页面。应用 spark 将剖析的 json 数据输入。应用 eharts 前端页面剖析 json 数据进行展现。

1.2 具体所用办法及输入输出。

 应用 spark 读取 csv 文件数据并剖析数据。

首先构建基于 spark 框架的环境进行开发。本次报告环境是 win10 零碎，所以须要下载对应的环境。

（1）Windows10 本地搭建单机版 Spark 开发环境

1、零碎环境

OS：Windows10_x64 专业版

2、所需软件或工具 

8.0_131
spark-2.3.0-bin-hadoop2.7.tgz
hadoop-2.8.3.tar.gz
scala-2.11.8.zip
hadoop-common-2.2.0-bin-master.zip（次要应用外面的 exe）IntelliJ IDEA（版本：1.2 Build #IU-171.4249.32,built on April 21,2017）scala-intellij-bin-2017.1.20.zip（IntelliJ IDEA scala 插件）apache-maven-3.5.0

环境须要

1. 运行环境：最好是 java jdk 1.8，咱们在这个平台上运行的。其余版本实践上也能够。
2.IDE 环境：IDEA，Eclipse,Myeclipse 都能够。举荐 IDEA;
3.tomcat 环境：Tomcat 7.x,8.x,9.x 版本均可
4. 硬件环境：windows 7/8/10 1G 内存以上；或者 Mac OS；
5. 数据库：MySql 5.7 版本；
6. 是否 Maven 我的项目：否；

技术栈

  1. 后端：Spring+SpringMVC+Mybatis
  2. 前端：JSP+CSS+JavaScript+jQuery

应用阐明

  1. 应用 Navicat 或者其它工具，在 mysql 中创立对应名称的数据库，并导入我的项目的 sql 文件；
  2. 应用 IDEA/Eclipse/MyEclipse 导入我的项目，Eclipse/MyEclipse 导入时，若为 maven 我的项目请抉择 maven;
    若为 maven 我的项目，导入胜利后请执行 maven clean;maven install 命令，而后运行；
  3. 将我的项目中 springmvc-servlet.xml 配置文件中的数据库配置改为本人的配置;
  4. 运行我的项目，在浏览器中输出 http://localhost:8080/ 登录

运行截图

​编辑

​编辑

​编辑

​编辑

​编辑

​编辑

​编辑

​编辑

​编辑

 用户管理控制层:

package com.houserss.controller;

import javax.servlet.http.HttpSession;

import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

import com.houserss.common.Const;
import com.houserss.common.Const.Role;
import com.houserss.common.ServerResponse;
import com.houserss.pojo.User;
import com.houserss.service.IUserService;
import com.houserss.service.impl.UserServiceImpl;
import com.houserss.util.MD5Util;
import com.houserss.util.TimeUtils;
import com.houserss.vo.DeleteHouseVo;
import com.houserss.vo.PageInfoVo;

/**

  • Created by admin
    */

@Controller
@RequestMapping(“/user/”)
public class UserController {

@Autowired
private IUserService iUserService;

/**
 * 用户登录
 * @param username
 * @param password
 * @param session
 * @return
 */
@RequestMapping(value = "login.do",method = RequestMethod.POST)
@ResponseBody
public ServerResponse<User> login(User user,String uvcode, HttpSession session){String code = (String)session.getAttribute("validationCode");
    if(StringUtils.isNotBlank(code)) {if(!code.equalsIgnoreCase(uvcode)) {return ServerResponse.createByErrorMessage("验证码不正确");
        }
    }
    ServerResponse<User> response = iUserService.login(user.getUsername(),user.getPassword());
    if(response.isSuccess()){session.setAttribute(Const.CURRENT_USER,response.getData());
    }
    return response;
}

}

管理员管理控制层:

package com.sxl.controller.admin;

import java.util.List;
import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;

import com.sxl.controller.MyController;

@Controller(“adminController”)
@RequestMapping(value = “/admin”)
public class AdminController extends MyController {



@RequestMapping(value = "/index")
public String frame(Model model, HttpServletRequest request)throws Exception {return "/admin/index";}

@RequestMapping(value = "/main")
public String main(Model model, HttpServletRequest request)throws Exception {return "/admin/main";}

@RequestMapping(value = "/tj1")
public String tj1(Model model, HttpServletRequest request)throws Exception {String sql="select DATE_FORMAT(insertDate,'%Y-%m-%d') dates,sum(allPrice) price from t_order order by DATE_FORMAT(insertDate,'%Y-%m-%d')  desc";
    List<Map> list = db.queryForList(sql);
    model.addAttribute("list", list);
    System.out.println(list);
    return "/admin/tj/tj1";
}


@RequestMapping(value = "/password")
public String password(Model model, HttpServletRequest request)throws Exception {return "/admin/password";}


@RequestMapping(value = "/changePassword")
public ResponseEntity<String> loginSave(Model model,HttpServletRequest request,String oldPassword,String newPassword) throws Exception {Map admin = getAdmin(request);
    if(oldPassword.equals(admin.get("password").toString())){
        String sql="update t_admin set password=? where id=?";
        db.update(sql, new Object[]{newPassword,admin.get("id")});
        return renderData(true,"1",null);
    }else{return renderData(false,"1",null);
    }
}

}

批改明码业务逻辑:

package com.sxl.controller.admin;

import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;

import com.sxl.controller.MyController;

@Controller(“userController”)
@RequestMapping(value = “/user”)
public class UserController extends MyController {



@RequestMapping(value = "/index")
public String frame(Model model, HttpServletRequest request)throws Exception {return "/user/index";}

@RequestMapping(value = "/main")
public String main(Model model, HttpServletRequest request)throws Exception {return "/user/main";}


@RequestMapping(value = "/password")
public String password(Model model, HttpServletRequest request)throws Exception {return "/user/password";}


@RequestMapping(value = "/changePassword")
public ResponseEntity<String> loginSave(Model model,HttpServletRequest request,String oldPassword,String newPassword) throws Exception {Map user = getUser(request);
    if(oldPassword.equals(user.get("password").toString())){
        String sql="update t_user set password=? where id=?";
        db.update(sql, new Object[]{newPassword,user.get("id")});
        return renderData(true,"1",null);
    }else{return renderData(false,"1",null);
    }
}
@RequestMapping(value = "/mine")
public String mine(Model model, HttpServletRequest request)throws Exception {

Map user =getUser(request);Map map = db.queryForMap(“select * from t_user where id=?”,new Object[]{user.get(“id”)});model.addAttribute(“map”, map); return “/user/mine”;

}



@RequestMapping(value = "/mineSave")
public ResponseEntity<String> mineSave(Model model,HttpServletRequest request,Long id
    ,String username,String password,String name,String gh,String mobile) throws Exception{
    int result = 0;
        String sql="update t_user set name=?,gh=?,mobile=? where id=?";
        result = db.update(sql, new Object[]{name,gh,mobile,id});
    if(result==1){return renderData(true,"操作胜利",null);
    }else{return renderData(false,"操作失败",null);
    }
}
}

通用治理模块:

package com.sxl.controller;

import java.nio.charset.Charset;
import java.util.Locale;
import java.util.ResourceBundle;

import javax.servlet.http.HttpServletRequest;

import org.apache.commons.lang.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;

import com.sxl.util.JacksonJsonUtil;
import com.sxl.util.StringUtil;
import com.sxl.util.SystemProperties;

public class BaseController {

public static final Long EXPIRES_IN = 1000 * 3600 * 24 * 1L;// 1 天

@Autowired
private SystemProperties systemProperties;

/**
 * 取得配置文件内容
 */
public String getConfig(String key) {return systemProperties.getProperties(key);
}

/**
 * 返回服务器地址 like http://192.168.1.1:8441/UUBean/
 */
public String getHostUrl(HttpServletRequest request) {String hostName = request.getServerName();
    Integer hostPort = request.getServerPort();
    String path = request.getContextPath();

    if (hostPort == 80) {return "http://" + hostName + path + "/";} else {return "http://" + hostName + ":" + hostPort + path + "/";}
}

/***
 * 获取以后的 website 门路 String
 */
public static String getWebSite(HttpServletRequest request) {String returnUrl = request.getScheme() + "://"
            + request.getServerName();

    if (request.getServerPort() != 80) {returnUrl += ":" + request.getServerPort();
    }

    returnUrl += request.getContextPath();

    return returnUrl;
}



/**
 * 初始化 HTTP 头.
 * 
 * @return HttpHeaders
 */
public HttpHeaders initHttpHeaders() {HttpHeaders headers = new HttpHeaders();
    MediaType mediaType = new MediaType("text", "html",
            Charset.forName("utf-8"));
    headers.setContentType(mediaType);
    return headers;
}

/**
 * 返回 信息数据
 * 
 * @param status
 * @param msg
 * @return
 */
public ResponseEntity<String> renderMsg(Boolean status, String msg) {if (StringUtils.isEmpty(msg)) {msg = "";}
    String str = "{\"status\":\"" + status + "\",\"msg\":\""+ msg +"\"}";
    ResponseEntity<String> responseEntity = new ResponseEntity<String>(str,
            initHttpHeaders(), HttpStatus.OK);
    return responseEntity;
}

/**
 * 返回 obj 数据
 * 
 * @param status
 * @param msg
 * @param obj
 * @return
 */
public ResponseEntity<String> renderData(Boolean status, String msg,
        Object obj) {if (StringUtils.isEmpty(msg)) {msg = "";}
    StringBuffer sb = new StringBuffer();
    sb.append("{");
    sb.append("\"status\":\"" + status + "\",\"msg\":\""+ msg +"\",");
    sb.append("\"data\":" + JacksonJsonUtil.toJson(obj) + "");
    sb.append("}");

    ResponseEntity<String> responseEntity = new ResponseEntity<String>(sb.toString(), initHttpHeaders(), HttpStatus.OK);
    return responseEntity;
}


/***
 * 获取 IP（如果是多级代理，则失去的是一串 IP 值）*/
public static String getIpAddr(HttpServletRequest request) {String ip = request.getHeader("x-forwarded-for");
    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {ip = request.getHeader("Proxy-Client-IP");
    }

    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {ip = request.getHeader("WL-Proxy-Client-IP");
    }

    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {ip = request.getRemoteAddr();
    }

    if (ip != null && ip.length() > 0) {String[] ips = ip.split(",");
        for (int i = 0; i < ips.length; i++) {if (!"unknown".equalsIgnoreCase(ips[i])) {ip = ips[i];
                break;
            }
        }
    }

    return ip;
}

/**
 * 国际化取得语言内容
 * 
 * @param key
 *            语言 key
 * @param args
 * @param argsSplit
 * @param defaultMessage
 * @param locale
 * @return
 */
public static String getLanguage(String key, String args, String argsSplit,
        String defaultMessage, String locale) {
    String language = "zh";
    String contry = "cn";
    String returnValue = defaultMessage;

    if (!StringUtil.isEmpty(locale)) {
        try {String[] localeArray = locale.split("_");
            language = localeArray[0];
            contry = localeArray[1];
        } catch (Exception e) {}}

    try {
        ResourceBundle resource = ResourceBundle.getBundle("lang.resource",
                new Locale(language, contry));
        returnValue = resource.getString(key);
        if (!StringUtil.isEmpty(args)) {String[] argsArray = args.split(argsSplit);
            for (int i = 0; i < argsArray.length; i++) {returnValue = returnValue.replace("{" + i + "}",
                        argsArray[i]);
            }
        }
    } catch (Exception e) { }

    return returnValue;
}

}

​

正文完
后端
发表至： 后端
2022-10-10
 0
关于后端:elixir-0070-不死的网络细数-Beam-集群和-Bitcoin-的类似之处
关于后端:面试题精讲continuebreak和return的区别是什么
关于后端:程序员必备的linux常用的26条命令
关于后端:MongoDB-分片集群