乐趣区

关于java:免费分享基于Python的新闻爬虫订阅展示系统设计与实现

jiezi

​

项目名称
（收费分享）基于 Python 的新闻爬虫订阅展现零碎设计与实现

零碎阐明
摘  要

随着互联网的迅速倒退，互联网大大晋升了信息的产生和传播速度，网络上每天都会产生大量的内容，如何高效地  从这些横七竖八的内容中发现并采集所需的信息显得越来越重要。网络中的新闻内容也一样，新闻散布在不同的网  站上，而且存在反复的内容，咱们往往只关怀其中的一部分新闻，网络中的新闻页面往往还充斥着大量许多与新闻  不相干的信息，影响了咱们的浏览效率和浏览体验，如何更加不便及时并高效地获取咱们所关怀的新闻内容，本系  统可能帮咱们做到这一点。本零碎利用网络爬虫咱们能够做到对网络上的新闻网站进行定时定向的剖析和采集，然  后把采集到的数据进行去重，分类等操作后存入数据库，最初提供个性化的新闻订阅服务。思考了如何应答网站的  反爬虫策略，防止被网站封闭爬虫。在具体实现上会应用 Python 配合 scrapy 等框架来编写爬虫，采纳特定的内容抽取算法来提取指标数据，最初应用 Django 加上 weui 来提供新闻订阅后盾和新闻内容展现页，应用微信向用户推    送信息。用户能够通过本零碎订阅指定关键字，当爬虫零碎爬取到了含有指定关键字的内容时会把新闻推送给用   户。

[关键词]网络爬虫；新闻；个性化；订阅；Python

Abstract
With the rapid development of the Internet, the Internet has greatly enhanced the production and dissemination of information, the networkwill produce a lot of content every day, how to find and collectthe information we needed from these disorganized contentefficiently is more and more important. The news content on the network is thesame, the news is distributed on different sites, and there are many duplicate content, we only care about part of thenews usually. The network news pages areoften filled with a lot of news and information is not related that impact ourreading efficiency and readingexperience. How to more convenient and efficient access to the news we are concerned about the content, thissystem can help us to do this. This system uses the web crawler to collect news on the network site. And then toclassify data and other operations like delete the duplication,store data byuse the database, and finally providepersonalized news subscription service. This system has considered how to deal with the sit&s anti-reptile strategy, toavoid being blocked by the site crawler. In the concrete implementation, I will use Python with Scrapy framework towrite the crawler, then use a specificcontent extraction algorithm to extract the target data, and finally use Django and WeUI to provide news subscription background and news contentdisplay page, use WeChat to push information to users. Users can subscribe tothe specified keywords through the    system, system will push thenews to the user when the crawler system crawled the contents contains thespecified keyword.

[Keywords] Web Crawler;News; Personalization; Subscription; Python

运行截图

 用户管理控制层:
package com.houserss.controller;

import javax.servlet.http.HttpSession;

import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

import com.houserss.common.Const;
import com.houserss.common.Const.Role;
import com.houserss.common.ServerResponse;
import com.houserss.pojo.User;
import com.houserss.service.IUserService;
import com.houserss.service.impl.UserServiceImpl;
import com.houserss.util.MD5Util;
import com.houserss.util.TimeUtils;
import com.houserss.vo.DeleteHouseVo;
import com.houserss.vo.PageInfoVo;

/**

  • Created by admin
    */

@Controller
@RequestMapping(“/user/”)
public class UserController {

@Autowired
private IUserService iUserService;

/**
 * 用户登录
 * @param username
 * @param password
 * @param session
 * @return
 */
@RequestMapping(value = "login.do",method = RequestMethod.POST)
@ResponseBody
public ServerResponse<User> login(User user,String uvcode, HttpSession session){String code = (String)session.getAttribute("validationCode");
    if(StringUtils.isNotBlank(code)) {if(!code.equalsIgnoreCase(uvcode)) {return ServerResponse.createByErrorMessage("验证码不正确");
        }
    }
    ServerResponse<User> response = iUserService.login(user.getUsername(),user.getPassword());
    if(response.isSuccess()){session.setAttribute(Const.CURRENT_USER,response.getData());
    }
    return response;
}

}
管理员管理控制层:

package com.sxl.controller.admin;

import java.util.List;
import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;

import com.sxl.controller.MyController;

@Controller(“adminController”)
@RequestMapping(value = “/admin”)
public class AdminController extends MyController {



@RequestMapping(value = "/index")
public String frame(Model model, HttpServletRequest request)throws Exception {return "/admin/index";}

@RequestMapping(value = "/main")
public String main(Model model, HttpServletRequest request)throws Exception {return "/admin/main";}

@RequestMapping(value = "/tj1")
public String tj1(Model model, HttpServletRequest request)throws Exception {String sql="select DATE_FORMAT(insertDate,'%Y-%m-%d') dates,sum(allPrice) price from t_order order by DATE_FORMAT(insertDate,'%Y-%m-%d')  desc";
    List<Map> list = db.queryForList(sql);
    model.addAttribute("list", list);
    System.out.println(list);
    return "/admin/tj/tj1";
}


@RequestMapping(value = "/password")
public String password(Model model, HttpServletRequest request)throws Exception {return "/admin/password";}


@RequestMapping(value = "/changePassword")
public ResponseEntity<String> loginSave(Model model,HttpServletRequest request,String oldPassword,String newPassword) throws Exception {Map admin = getAdmin(request);
    if(oldPassword.equals(admin.get("password").toString())){
        String sql="update t_admin set password=? where id=?";
        db.update(sql, new Object[]{newPassword,admin.get("id")});
        return renderData(true,"1",null);
    }else{return renderData(false,"1",null);
    }
}

}
批改明码业务逻辑:

package com.sxl.controller.admin;

import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;

import com.sxl.controller.MyController;

@Controller(“userController”)
@RequestMapping(value = “/user”)
public class UserController extends MyController {



@RequestMapping(value = "/index")
public String frame(Model model, HttpServletRequest request)throws Exception {return "/user/index";}

@RequestMapping(value = "/main")
public String main(Model model, HttpServletRequest request)throws Exception {return "/user/main";}


@RequestMapping(value = "/password")
public String password(Model model, HttpServletRequest request)throws Exception {return "/user/password";}


@RequestMapping(value = "/changePassword")
public ResponseEntity<String> loginSave(Model model,HttpServletRequest request,String oldPassword,String newPassword) throws Exception {Map user = getUser(request);
    if(oldPassword.equals(user.get("password").toString())){
        String sql="update t_user set password=? where id=?";
        db.update(sql, new Object[]{newPassword,user.get("id")});
        return renderData(true,"1",null);
    }else{return renderData(false,"1",null);
    }
}
@RequestMapping(value = "/mine")
public String mine(Model model, HttpServletRequest request)throws Exception {

Map user =getUser(request);Map map = db.queryForMap(“select * from t_user where id=?”,new Object[]{user.get(“id”)});model.addAttribute(“map”, map); return “/user/mine”;

}



@RequestMapping(value = "/mineSave")
public ResponseEntity<String> mineSave(Model model,HttpServletRequest request,Long id
    ,String username,String password,String name,String gh,String mobile) throws Exception{
    int result = 0;
        String sql="update t_user set name=?,gh=?,mobile=? where id=?";
        result = db.update(sql, new Object[]{name,gh,mobile,id});
    if(result==1){return renderData(true,"操作胜利",null);
    }else{return renderData(false,"操作失败",null);
    }
}
}

通用治理模块:
package com.sxl.controller;

import java.nio.charset.Charset;
import java.util.Locale;
import java.util.ResourceBundle;

import javax.servlet.http.HttpServletRequest;

import org.apache.commons.lang.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;

import com.sxl.util.JacksonJsonUtil;
import com.sxl.util.StringUtil;
import com.sxl.util.SystemProperties;

public class BaseController {

public static final Long EXPIRES_IN = 1000 * 3600 * 24 * 1L;// 1 天

@Autowired
private SystemProperties systemProperties;

/**
 * 取得配置文件内容
 */
public String getConfig(String key) {return systemProperties.getProperties(key);
}

/**
 * 返回服务器地址 like http://192.168.1.1:8441/UUBean/
 */
public String getHostUrl(HttpServletRequest request) {String hostName = request.getServerName();
    Integer hostPort = request.getServerPort();
    String path = request.getContextPath();

    if (hostPort == 80) {return "http://" + hostName + path + "/";} else {return "http://" + hostName + ":" + hostPort + path + "/";}
}

/***
 * 获取以后的 website 门路 String
 */
public static String getWebSite(HttpServletRequest request) {String returnUrl = request.getScheme() + "://"
            + request.getServerName();

    if (request.getServerPort() != 80) {returnUrl += ":" + request.getServerPort();
    }

    returnUrl += request.getContextPath();

    return returnUrl;
}



/**
 * 初始化 HTTP 头.
 * 
 * @return HttpHeaders
 */
public HttpHeaders initHttpHeaders() {HttpHeaders headers = new HttpHeaders();
    MediaType mediaType = new MediaType("text", "html",
            Charset.forName("utf-8"));
    headers.setContentType(mediaType);
    return headers;
}

/**
 * 返回 信息数据
 * 
 * @param status
 * @param msg
 * @return
 */
public ResponseEntity<String> renderMsg(Boolean status, String msg) {if (StringUtils.isEmpty(msg)) {msg = "";}
    String str = "{\"status\":\"" + status + "\",\"msg\":\""+ msg +"\"}";
    ResponseEntity<String> responseEntity = new ResponseEntity<String>(str,
            initHttpHeaders(), HttpStatus.OK);
    return responseEntity;
}

/**
 * 返回 obj 数据
 * 
 * @param status
 * @param msg
 * @param obj
 * @return
 */
public ResponseEntity<String> renderData(Boolean status, String msg,
        Object obj) {if (StringUtils.isEmpty(msg)) {msg = "";}
    StringBuffer sb = new StringBuffer();
    sb.append("{");
    sb.append("\"status\":\"" + status + "\",\"msg\":\""+ msg +"\",");
    sb.append("\"data\":" + JacksonJsonUtil.toJson(obj) + "");
    sb.append("}");

    ResponseEntity<String> responseEntity = new ResponseEntity<String>(sb.toString(), initHttpHeaders(), HttpStatus.OK);
    return responseEntity;
}


/***
 * 获取 IP（如果是多级代理，则失去的是一串 IP 值）*/
public static String getIpAddr(HttpServletRequest request) {String ip = request.getHeader("x-forwarded-for");
    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {ip = request.getHeader("Proxy-Client-IP");
    }

    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {ip = request.getHeader("WL-Proxy-Client-IP");
    }

    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {ip = request.getRemoteAddr();
    }

    if (ip != null && ip.length() > 0) {String[] ips = ip.split(",");
        for (int i = 0; i < ips.length; i++) {if (!"unknown".equalsIgnoreCase(ips[i])) {ip = ips[i];
                break;
            }
        }
    }

    return ip;
}

/**
 * 国际化取得语言内容
 * 
 * @param key
 *            语言 key
 * @param args
 * @param argsSplit
 * @param defaultMessage
 * @param locale
 * @return
 */
public static String getLanguage(String key, String args, String argsSplit,
        String defaultMessage, String locale) {
    String language = "zh";
    String contry = "cn";
    String returnValue = defaultMessage;

    if (!StringUtil.isEmpty(locale)) {
        try {String[] localeArray = locale.split("_");
            language = localeArray[0];
            contry = localeArray[1];
        } catch (Exception e) {}}

    try {
        ResourceBundle resource = ResourceBundle.getBundle("lang.resource",
                new Locale(language, contry));
        returnValue = resource.getString(key);
        if (!StringUtil.isEmpty(args)) {String[] argsArray = args.split(argsSplit);
            for (int i = 0; i < argsArray.length; i++) {returnValue = returnValue.replace("{" + i + "}",
                        argsArray[i]);
            }
        }
    } catch (Exception e) { }

    return returnValue;
}

}

​

退出移动版