使用scrapy抓取Youtube播放页数据

可参看Knowsmore抓取Youtube播放页数据的前提是scrapy部署的机器可以正常访问Youtube网站示例网址抓取的原理是读取Youtube播放页桌面版网页源代码中的全局变量： ytInitialData存取到Mongo中的数据如下：{ “title” : “20130410 锵锵三人行 陈平原谈中国教育问题”, “view_count” : “12,407 views”}代码如下：# -- coding: utf-8 --import scrapyimport reimport jsonfrom scrapy import Selectorfrom knowsmore.items import YoutubeItemfrom ..common import class YoutubeSpider(scrapy.Spider): custom_settings = { ‘DOWNLOADER_MIDDLEWARES’ : { } } name = ‘youtube’ allowed_domains = [‘www.youtube.com’] start_urls = [‘https://www.youtube.com/watch?v=3vkqOdMBP48’] def parse(self, response): ytInitialData = r1(r’ytInitialData"] = (.?)}};’, response.body) if ytInitialData: ytInitialData = ‘%s}}’ % ytInitialData ytInitialDataObj = json.loads(ytInitialData) videoInfo = ytInitialDataObj[‘contents’][’twoColumnWatchNextResults’][‘results’][‘results’][‘contents’][0][‘videoPrimaryInfoRenderer’] Item = YoutubeItem( title = videoInfo[’title’][‘simpleText’].encode(‘utf-8’), view_count = videoInfo[‘viewCount’][‘videoViewCountRenderer’][‘viewCount’][‘simpleText’] ) yield Item