使用scrapy抓取Youtube播放页数据

可参看 Knowsmore
抓取 Youtube 播放页数据的前提是 scrapy 部署的机器可以正常访问 Youtube 网站
示例网址
抓取的原理是读取 Youtube 播放页桌面版网页源代码中的全局变量：ytInitialData
存取到 Mongo 中的数据如下：
{
“title” : “20130410 锵锵三人行 陈平原谈中国教育问题 ”,
“view_count” : “12,407 views”
}

代码如下：
# -*- coding: utf-8 -*-
import scrapy
import re
import json
from scrapy import Selector
from knowsmore.items import YoutubeItem
from ..common import *

class YoutubeSpider(scrapy.Spider):
custom_settings = {
‘DOWNLOADER_MIDDLEWARES’ : {
}
}

name = ‘youtube’
allowed_domains = [‘www.youtube.com’]
start_urls = [‘https://www.youtube.com/watch?v=3vkqOdMBP48′]

def parse(self, response):
ytInitialData = r1(r’ytInitialData”\] = (.*?)}};’, response.body)

if ytInitialData:
ytInitialData = ‘%s}}’ % ytInitialData
ytInitialDataObj = json.loads(ytInitialData)

videoInfo = ytInitialDataObj[‘contents’][‘twoColumnWatchNextResults’][‘results’][‘results’][‘contents’][0][‘videoPrimaryInfoRenderer’]
Item = YoutubeItem(
title = videoInfo[‘title’][‘simpleText’].encode(‘utf-8’),
view_count = videoInfo[‘viewCount’][‘videoViewCountRenderer’][‘viewCount’][‘simpleText’]
)
yield Item