共计 978 个字符，预计需要花费 3 分钟才能阅读完成。
可参看 Knowsmore
抓取 Youtube 播放页数据的前提是 scrapy 部署的机器可以正常访问 Youtube 网站
示例网址
抓取的原理是读取 Youtube 播放页桌面版网页源代码中的全局变量：ytInitialData
存取到 Mongo 中的数据如下：
{
“title” : “20130410 锵锵三人行 陈平原谈中国教育问题 ”,
“view_count” : “12,407 views”
}
代码如下：
# -*- coding: utf-8 -*-
import scrapy
import re
import json
from scrapy import Selector
from knowsmore.items import YoutubeItem
from ..common import *
class YoutubeSpider(scrapy.Spider):
custom_settings = {
‘DOWNLOADER_MIDDLEWARES’ : {
}
}
name = ‘youtube’
allowed_domains = [‘www.youtube.com’]
start_urls = [‘https://www.youtube.com/watch?v=3vkqOdMBP48′]
def parse(self, response):
ytInitialData = r1(r’ytInitialData”\] = (.*?)}};’, response.body)
if ytInitialData:
ytInitialData = ‘%s}}’ % ytInitialData
ytInitialDataObj = json.loads(ytInitialData)
videoInfo = ytInitialDataObj[‘contents’][‘twoColumnWatchNextResults’][‘results’][‘results’][‘contents’][0][‘videoPrimaryInfoRenderer’]
Item = YoutubeItem(
title = videoInfo[‘title’][‘simpleText’].encode(‘utf-8’),
view_count = videoInfo[‘viewCount’][‘videoViewCountRenderer’][‘viewCount’][‘simpleText’]
)
yield Item
