前言
最近,我应用爬虫技术,爬取了美国航空航天局,也就是你电影里常常见到的 NASA
, 火星摸索的相干图片,有 10000
张吧。
嗯嗯,小事件,小事件。
完事儿之后,有点小冲动,于是就有了这篇文章,将有以下内容:
- 我为什么要爬取 NASA 的图片
- 我是如何爬取 NASA 图片的(超具体)
- 我失去了什么(高清大图)
- 我发现了什么机密(超劲爆)
我为什么要爬 NASA 的图片
我曾经超过 35 了,瑟瑟发抖啥时候被开了。
天天想着万一哪天就业了干点啥,想着玩个自媒体吧,天天给大家瞎文言。文言点历史谜团,宇宙神秘啥的,于是我就盯上了 NASA
。
NASA
有各种宇宙摸索工作,并且有相干的文章,访谈,图片,视频公开,这是不可多得的资源库。
我是如何爬取 NASA 图片的(超具体)
NASA
的网站是能够公开拜访的,地址是
https://www.nasa.gov/
关上当前,它首页长这样,能够看到各种内容。右上角还有个搜寻框,咱们输出 Mars
也就是 火星
稍等片刻,会展现出与 Mars
相干的各种内容,其中有一项 Mars Exploration
也就是 火星摸索
点开之后,就到了一个新页面,而后找到 Images
图片,就到了咱们爬取的指标页
https://www.nasa.gov/mission_pages/mars/images/index.html
页面下拉,你将会看到有个大大的按钮,下面写着 MORE IMAGES
,点击试试你就发现:
页面的内容,并不是页面间接加载的,而是通过 api
申请后,异步渲染的
F12, 关上浏览器开发者模式, 从新执行方才的步骤,察看申请信息,发现都会有如下的状况
看上去这个 url
地址十分重要,咱们先看他的申请地址:
https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi
留神看外面的参数
size=24&from=24
很显然,size
就是每次申请图片的数量,from
通过试验,是查问初始地位,咱们能够改它来获取其余内容
咱们再来看看它的返回信息:
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 659,
"max_score": null,
"hits": [{
"_index": "nasa-public",
"_type": "ubernode",
"_id": "450040",
"_score": null,
"_source": {
"image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover.",
"topics": ["3140", "3152"],
"nid": "450040",
"title": "NASA'Optometrists'Verify Mars 2020 Rover's 20/20 Vision","type":"ubernode","uri":"/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision","collections": ["4525","5246"],"link-or-attachment":"link","missions": ["6336"],"primary-tag":"6336","cardfeed-title":"NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",
"promo-date-time": "2019-08-05T17:49:00-04:00",
"secondary-tag": "3140",
"master-image": {
"fid": "603128",
"alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis.",
"width": "1600",
"id": "603128",
"title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis.",
"uri": "public://thumbnails/image/pia23314-16.jpg",
"height": "900"
},
"ubernode-type": "image"
},
"sort": [1565041740000]
}, {
"_index": "nasa-public",
"_type": "ubernode",
"_id": "433172",
"_score": null,
"_source": {"image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.","topics": ["3152"],"nid":"433172","title":"Opportunity Emerges in a Dusty Picture","type":"ubernode","uri":"/image-feature/opportunity-emerges-in-a-dusty-picture","collections": ["7628"],"link-or-attachment":"link","missions": ["3639"],"primary-tag":"3152","cardfeed-title":"Opportunity Emerges in a Dusty Picture","promo-date-time":"2018-09-26T12:39:00-04:00","secondary-tag":"7628","master-image": {"fid":"584263","alt":"NASA's Opportunity rover appears as a blip in the center of this square",
"width": "1400",
"id": "584263",
"title": "NASA's Opportunity rover appears as a blip in the center of this square","uri":"public://thumbnails/image/pia22549-16.jpg","height":"788"},"ubernode-type":"image"},"sort": [1537979940000]
}]
}
}
下面的 json
内容太长,我删除了一些反复的,实际上 hits
这个数组,也是 24 个,和页面上显示的图片数量是一样的。那根本能够判定,页面上的信息,就是从这个数组来的。
进而比照发现,master-image
字段下,就是咱们须要的信息,包含 图片地址
, 图片尺寸
, 图片题目
。
上面就上代码,组装申请 URL, 获取内容,下载图片 三步走
我应用了 Dart
语言,你们随便
import 'dart:convert';
import 'package:dio/dio.dart';
main() async {
// 每页数量是固定 24 个, 扭转初始值即可
for (int from = 0; from < 24 * 100; from = from + 24) {await getPage(from);
}
}
// 获取每一页的信息并且下载
Future<void> getPage(int from) async {
String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' +
from.toString() +
'&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi';
// 获取到内容
var res = await Dio().get(url);
var map = jsonDecode(res.toString());
(map['hits']['hits'] as List<dynamic>).forEach((element) async {Uri fileUri = Uri.parse(getUri(element));
String savePath = getSavePath(element);
await Dio().downloadUri(fileUri, savePath);
print('已下载:' + savePath);
});
}
// 获取图片下载地址
String getUri(dynamic element) {String uri = element['_source']['master-image']['uri'].toString();
uri = uri.replaceAll('public://',
'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/');
return uri;
}
// 解决信息,并且返回图片保留地址
String getSavePath(dynamic element) {String id = element['_id'];
String fid = element['_source']['master-image']['fid'].toString();
String title = element['_source']['master-image']['title'].toString();
String uri = element['_source']['master-image']['uri'].toString();
String savePath =
id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last;
savePath = savePath.replaceAll('/', '');
savePath = savePath.replaceAll('\\', '');
savePath = savePath.replaceAll('"','');
savePath = 'images/' + savePath;
return savePath;
}
下面的代码,还是很简略的,有教训的同学应该一看就懂。
走起来吧。
已下载: images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg
已下载: images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg
已下载: images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg
已下载: images/452007_605784_Danielson Crater on Mars.jpg
已下载: images/458478_615132_Gullies on Mars.jpg
已下载: images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg
已下载: images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg
已下载: images/470381_643473_CME.jpg
已下载: images/458813_615896_Mars.jpg
已下载: images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg
已下载: images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg
已下载: images/465488_631398_Cliffs in Ancient Ice on Mars.jpg
已下载: images/463659_626874_Avalanche on Mars.jpg
已下载: images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg
已下载: images/468636_639726_Mars' Jezero Crater.jpg
我失去了什么
这些图片
还有这些
图片,图片题目都有了, 够看一个月了我预计。
我发现了什么机密
这张图,我最喜爱了。一个如此清晰,一个如此浑浊,这是为什么?火星人的裂缝产生器?
好吧,真正的机密是:
NASA
的网站居然没有防采集,不信你也试试。。。