前言

最近，我应用爬虫技术，爬取了美国航空航天局，也就是你电影里常常见到的 NASA, 火星摸索的相干图片，有 10000 张吧。

嗯嗯，小事件，小事件。

完事儿之后，有点小冲动，于是就有了这篇文章，将有以下内容：

  1. 我为什么要爬取NASA的图片
  2. 我是如何爬取NASA图片的（超具体）
  3. 我失去了什么(高清大图)
  4. 我发现了什么机密（超劲爆）

我为什么要爬NASA的图片

我曾经超过35了，瑟瑟发抖啥时候被开了。

天天想着万一哪天就业了干点啥，想着玩个自媒体吧，天天给大家瞎文言。文言点历史谜团，宇宙神秘啥的，于是我就盯上了 NASA。

NASA 有各种宇宙摸索工作，并且有相干的文章，访谈，图片，视频公开，这是不可多得的资源库。

我是如何爬取NASA图片的（超具体）

NASA 的网站是能够公开拜访的，地址是

https://www.nasa.gov/

关上当前，它首页长这样，能够看到各种内容。右上角还有个搜寻框，咱们输出 Mars 也就是 火星

稍等片刻，会展现出与 Mars 相干的各种内容，其中有一项 Mars Exploration 也就是 火星摸索

点开之后，就到了一个新页面，而后找到 Images 图片，就到了咱们爬取的指标页

https://www.nasa.gov/mission_pages/mars/images/index.html

页面下拉，你将会看到有个大大的按钮，下面写着 MORE IMAGES， 点击试试你就发现：

页面的内容，并不是页面间接加载的，而是通过 api 申请后，异步渲染的

F12, 关上浏览器开发者模式,从新执行方才的步骤，察看申请信息，发现都会有如下的状况

看上去这个url地址十分重要，咱们先看他的申请地址：

https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi

留神看外面的参数

size=24&from=24

很显然，size 就是每次申请图片的数量，from 通过试验，是查问初始地位，咱们能够改它来获取其余内容

咱们再来看看它的返回信息：

{    "took": 3,    "timed_out": false,    "_shards": {        "total": 5,        "successful": 5,        "skipped": 0,        "failed": 0    },    "hits": {        "total": 659,        "max_score": null,        "hits": [{            "_index": "nasa-public",            "_type": "ubernode",            "_id": "450040",            "_score": null,            "_source": {                "image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover. ",                "topics": ["3140", "3152"],                "nid": "450040",                "title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",                "type": "ubernode",                "uri": "/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision",                "collections": ["4525", "5246"],                "link-or-attachment": "link",                "missions": ["6336"],                "primary-tag": "6336",                "cardfeed-title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",                "promo-date-time": "2019-08-05T17:49:00-04:00",                "secondary-tag": "3140",                "master-image": {                    "fid": "603128",                    "alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ",                    "width": "1600",                    "id": "603128",                    "title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ",                    "uri": "public://thumbnails/image/pia23314-16.jpg",                    "height": "900"                },                "ubernode-type": "image"            },            "sort": [1565041740000]        }, {            "_index": "nasa-public",            "_type": "ubernode",            "_id": "433172",            "_score": null,            "_source": {                "image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.",                "topics": ["3152"],                "nid": "433172",                "title": "Opportunity Emerges in a Dusty Picture",                "type": "ubernode",                "uri": "/image-feature/opportunity-emerges-in-a-dusty-picture",                "collections": ["7628"],                "link-or-attachment": "link",                "missions": ["3639"],                "primary-tag": "3152",                "cardfeed-title": "Opportunity Emerges in a Dusty Picture",                "promo-date-time": "2018-09-26T12:39:00-04:00",                "secondary-tag": "7628",                "master-image": {                    "fid": "584263",                    "alt": "NASA's Opportunity rover appears as a blip in the center of this square",                    "width": "1400",                    "id": "584263",                    "title": "NASA's Opportunity rover appears as a blip in the center of this square",                    "uri": "public://thumbnails/image/pia22549-16.jpg",                    "height": "788"                },                "ubernode-type": "image"            },            "sort": [1537979940000]        }]    }}

下面的 json 内容太长，我删除了一些反复的，实际上 hits 这个数组，也是24个，和页面上显示的图片数量是一样的。那根本能够判定，页面上的信息，就是从这个数组来的。

进而比照发现，master-image字段下，就是咱们须要的信息，包含 图片地址, 图片尺寸, 图片题目。

上面就上代码，组装申请URL, 获取内容，下载图片 三步走

我应用了 Dart 语言，你们随便

import 'dart:convert';import 'package:dio/dio.dart';main() async {  // 每页数量是固定24个,扭转初始值即可  for (int from = 0; from < 24 * 100; from = from + 24) {    await getPage(from);  }}//获取每一页的信息并且下载Future<void> getPage(int from) async {  String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' +      from.toString() +      '&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi';  //获取到内容  var res = await Dio().get(url);  var map = jsonDecode(res.toString());  (map['hits']['hits'] as List<dynamic>).forEach((element) async {    Uri fileUri = Uri.parse(getUri(element));    String savePath = getSavePath(element);    await Dio().downloadUri(fileUri, savePath);    print('已下载: ' + savePath);  });}//获取图片下载地址String getUri(dynamic element) {  String uri = element['_source']['master-image']['uri'].toString();  uri = uri.replaceAll('public://',      'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/');  return uri;}//解决信息，并且返回图片保留地址String getSavePath(dynamic element) {  String id = element['_id'];  String fid = element['_source']['master-image']['fid'].toString();  String title = element['_source']['master-image']['title'].toString();  String uri = element['_source']['master-image']['uri'].toString();  String savePath =      id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last;  savePath = savePath.replaceAll('/', '');  savePath = savePath.replaceAll('\\', '');  savePath = savePath.replaceAll('"', '');  savePath = 'images/' + savePath;  return savePath;}

下面的代码，还是很简略的，有教训的同学应该一看就懂。

走起来吧。

已下载: images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg已下载: images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg已下载: images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg已下载: images/452007_605784_Danielson Crater on Mars.jpg已下载: images/458478_615132_Gullies on Mars.jpg已下载: images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg已下载: images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg已下载: images/470381_643473_CME.jpg已下载: images/458813_615896_Mars.jpg已下载: images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg已下载: images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg已下载: images/465488_631398_Cliffs in Ancient Ice on Mars.jpg已下载: images/463659_626874_Avalanche on Mars.jpg已下载: images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg已下载: images/468636_639726_Mars' Jezero Crater.jpg

我失去了什么

这些图片

还有这些

图片，图片题目都有了, 够看一个月了我预计。

我发现了什么机密

这张图，我最喜爱了。一个如此清晰，一个如此浑浊，这是为什么？ 火星人的裂缝产生器？

好吧，真正的机密是：

NASA 的网站居然没有防采集，不信你也试试。。。