关于后端:爬了10000张NASA关于火星探索的图片我发现了一个秘密

44次阅读

共计 6274 个字符，预计需要花费 16 分钟才能阅读完成。

前言

最近，我应用爬虫技术，爬取了美国航空航天局，也就是你电影里常常见到的 NASA, 火星摸索的相干图片，有 10000 张吧。

嗯嗯，小事件，小事件。

完事儿之后，有点小冲动，于是就有了这篇文章，将有以下内容：

  1. 我为什么要爬取 NASA 的图片
  2. 我是如何爬取 NASA 图片的（超具体）
  3. 我失去了什么(高清大图)
  4. 我发现了什么机密（超劲爆）

我为什么要爬 NASA 的图片

我曾经超过 35 了，瑟瑟发抖啥时候被开了。

天天想着万一哪天就业了干点啥，想着玩个自媒体吧，天天给大家瞎文言。文言点历史谜团，宇宙神秘啥的，于是我就盯上了 NASA。

NASA 有各种宇宙摸索工作，并且有相干的文章，访谈，图片，视频公开，这是不可多得的资源库。

我是如何爬取 NASA 图片的（超具体）

NASA 的网站是能够公开拜访的，地址是

https://www.nasa.gov/

关上当前，它首页长这样，能够看到各种内容。右上角还有个搜寻框，咱们输出 Mars 也就是 火星

稍等片刻，会展现出与 Mars 相干的各种内容，其中有一项 Mars Exploration 也就是 火星摸索

点开之后，就到了一个新页面，而后找到 Images 图片，就到了咱们爬取的指标页

https://www.nasa.gov/mission_pages/mars/images/index.html

页面下拉，你将会看到有个大大的按钮，下面写着 MORE IMAGES，点击试试你就发现：

页面的内容，并不是页面间接加载的，而是通过 api 申请后，异步渲染的

F12, 关上浏览器开发者模式, 从新执行方才的步骤，察看申请信息，发现都会有如下的状况

看上去这个 url 地址十分重要，咱们先看他的申请地址：

https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi

留神看外面的参数

size=24&from=24

很显然，size 就是每次申请图片的数量，from 通过试验，是查问初始地位，咱们能够改它来获取其余内容

咱们再来看看它的返回信息：

{
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 659,
        "max_score": null,
        "hits": [{
            "_index": "nasa-public",
            "_type": "ubernode",
            "_id": "450040",
            "_score": null,
            "_source": {
                "image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover.",
                "topics": ["3140", "3152"],
                "nid": "450040",
                "title": "NASA'Optometrists'Verify Mars 2020 Rover's 20/20 Vision","type":"ubernode","uri":"/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision","collections": ["4525","5246"],"link-or-attachment":"link","missions": ["6336"],"primary-tag":"6336","cardfeed-title":"NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",
                "promo-date-time": "2019-08-05T17:49:00-04:00",
                "secondary-tag": "3140",
                "master-image": {
                    "fid": "603128",
                    "alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis.",
                    "width": "1600",
                    "id": "603128",
                    "title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis.",
                    "uri": "public://thumbnails/image/pia23314-16.jpg",
                    "height": "900"
                },
                "ubernode-type": "image"
            },
            "sort": [1565041740000]
        }, {
            "_index": "nasa-public",
            "_type": "ubernode",
            "_id": "433172",
            "_score": null,
            "_source": {"image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.","topics": ["3152"],"nid":"433172","title":"Opportunity Emerges in a Dusty Picture","type":"ubernode","uri":"/image-feature/opportunity-emerges-in-a-dusty-picture","collections": ["7628"],"link-or-attachment":"link","missions": ["3639"],"primary-tag":"3152","cardfeed-title":"Opportunity Emerges in a Dusty Picture","promo-date-time":"2018-09-26T12:39:00-04:00","secondary-tag":"7628","master-image": {"fid":"584263","alt":"NASA's Opportunity rover appears as a blip in the center of this square",
                    "width": "1400",
                    "id": "584263",
                    "title": "NASA's Opportunity rover appears as a blip in the center of this square","uri":"public://thumbnails/image/pia22549-16.jpg","height":"788"},"ubernode-type":"image"},"sort": [1537979940000]
        }]
    }
}

下面的 json 内容太长，我删除了一些反复的，实际上 hits 这个数组，也是 24 个，和页面上显示的图片数量是一样的。那根本能够判定，页面上的信息，就是从这个数组来的。

进而比照发现，master-image字段下，就是咱们须要的信息，包含 图片地址 , 图片尺寸 , 图片题目。

上面就上代码，组装申请 URL, 获取内容，下载图片 三步走

我应用了 Dart 语言，你们随便

import 'dart:convert';
import 'package:dio/dio.dart';

main() async {
  // 每页数量是固定 24 个, 扭转初始值即可
  for (int from = 0; from < 24 * 100; from = from + 24) {await getPage(from);
  }
}

// 获取每一页的信息并且下载
Future<void> getPage(int from) async {
  String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' +
      from.toString() +
      '&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi';

  // 获取到内容
  var res = await Dio().get(url);
  var map = jsonDecode(res.toString());

  (map['hits']['hits'] as List<dynamic>).forEach((element) async {Uri fileUri = Uri.parse(getUri(element));
    String savePath = getSavePath(element);

    await Dio().downloadUri(fileUri, savePath);
    print('已下载:' + savePath);
  });
}

// 获取图片下载地址
String getUri(dynamic element) {String uri = element['_source']['master-image']['uri'].toString();

  uri = uri.replaceAll('public://',
      'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/');

  return uri;
}

// 解决信息，并且返回图片保留地址
String getSavePath(dynamic element) {String id = element['_id'];
  String fid = element['_source']['master-image']['fid'].toString();
  String title = element['_source']['master-image']['title'].toString();
  String uri = element['_source']['master-image']['uri'].toString();

  String savePath =
      id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last;
  savePath = savePath.replaceAll('/', '');
  savePath = savePath.replaceAll('\\', '');
  savePath = savePath.replaceAll('"','');
  savePath = 'images/' + savePath;

  return savePath;
}

下面的代码，还是很简略的，有教训的同学应该一看就懂。

走起来吧。

已下载: images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg
已下载: images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg
已下载: images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg
已下载: images/452007_605784_Danielson Crater on Mars.jpg
已下载: images/458478_615132_Gullies on Mars.jpg
已下载: images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg
已下载: images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg
已下载: images/470381_643473_CME.jpg
已下载: images/458813_615896_Mars.jpg
已下载: images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg
已下载: images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg
已下载: images/465488_631398_Cliffs in Ancient Ice on Mars.jpg
已下载: images/463659_626874_Avalanche on Mars.jpg
已下载: images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg
已下载: images/468636_639726_Mars' Jezero Crater.jpg

我失去了什么

这些图片

还有这些

图片，图片题目都有了, 够看一个月了我预计。

我发现了什么机密

这张图，我最喜爱了。一个如此清晰，一个如此浑浊，这是为什么？火星人的裂缝产生器？

好吧，真正的机密是：

NASA 的网站居然没有防采集，不信你也试试。。。

正文完
后端
发表至： 后端
2021-06-30
 0
关于后端:跨全端SDK技术演进
关于后端:Mybatis系列教材-二十一-相关概念-PageHelper
关于后端:Flink-ML的新特性解析与应用
关于后端:牛客网论坛最具争议的Java面试成神笔记GitHub已下载量已过百万