乐趣区

Linux升级wget/curl用于下载https文件的过程

因为需要提升服务器的性能以及支持 mysql 更新版本的某些特性,因此决定升级 mysql 版本从 5.1.30 到 8.0,目标确定下来就开始干。

Mysql 安装方式选择
在 Linux 上安装应用,一般有三种方式,优劣对比分别为:
因此我们选择二进制安装,安装简单、方便,支持多个 Mysql 版本同时存在。
在 Linux 上安装二进制版本的应用,统一为三步:
// 通过配置自动生成文件
./configure
// 编译文件
make
// 检查自测单元,看编译是否通过,可以省略该步,不影响安装
make check
// 安装
make install
卸载通过二进制安装的程序:
// 方式一: 在编译目录里执行卸载
make uninstall
// 方式二:找到安装目录,然后删除,如 nettle 程序
$find / -name nettle
/usr/include/nettle
rm -rf /usr/include/nettle

wget 不支持 https

我们可以在 mysql 官网下载最新版本的 mysql8.0.13 二进制文件,注意官网提供的下载链接是 https 协议的,当我们在服务器执行下载命令:
// 使用 wget 或者 curl 来下载文件
wget https://dev.mysql.com/downloads/file/?id=480751
curl -O https://dev.mysql.com/downloads/file/?id=480751
会报错:
// wget 加上 –no-check-certificate 依然不可以
$wget https://dev.mysql.com/downloads/file/?id=480751
–2018-12-12 16:57:54– https://dev.mysql.com/downloads/file/?id=480751
Resolving dev.mysql.com (dev.mysql.com)… 137.254.60.11
Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:443… connected.
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [40]: Handshake failed
Unable to establish SSL connection.

// curl 加上 –insecure 依然不可以
$curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error
根据网上查询到的答案,原因均为版本过低,当前的版本不支持 https 协议的下载:
So the error actually happens with www.coursera.org and the reason is missing support for SNI. You need to upgrade your version of wget.
当前的版本:
$wget –version
GNU Wget 1.16.3 built on linux-gnu.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie -psl +ssl/gnutls

Wgetrc:
/usr/local/etc/wgetrc (system)
Locale:
/usr/local/share/locale
Compile:
gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC=”/usr/local/etc/wgetrc”
-DLOCALEDIR=”/usr/local/share/locale” -I. -I../lib -I../lib
-DHAVE_LIBGNUTLS -DNDEBUG
Link:
gcc -DHAVE_LIBGNUTLS -DNDEBUG -lpcre -lnettle -lgnutls -lz -lidn
-lrt ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a

Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.

——————

$curl.7.19.7 –version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

wget 升级
既然是版本过低,升级即可,直接安装新版本的 wget,然后卸载掉原有的 wget 文件即可。
下载完成 1.20 版本之后
// 获得文件 wget-1.20.tar.gz
wget http://mirror.sergal.org/gnu/wget/wget-1.20.tar.gz
// 解压缩
tar -xzvf wget-1.20.tar.gz
// 进入解压后的文件夹
cd wget-1.20
// 开始配置
./configure
// 然而报错了:

checking for GNUTLS… no
configure: error: Package requirements (gnutls) were not met:

No package ‘gnutls’ found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GNUTLS_CFLAGS
and GNUTLS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
报错信息显示我们没有安装 gnutls 依赖,需要继续进行安装,更为详细的报错信息,可以查看 config.log:
// 查看详细报错信息
vim config.log


PKG_CONFIG=’/usr/local/bin/pkg-config’

configure:44443: checking for GNUTLS
configure:44450: $PKG_CONFIG –exists –print-errors “gnutls”
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc’
to the PKG_CONFIG_PATH environment variable
No package ‘gnutls’ found
configure:44453: $? = 1
configure:44467: $PKG_CONFIG –exists –print-errors “gnutls”
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc’
to the PKG_CONFIG_PATH environment variable
No package ‘gnutls’ found
configure:44470: $? = 1
configure:44484: result: no
No package ‘gnutls’ found
configure:44500: error: Package requirements (gnutls) were not met:

No package ‘gnutls’ found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GNUTLS_CFLAGS
and GNUTLS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
这里可以看出配置脚本实际是执行了:
$ /usr/local/bin/pkg-config –exists –print-errors “gnutls”
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc’
to the PKG_CONFIG_PATH environment variable
No package ‘gnutls’ found
安装 gnutls 库
gnutls 全称 GNU Transport Layer Security Library,即基于 GNU 版权协议的传输层安全协议,是 wget 支持 https 中的 ssl 协议的基础库。
我们可以在官方提供的镜像库里快速下载并安装:
// 下载 gnutls 二进制文件
wget http://www.ring.gr.jp/pub/net/gnupg/gnutls/v3.6/gnutls-3.6.4.tar.xz
// 解压 xz 文件
xz -d gnutls-3.6.4.tar.xz
tar -xvf gnutls-3.6.4.tar
cd gnutls-3.6.4
./configure

// 报错:

checking for NETTLE… no
configure: error:
***
*** Libnettle 3.4 was not found.
// 如果觉得可能不安全,可以下载 md5 签名文件验证文件,但是这个文件验证后发现签名过期了,所以没办法验证了 wget http://www.ring.gr.jp/pub/net…gpg –verify gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.targpg –recv-key F1679A65gpg –verify –verbose gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.tar
查看详细报错信息:
$ vim config.log


configure:10032: checking for NETTLE
configure:10039: $PKG_CONFIG –exists –print-errors “nettle >= 3.4”
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc’
to the PKG_CONFIG_PATH environment variable
No package ‘nettle’ found
configure:10042: $? = 1
configure:10056: $PKG_CONFIG –exists –print-errors “nettle >= 3.4”
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc’
to the PKG_CONFIG_PATH environment variable
No package ‘nettle’ found
configure:10059: $? = 1
configure:10073: result: no
No package ‘nettle’ found
configure:10090: error:
***
*** Libnettle 3.4 was not found.
结果显示我们需要 3.4 版本以上的 Libnettle 库,继续安装。
安装 Libnettle 库
Nettle 库是用于跨平台的底层密码库,包含加密和解密的不同算法。我们下载并安装 nettle 库:
wget ftp://ftp.gnu.org/gnu/nettle/nettle-3.4.1.tar.gz
tar -xzvf nettle-3.4.1.tar.gz
cd nettle-3.4.1
./configure
// 安装成功

configure: summary of build options:

Version: nettle 3.4.1
Host type: x86_64-unknown-linux-gnu
ABI: 64
Assembly files: x86_64
Install prefix: /usr/local
Library directory: ${exec_prefix}/lib64
Compiler: gcc
Static libraries: yes
Shared libraries: yes
Public key crypto: no
Using mini-gmp: no
Documentation: yes

make
make install
根据官方文档,我们安装完成后应该会有两个文件 lib{hogweed,nettle}.so,然而我们只能发现其中一个:
make install &&chmod -v 755 /usr/lib/lib{hogweed,nettle}.so &&install -v -m755 -d /usr/share/doc/nettle-3.4.1 &&install -v -m644 nettle.html /usr/share/doc/nettle-3.4.1
$ ll | grep ‘\.so’
-rwxr-xr-x 1 root root 3675341 Dec 12 19:15 libnettle.so
$ ll | grep weed
-rw-rw-r– 1 work work 529 Dec 10 15:30 hogweed.pc
-rw-r–r– 1 work work 590 Nov 19 2017 hogweed.pc.in
-rw-rw-r– 1 work work 298 Dec 10 15:30 libhogweed.map
-rw-r–r– 1 work work 338 Nov 19 2017 libhogweed.map.in
少了一个 libhogweed.so 文件,稍后我们编译 gnutls 时会发现这个导致的问题。
继续编译 gnutls

既然 nettle 安装完成了,我们可以继续安装 gnutls:
./configure


configure: error:
***
*** Libnettle 3.4 was not found.
依然报错缺失库,但我们明明已经安装了,为什么找不到呢?我们用包管理工具查找一下:
$ pkg-config –modversion nettle
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc’
to the PKG_CONFIG_PATH environment variable
No package ‘nettle’ found
我们找下这个 nettle.pc 刚才安装到哪里去了:
$ locate nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc.in
/usr/lib64/pkgconfig/nettle.pc
/usr/local/lib64/pkgconfig/nettle.pc
而我们 pkg-config 默认的管理包检索路径为 /usr/lib/pkgconfig,所以无法正常找到,参考 pkgconfig 文档,有两种方案:
// 方案一:链接该文件到默认目录中
ln -s /usr/local/lib64/pkgconfig/nettle.pc /usr/lib/pkgconfig/nettle.pc
// 方案二:全局变量中更改包的检索路径(只在本次终端窗口生效,退出后恢复,所以只能临时使用一下)
$ echo $PKG_CONFIG_PATH

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64/pkgconfig/
$ echo $PKG_CONFIG_PATH
:/usr/local/lib64/pkgconfig/

// 任一方案执行后结果
$ pkg-config –modversion nettle
3.4.1
此时,我们继续安装:
$ ./configure

checking for NETTLE… yes
checking for HOGWEED… no
configure: error:
***
*** Libhogweed (nettle’s companion library) was not found. Note that you must compile nettle with gmp support.
可以看到,我们的 nettle 库已经安装成功,但是 hogweed 却检查没有找到,提示中也写明了 libhogweed 需要字 gmp 库已经安装完成的情况下重新编译 nettle 才可以被安装。
有人提出过相关的问题,我们也可以从官网文档上更详细的知道这个 Nettle 对于 libhogweed 的依赖:
5 LinkingNettle actually consists of two libraries, libnettle and libhogweed. The libhogweed library contains those functions of Nettle that uses bignum operations, and depends on the GMP library. With this division, linking works the same for both static and dynamic libraries.If an application uses only the symmetric crypto algorithms of Nettle (i.e., block ciphers, hash functions, and the like), it’s sufficient to link with -lnettle. If an application also uses public-key algorithms, the recommended linker flags are -lhogweed -lnettle -lgmp. If the involved libraries are installed as dynamic libraries, it may be sufficient to link with just -lhogweed, and the loader will resolve the dependencies automatically.

总而言之,就是没有 libhogweed.so 这个文件不行,而它只能由 nettle 进行安装。根据 nettle 库官方资料显示,libhogweed.so 应该在 nettle 安装时被自动生成,然而我们在上面的安装过程中并没有生成。那是不是因为我没有安装 gmp 导致的呢?
安装 gmp 库
我们下载 gmp 库并安装,可以在编译 Nettle 的 config.log 中查看有一条 warning,指明了版本需求:
$ vim config.log


configure:6583: result: no
configure:6594: WARNING: GNU MP not found, or too old. GMP-6.0 or later is needed, see https://gmplib.org/.
Support for public key algorithms will be unavailable.
所以我们需要下载 6.0 版本后的:
// 这里我只找到了官网的 https 版本,没办法,只好本地下载,然后 rz 到服务器,因为是二进制文件,要带上 -be 参数
rz -be
// 然后正常编译
$ ./configure & make & make install

Libraries have been installed in:
/usr/local/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the ‘-LLIBDIR’
flag during linking and do at least one of the following:
– add LIBDIR to the ‘LD_LIBRARY_PATH’ environment variable
during execution
– add LIBDIR to the ‘LD_RUN_PATH’ environment variable
during linking
– use the ‘-Wl,-rpath -Wl,LIBDIR’ linker flag
– have your system administrator add LIBDIR to ‘/etc/ld.so.conf’
这里提醒我们需要将动态库链接到缓存中,我们采用第四种方案,可以参考 ldconfig 命令:
$ vim /etc/ld.so.conf

// 添上安装的.so 文件路径
/usr/local/lib
:wq

$ ldconfig
$ ldconfig -v | grep gmp
libgmp.so.10 -> libgmp.so.10.3.2
libgmpxx.so.4 -> libgmpxx.so.4.1.0
libgmp.so.3 -> libgmp.so.3.5.0
看到 libgmp.so.10 就是我们安装的最新版本,现在 OK 了。
然后重新编译安装 nettle,会生成 libhogweed.so 文件:
$ ll | grep weed
-rw-r–r– 1 root root 541 Dec 12 22:12 hogweed.pc
-rw-r–r– 1 work work 590 Nov 19 2017 hogweed.pc.in
-rw-r–r– 1 root root 6154192 Dec 12 22:13 libhogweed.a
-rw-r–r– 1 root root 298 Dec 12 22:12 libhogweed.map
-rw-r–r– 1 work work 338 Nov 19 2017 libhogweed.map.in
-rwxr-xr-x 1 root root 5519996 Dec 12 22:13 libhogweed.so
-rw-r–r– 1 root root 8 Dec 12 22:13 libhogweed.stamp
请注意如果安装完成后,如果出现多个版本的 gmp 库,请删除老版本的。具体删除哪一项请自行斟酌,我删除了所有的,然后在编译的过程中,会报错:can’t find libgmp.so.3,说明 libgmp.so.3 这个是基础库,请不要动!等我删除了老版本的,重新编译 nettle 就 OK。如果你安装成功了新版本,依然编译不成功,请参考这个。
恐怖的依赖地狱
用二进制来安装的时候,总是会出现各种各样的问题,缺少各种依赖的包,解决方法就是缺什么就去安什么,但是会非常恐怖。为了解决 nettle 安装的问题,除了上面的 gmp,我还安装了最新版本的各种库:

libunistring:https://www.gnu.org/software/…

p11-kit:https://github.com/p11-glue/p…

libffi:https://sourceware.org/libffi/

pkg-config: https://pkg-config.freedeskto…

libtasn1: https://ftp.gnu.org/gnu/libta…

同时,由于 gnutls 编译不通过的问题,又升级了 pkg-config,它依赖于 Libtasn1。
继续安装 gnutls 库(失败、暂时放弃更新)
./configure
// 此时没有错误信息了,但是还有很多 WARNING 信息
*** autogen not found. Will not link against libopts.
*** You will not be able to create source packages with ‘make dist’
because gtk-doc >= 1.14 is not found.
*** LIBIDN2 was not found. You will not be able to use IDN2008 support
*** libunbound was not found. Libdane will not be built.
*** trousers was not found. TPM support will be disabled.
*** `guile-snarf’ from Guile not found. Guile bindings not built.

*** The DNSSEC root key file in /etc/unbound/root.key was not found.
*** This file is needed for the verification of DNSSEC responses.
*** Use the command: unbound-anchor -a “/etc/unbound/root.key”
*** to generate or update it.

// 继续编译,又报错了
make

tlsproxy/buffer.c:40: error: redefinition of typedef ‘buffer_t’
tlsproxy/buffer.h:31: note: previous declaration of ‘buffer_t’ was here
暂时放弃更新 wget,过几天继续尝试,解决各种问题太费时间了
如果想要减少 warning 信息,可以更新 autogen 等库:

autogen: https://ftp.gnu.org/gnu/autog…

guile: http://alpha.gnu.org/gnu/guile/

gcc: http://ftp.tsukuba.wide.ad.jp…

mpc: https://ftp.gnu.org/gnu/mpc/

安装 autogen 过程中需要依赖 guile,然而安装 guile 时又报错:guile configure: error: Cannot find a type to use in place of socklen_t 放弃更新 autogen。
尝试 curl 更新,层层依赖,放弃
$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error
根据报错原因和网上资料是由于版本过老,需要更新 curl 版本。
从官方地址下载 curl 后安装,再次用新版本的 curl 请求:
$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
还是报错,根据上面的资料,如果依然不能解决问题,需要更新 NSS,NSS 和 OpenSSl 类似,都属于底层密码学,由 Mozilla 维护,MDN 文档提供安装说明,它跟前面的二进制文件略有不同,不提供 configure 自动配置,详细的查看它的文档。
安装 NSS 库比较麻烦,还需要再安装 GYP 库,想起来又是层层嵌套的依赖关系,放弃更新。
解决方案:本地下载,rz 上传
在耗费两天的时间后,我及时的终止了无畏的尝试,转而使用本地下载 mysql8.0 文件,然后 rz -be 上传到服务器,搞定。
结论:
Linux 上层层依赖的二进制文件安装简直是地狱版的体验,在给我们带来高自由度的同时也有无尽的烦恼,然而 yum 安装版本又过低,不能满足需求。虽然最终还是没有成功更新 wget 或者 curl,但是在整个过程中,也学习到了很多的新东西,在这篇文章总结一下过程,希望也能帮助一些人在某一步骤遇到的问题。
解决问题整体思路和过程

参考资料

mysql8.0 官网下载地址:https://dev.mysql.com/downloa…

Mysql 三种安装方式详解:https://www.jianshu.com/p/a04…

[StackOverFlow] wget ssl alert handshake failure:https://stackoverflow.com/que…

卸载二进制程序:http://www.blogjava.net/zhyiw…

wget 下载地址:http://mirror.sergal.org/gnu/…

gnutls 下载地址:http://www.ring.gr.jp/pub/net…

利用.sig 文件验证数据的完整性:https://blog.csdn.net/xiazhiy…

下载安装 nettle http://www.linuxfromscratch.o…

nettle 官方文档:http://www.lysator.liu.se/~ni…

gmp 下载地址:https://gmplib.org/

ldconfig 命令:http://man.linuxde.net/ldconfig

SecureCRT rz 上传文件失败问题:https://blog.csdn.net/heavend…

初识 NSS,一文了解全貌:https://cloud.tencent.com/dev…

MDN 文档 NSS:https://developer.mozilla.org…

curl: (35) SSL connect error:https://stackoverflow.com/que…

简述 configure、pkg-config、pkg_config_path 三者的关系:http://www.mike.org.cn/articl…

How to compile GnuTLS: https://stackoverflow.com/que…

退出移动版