Mac 下编译 Hadoop-2.9.2
系统环境
系统：Mac OS_10.14.4
maven：Apache Maven 3.6.0
jdk: jdk_1.8.0_201
ProtocolBuffer：portoc-2.5.0
zlib：1.2.3-lib
OpenSSL：1_0_2r
cmale：3.14.3-win64-x64
Cygwin：2.897_x86_64
hadoop：hadoop-2.9.2
snappy: 搭建编译环境
1.java 环境在 oracle 官网下载安装安装好配置好环境变量
2.maven 我用的是 brew 直接安装的 `brew install maven`
3.cmake 使用 brew 安装 `brew install cmake`
4.openssl 使用 brew 安装 `brew install openssl`
5.snappy 使用 brew 安装 `brew install snappy`
6.zlib 使用 brew 安装 `brew install zlib`
7.protobuf 由于必须使用 2.5.0 版本需要自行 [下载](https://github.com/protocolbuffers/protobuf/releases) 安装
下载后解压后进入 protobuf 目录
# 配置
$ ./configure --prefix=/usr/local/Cellow/protoc/2.5/
# 编译
$ make
# 安装
$ make install
# 配置环境变量 `~/.bash_profile` 或 `~/.zshrc`
export PATH="/usr/local/Cellow/protoc/2.5/bin:$PATH"
# 验证 protoc 版本
$ protoc --version
过程中可能遇到的问题
- 1.[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part …<exec dir=”/Users/Downloads/hadoop-2.9.2/hadoop-tools/hadoop-pipes/target/native” executable=”cmake” failonerror=”true”>… @ 5:153 in /Users/Downloads/hadoop-2.9.2/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
解决办法：由于这个 xml 中的编译指令中需要用到一个环境变量 OPENSSL_ROOT_DIR 由于我们前面没有配置所以他在执行的时候就会报错，需要在 ~/.bash_profile 或 ~/.zshrc 中指定 OPENSSL_ROOT_DIR 的安装目录我的配置是export OPENSSL_ROOT_DIR="/usr/local/Cellor/openssl/1.0.2r"
- 2. 由于 OpenSSL 爆出有重大的漏洞 Mac OS 没有使用 OpenSSL 而是使用了
LibreSSl可以在你的 Mac 下执行which openssl然后找到/usr/bin/openssl执行/usr/bin/openssl version就会出现 LibreSSl 2.6.4，但是编译 hadoop 必须要使用 openssl，可以吧 openssl 的引用改一个名称，但是 Mac 是不允许你关闭 Mac 的保护模式但是不建议这样做，我们可以修改系统的 shell 的环境变量使自己配置的环境变量覆盖系统的环境变量，例如：export PATH="/usr/local/Cellor/openssl/1.0.2r/bin:$PATH"
编译 hadoop
# 下载[hadoop 源码](https://hadoop.apache.org/releases.html)
# 解压缩后进入 hadoop 源码路径
# 执行一条 maven 命令（你的 maven 需要配置到环境变量中）$ mvn package -Pdist,native -DskipTests -Dtar
# 编译过程不会太长，但是下载各种依赖就会浪费很长时间，耐心等待！# 过程中可能会出现各各样的问题我只是列举了我花费比较久时间解决的问题，如果无法解决可以留言。编译完成
经过漫长的编译终于出现了令人兴奋的画面
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [1.679 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [1.158 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [1.071 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [2.897 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [0.297 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [1.699 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [4.280 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [2.940 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [4.245 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [2.548 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:05 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [3.707 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [9.596 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [0.046 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [18.270 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [46.683 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [7.154 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [14.156 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [3.215 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [2.857 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [13.849 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [0.100 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [0.073 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [10.318 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [23.624 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [3.413 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [0.091 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [8.550 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [26.322 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [2.491 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [4.873 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [3.360 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [16.652 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [0.993 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [4.276 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [2.371 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [2.464 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [3.747 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [5.310 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [1.894 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [0.053 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [2.099 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [1.366 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [0.061 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [0.098 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [7.639 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [0.157 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [15.248 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [9.780 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [2.561 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [8.028 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [4.123 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [4.353 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [1.349 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [3.550 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [3.609 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [2.987 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [3.530 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [1.456 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [1.654 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [3.363 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [2.833 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [1.550 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [1.378 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [2.101 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [6.900 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [2.804 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [7.562 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [4.022 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [9.670 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [5.619 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [0.638 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [4.075 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [10.540 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [12.875 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [14.670 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [0.058 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:02 min]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [0.473 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [0.042 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:17 min
[INFO] Finished at: 2019-05-06T05:03:29-07:00
[INFO] ------------------------------------------------------------------------
MacBookPro:hadoop-2.9.2-src leone$