1、背景
最近在学习hadoop，此篇文章简略记录一下通过源码来编译hadoop。为什么要从新编译hadoop源码，是因为为了匹配不同操作系统的本地库环境。
2、编译源码
2.1 下载并解压源码
[root@hadoop01 ~]# mkdir /opt/hadoop[root@hadoop01 ~]# cd /opt/hadoop/[root@hadoop01 hadoop]# wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4-src.tar.gz[root@hadoop01 hadoop]# tar -zxvf hadoop-3.3.4-src.tar.gz[root@hadoop01 hadoop]# rm -rvf hadoop-3.3.4-src.tar.gz2.2 查看编译hadoop必要的环境
[root@hadoop01 hadoop]# pwd/opt/hadoop[root@hadoop01 hadoop]# cd hadoop-3.3.4-src/[root@hadoop01 hadoop-3.3.4-src]# cat BUILDING.txtBuild instructions for Hadoop----------------------------------------------------------------------------------Requirements:* Unix System* JDK 1.8* Maven 3.3 or later* Protocol Buffers 3.7.1 (if compiling native code)* CMake 3.1 or newer (if compiling native code)* Zlib devel (if compiling native code)* Cyrus SASL devel (if compiling native code)* One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio, Clang (community version), Clang (version for iOS 9 and later) (if compiling native code)* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)* Doxygen ( if compiling libhdfspp and generating the documents )* Internet connection for first build (to fetch all Maven and Hadoop dependencies)* python (for releasedocs)* bats (for shell code testing)* Node.js / bower / Ember-cli (for YARN UI v2 building)----------------------------------------------------------------------------------能够看到须要JDK1.8及以上、maven3.3及以上等等
2.3 装置JDK
须要留神JDK的版本,参考这个文档。https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
2.4 装置maven
[root@hadoop01 hadoop]# wget https://dlcdn.apache.org/maven/maven-3/3.9.0/binaries/apache-maven-3.9.0-bin.tar.gz[root@hadoop01 hadoop]# tar -zxvf apache-maven-3.9.0-bin.tar.gz -C /usr/local# 编辑环境变量[root@hadoop01 hadoop]# vim /etc/profile# 配置mavenexport M2_HOME=/usr/local/apache-maven-3.9.0export PATH=${M2_HOME}/bin:$PATH[root@hadoop01 hadoop]# source /etc/profile# 查看maven的版本[root@hadoop01 apache-maven-3.9.0]# mvn -versionApache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584)Maven home: /usr/local/apache-maven-3.9.0Java version: 1.8.0_333, vendor: Oracle Corporation, runtime: /usr/local/jdk8/jreDefault locale: zh_CN, platform encoding: UTF-8OS name: "linux", version: "5.11.12-300.el7.aarch64", arch: "aarch64", family: "unix"# 配置aliyun镜像减速拜访[root@hadoop01 hadoop]# vim /usr/local/apache-maven-3.9.0/conf/settings.xml<mirrors> <mirror> <id>alimaven</id> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> <mirrorOf>central</mirrorOf> </mirror></mirrors>2.5 装置编译相干的依赖
[root@hadoop01 hadoop]# yum install gcc gcc-c++ make autoconf automake libtool curl lzo-devel zlib-devel openssl openssl-devel ncurses-devel snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst zlib doxygen cyrus-sasl* saslwrapper-devel* -y2.6 装置cmake
hadoop要求* CMake 3.1 or newer (if compiling native code) cmake的版本在3.1及其以上。cmake不是必须的。
# 卸载已有的cmake[root@hadoop01 hadoop]# yum erase cmake# 下载cmake(此处须要依据本人的操作系统进行下载)[root@hadoop01 hadoop]# wget https://github.com/Kitware/CMake/releases/download/v3.25.2/cmake-3.25.2.tar.gz[root@hadoop01 hadoop]# tar -zxvf cmake-3.25.2.tar.gz# 编译和装置cmake[root@hadoop01 cmake-3.25.2-linux-aarch64]# cd cmake-3.25.2/ && ./configure && make && make install# 查看cmake的版本[root@hadoop01 cmake-3.25.2]# cmake -versioncmake version 3.25.2CMake suite maintained and supported by Kitware (kitware.com/cmake).2.7 装置Snappy
* Snappy compression (only used for hadoop-mapreduce-client-nativetask)
# 卸载已装置的snappy[root@hadoop01 hadoop]# rm -rf /usr/local/lib/libsnappy* && rm -rf /lib64/libsnappy*[root@hadoop01 hadoop]# wget https://src.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.3.tar.gz/7358c82f133dc77798e4c2062a749b73/snappy-1.1.3.tar.gz[root@hadoop01 snappy]# tar -zxvf snappy-1.1.3.tar.gz[root@hadoop01 snappy]# cd snappy-1.1.3/ && ./configure && make && make install[root@hadoop01 build]#2.8 装置ProtocolBuffer
* Protocol Buffers 3.7.1 (if compiling native code) 装置3.7.1的版本
[root@hadoop01 hadoop]# wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz[root@hadoop01 hadoop]# tar -zxvf protobuf-java-3.7.1.tar.gz# 编译和装置[root@hadoop01 hadoop]# cd protobuf-3.7.1/ && ./autogen.sh && ./configure && make && make install# 验证是否装置胜利[root@hadoop01 protobuf-3.7.1]# protoc --versionlibprotoc 3.7.12.9 编译hadoop
[root@hadoop01 hadoop-3.3.4-src]# pwd/opt/hadoop/hadoop-3.3.4-src[root@hadoop01 hadoop-3.3.4-src]# export MAVEN_OPTS="-Xms3072m -Xmx3072m" && mvn clean package -Pdist,native -DskipTests -Dtar -Dbundle.snappy -Dsnappy.lib=/usr/local/lib -e此处的mvn 命令也能够从BUILDING.txt文件中获取。
2.10 编译后的安装包门路
hadoop-3.3.4-src/hadoop-dist/target/hadoop-3.3.4.tar.gz
2.11 检测native
[root@hadoop01 hadoop]# tar -zxvf hadoop-3.3.4.tar.gz[root@hadoop01 bin]# cd hadoop-3.3.4/bin[root@hadoop01 bin]# ./hadoop checknative -a2023-02-18 16:58:39,698 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native2023-02-18 16:58:39,700 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library2023-02-18 16:58:39,700 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable2023-02-18 16:58:39,760 INFO nativeio.NativeIO: The native code was built without PMDK support.Native library checking:hadoop: true /opt/hadoop/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0zlib: true /lib64/libz.so.1zstd : falsebzip2: true /lib64/libbz2.so.1openssl: true /lib64/libcrypto.soISA-L: false libhadoop was built without ISA-L supportPMDK: false The native code was built without PMDK support.2023-02-18 16:58:39,764 INFO util.ExitUtil: Exiting with status 1: ExitException能够看到上方还有很多false的，不过这不影响hadoop的应用。 如果要解决的话，能够装置这些依赖，而后从新编译hadoop。
3、参考文章
1、https://www.vvave.net/archives/how-to-build-hadoop-334-native-libraries-full-kit-on-amd64.html
2、https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions