1、开发第一个 Spark 程序
1) 创立一个 Spark Context
2)加载数据
3) 把每一行宰割成单词
4) 转换成 pairs 并且计数
2、wordCount 程序
import org.apache.spark.{SparkConf, SparkContext}
/**
- @author
- @date 2020-05-11 20:19
- @version 1.0
*/
def main(args: Array[String]) {
val conf = new SparkConf().setAppName(“wordcount”)
val sc= new SparkContext(conf)
val input=sc.textFile(“/Users/navyliu/Downloads/spark/helloSpark.txt”)
val lines = input.flatMap(line=>line.split(” “))
val count = lines.map(word=>(word,1)).reduceByKey{case (x,y)=>x+y}
val output = count.saveAsTextFile(“/Users/navyliu/Downloads/spark/helloSparkRes”)
}
3、打包
配置 jar 包大数据培训
build
而后 build
4、启动集群
1) 启动 master
./sbin/start-master.sh
master 启动之后, 能够拜访上面地址:
http://localhost:8080/
2)启动 worker
./bin/spark-class
启动 work 要用到这个地址:
spark://deMacBook-Pro.local:7077
启动 work 的命令为:
./bin/spark-class org.apache.spark.deploy.worker.Worker spark://navydeMacBook-Pro.local:7077
查看过程
也能够在页面中看到 worker 过程
3)提交作业
./bin/spark-submit
提交命令
./bin/spark-submit –master spark://deMacBook-Pro.local:7077 –class CountWord untitled.jar
把生成的 jar 包拷到当前目录
提交作业的命令为:
./spark-2.4.5-bin-hadoop2.7/bin/spark-submit –master spark://deMacBook-Pro.local:7077 –class CountWord untitled.jar
后果为:
能够看到生成了文件目录 helloSparkRes
jobs 工作截图:
5、遇到的问题
如果报这种谬误:
dyld: lazy symbol binding failed: Symbol not found: ____chkstk_darwin
Referenced from: /private/var/folders/91/6g1y3wp163jbr7tgdrqgkjg80000gn/T/liblz4-java-8731604412047028366.dylib
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: ____chkstk_darwin
Referenced from: /private/var/folders/91/6g1y3wp163jbr7tgdrqgkjg80000gn/T/liblz4-java-8731604412047028366.dylib
Expected in: /usr/lib/libSystem.B.dylib
起因是 scala 版本与 spark 版本抵触，从新下载 spark，版本是 2.4.5(spark-2.4.5-bin-hadoop2.7)，对应的 scala 是 scala-2.12.11.
从新运行即可.
作者：归来少年 666