摘要:通过建设GaussDB(DWS)与MRS的连贯,反对数据仓库服务SQL on Hadoop,以表面形式实现Hive数据的快捷导入,满足大数据交融剖析的利用场景。本文分享自华为云社区《【云小课】EI第17课 大数据交融剖析:GaussDB(DWS)轻松导入MRS-Hive数据源》,原文作者:Hi,EI 。
大数据交融剖析时代,GaussDB(DWS)如需拜访MRS数据源,该如何实现?本期云小课带您开启MRS数据源之门,通过近程读取MRS集群Hive上的ORC数据表实现数据导入DWS。
筹备环境已创立DWS集群,需确保MRS和DWS集群在同一个区域、可用区、同一VPC子网内,确保集群网络互通。
根本流程本实际预计时长:1小时,根本流程如下:
1、创立MRS剖析集群(抉择Hive、Spark、Tez组件)。
2、通过将本地txt数据文件上传至OBS桶,再通过OBS桶导入Hive,并由txt存储表导入ORC存储表。
3、创立MRS数据源连贯。
4、创立内部服务器。
5、创立表面。
6、通过表面导入DWS本地表。
一、创立MRS剖析集群1、登录华为云控制台,抉择“EI企业智能 > MapReduce服务”,单击“购买集群”,抉择“自定义购买”,填写软件配置参数,单击“下一步”。
2、填写硬件配置参数,单击“下一步”。
3、填写高级配置参数如下表,单击“立刻购买”,期待约15分钟,集群创立胜利。
二、筹备MRS的ORC表数据源1、本地PC新建一个product_info.txt,并拷贝以下数据,保留到本地。
100,XHDK-A-1293-#fJ3,2017-09-01,A,2017 Autumn New Shirt Women,red,M,328,2017-09-04,715,good205,KDKE-B-9947-#kL5,2017-09-01,A,2017 Autumn New Knitwear Women,pink,L,584,2017-09-05,406,very good!300,JODL-X-1937-#pV7,2017-09-01,A,2017 autumn new T-shirt men,red,XL,1245,2017-09-03,502,Bad.310,QQPX-R-3956-#aD8,2017-09-02,B,2017 autumn new jacket women,red,L,411,2017-09-05,436,It's really super nice150,ABEF-C-1820-#mC6,2017-09-03,B,2017 Autumn New Jeans Women,blue,M,1223,2017-09-06,1200,The seller's packaging is exquisite200,BCQP-E-2365-#qE4,2017-09-04,B,2017 autumn new casual pants men,black,L,997,2017-09-10,301,The clothes are of good quality.250,EABE-D-1476-#oB1,2017-09-10,A,2017 autumn new dress women,black,S,841,2017-09-15,299,Follow the store for a long time.108,CDXK-F-1527-#pL2,2017-09-11,A,2017 autumn new dress women,red,M,85,2017-09-14,22,It's really amazing to buy450,MMCE-H-4728-#nP9,2017-09-11,A,2017 autumn new jacket women,white,M,114,2017-09-14,22,Open the package and the clothes have no odor260,OCDA-G-2817-#bD3,2017-09-12,B,2017 autumn new woolen coat women,red,L,2004,2017-09-15,826,Very favorite clothes980,ZKDS-J-5490-#cW4,2017-09-13,B,2017 Autumn New Women's Cotton Clothing,red,M,112,2017-09-16,219,The clothes are small98,FKQB-I-2564-#dA5,2017-09-15,B,2017 autumn new shoes men,green,M,4345,2017-09-18,5473,The clothes are thick and it's better this winter.150,DMQY-K-6579-#eS6,2017-09-21,A,2017 autumn new underwear men,yellow,37,2840,2017-09-25,5831,This price is very cost effective200,GKLW-l-2897-#wQ7,2017-09-22,A,2017 Autumn New Jeans Men,blue,39,5879,2017-09-25,7200,The clothes are very comfortable to wear300,HWEC-L-2531-#xP8,2017-09-23,A,2017 autumn new shoes women,brown,M,403,2017-09-26,607,good100,IQPD-M-3214-#yQ1,2017-09-24,B,2017 Autumn New Wide Leg Pants Women,black,M,3045,2017-09-27,5021,very good.350,LPEC-N-4572-#zX2,2017-09-25,B,2017 Autumn New Underwear Women,red,M,239,2017-09-28,407,The seller's service is very good110,NQAB-O-3768-#sM3,2017-09-26,B,2017 autumn new underwear women,red,S,6089,2017-09-29,7021,The color is very good 210,HWNB-P-7879-#tN4,2017-09-27,B,2017 autumn new underwear women,red,L,3201,2017-09-30,4059,I like it very much and the quality is good.230,JKHU-Q-8865-#uO5,2017-09-29,C,2017 Autumn New Clothes with Chiffon Shirt,black,M,2056,2017-10-02,3842,very good2、登录OBS控制台,单击“创立桶”,填写以下参数,单击“立刻创立”。
...