我的项目简介
Exchangis是一个轻量级的、高扩展性的数据交换平台，反对对结构化及无结构化的异构数据源之间的数据传输，在应用层上具备数据权限管控、节点服务高可用和多租户资源隔离等业务个性，而在数据层上又具备传输架构多样化、模块插件化和组件低耦合等架构特点。
Exchangis的传输替换能力依赖于其底层聚合的传输引擎，其顶层对各类数据源定义对立的参数模型，每种传输引擎对参数模型进行映射配置，转化为引擎的输出模型。每聚合一种引擎，都将减少Exchangis一类个性，对某类引擎的个性强化，都是对Exchangis个性的欠缺。默认聚合以及强化Alibaba的DataX传输引擎。
外围特点
数据源治理
以绑定我的项目的形式共享本人的数据源；
设置数据源对外权限，控制数据的流入和流出。
多传输引擎反对
传输引擎可横向扩大；
以后版本残缺聚合了离线批量引擎DataX、局部聚合了大数据批量导数引擎SQOOP
近实时工作管控
疾速抓取传输工作日志以及传输速率等信息，实时敞开工作；
可依据带宽情况对工作进行动静限流
反对无结构化传输
DataX框架革新，独自构建二进制流快速通道，实用于无数据转换的纯数据同步场景。
工作状态自检
监控长时间运行的工作和状态异样工作，及时开释占用的资源并收回告警。
架构设计
环境筹备
根底软件装置
MySQL (5.5+) 必选，对应客户端能够选装, Linux服务上若装置mysql的客户端能够通过部署脚本疾速初始化数据库
JDK (1.8.0\_141) 必选
Maven (3.6.1+) 必选
SQOOP (1.4.6) 可选，如果想要SQOOP做传输引擎，能够装置SQOOP，SQOOP装置依赖Hive,Hadoop环境，这里就不开展来讲
Python (2.x) 可选，次要用于调度执行底层DataX的启动脚本，默认的形式是以Java子过程形式执行DataX，用户能够抉择以Python形式来做自定义的革新
mysql 数据库装置
[root@localhost ~]# mkdir mysql[root@localhost ~]# cd mysql[root@localhost mysql]# wget https://downloads.mysql.com/archives/get/p/23/file/mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar[root@localhost mysql]# tar xvf mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar[root@localhost mysql]# yum install ./*.rpm[root@localhost mysql]# systemctl start mysqld.service[root@localhost mysql]#[root@localhost mysql]# systemctl enable mysqld.service[root@localhost mysql]#[root@localhost mysql]#[root@localhost mysql]# sudo grep 'temporary password' /var/log/mysqld.log2021-10-25T06:57:46.569037Z 1 [Note] A temporary password is generated for root@localhost: (l5aFfIxfNuu[root@localhost mysql]#[root@localhost mysql]#[root@localhost mysql]#[root@localhost mysql]# mysql -u root -pEnter password:Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 3Server version: 5.7.35 MySQL Community Server (GPL)Copyright (c) 2000, 2021, Oracle and/or its affiliates.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Cby123..';Query OK, 0 rows affected (0.00 sec)mysql>mysql>mysql>mysql> use mysql;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedmysql>mysql>mysql> update user set host='%' where user ='root';Query OK, 1 row affected (0.00 sec)Rows matched: 1 Changed: 1 Warnings: 0mysql> set global validate_password_policy=0;Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_mixed_case_count=0;Query OK, 0 rows affected (0.01 sec)mysql> set global validate_password_number_count=3;Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_special_char_count=0;Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_length=3;Query OK, 0 rows affected (0.00 sec)mysql>jdk装置
[root@localhost ~]# mkdir jdk[root@localhost ~]# cd jdk[root@localhost jdk]#[root@localhost jdk]# tar xf jdk-8u141-linux-x64.tar.gz[root@localhost jdk]#[root@localhost jdk]# lltotal 181172drwxr-xr-x. 8 10 143 255 Jul 12 2017 jdk1.8.0_141-rw-r--r--. 1 root root 185516505 Jul 25 2017 jdk-8u141-linux-x64.tar.gz[root@localhost jdk]#[root@localhost jdk]# vim /etc/profile[root@localhost jdk]# tail -n 3 /etc/profileexport JAVA_HOME=/root/jdk/jdk1.8.0_141/export PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar[root@localhost jdk]#[root@localhost jdk]# source /etc/profile[root@localhost jdk]# java -versionjava version "1.8.0_141"Java(TM) SE Runtime Environment (build 1.8.0_141-b15)Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)[root@localhost jdk]#maven 装置
[root@localhost ~]#[root@localhost ~]# mkdir maven[root@localhost ~]# cd maven[root@localhost maven]# wget https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz[root@localhost maven]#[root@localhost maven]# tar xf apache-maven-3.6.1-bin.tar.gz[root@localhost maven]# lltotal 8924drwxr-xr-x. 6 root root 99 Oct 25 15:08 apache-maven-3.6.1-rw-r--r--. 1 root root 9136463 Sep 4 2019 apache-maven-3.6.1-bin.tar.gz[root@localhost maven]# vim /etc/profile[root@localhost maven]#[root@localhost maven]# tail -n 3 /etc/profileexport MAVEN_HOME=/root/maven/apache-maven-3.6.1export PATH=$MAVEN_HOME/bin:$PATH:$HOME/bin[root@localhost maven]#[root@localhost maven]# source /etc/profile[root@localhost maven]# mvn -versionApache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)Maven home: /root/maven/apache-maven-3.6.1Java version: 1.8.0_141, vendor: Oracle Corporation, runtime: /root/jdk/jdk1.8.0_141/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "3.10.0-1160.31.1.el7.x86_64", arch: "amd64", family: "unix"[root@localhost maven]#Exchangis装置
[root@localhost ~]#[root@localhost ~]# mkdir Exchangis[root@localhost ~]# cd Exchangis[root@localhost Exchangis]# wget https://github.com/WeBankFinTech/Exchangis/releases/download/release-0.5.0/wedatasphere-exchangis-0.5.0.RELEASE.tar.gz[root@localhost Exchangis]# lltotal 552904-rw-r--r--. 1 root root 566172217 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE.tar.gz[root@localhost Exchangis]# tar xf wedatasphere-exchangis-0.5.0.RELEASE.tar.gz[root@localhost Exchangis]# lltotal 552904drwxr-xr-x. 6 root root 91 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE-rw-r--r--. 1 root root 566172217 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE.tar.gz[root@localhost Exchangis]# cd wedatasphere-exchangis-0.5.0.RELEASE[root@localhost wedatasphere-exchangis-0.5.0.RELEASE]# lltotal 20drwxrwxrwx. 2 root root 120 Oct 29 2020 bindrwxrwxrwx. 4 root root 32 May 12 2020 docsdrwxrwxrwx. 4 root root 57 May 12 2020 images-rwxrwxrwx. 1 root root 11357 Oct 29 2020 LICENSEdrwxr-xr-x. 2 root root 198 Oct 25 15:14 packages-rwxrwxrwx. 1 root root 4582 Oct 29 2020 README.md[root@localhost wedatasphere-exchangis-0.5.0.RELEASE]# cd bin/[root@localhost bin]# ./install.sh2021-10-25 15:16:19.723 [INFO] (12476) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/../modules].2021-10-25 15:16:19.728 [INFO] (12476) ####### Start To Uncompress Packages ######2021-10-25 15:16:19.730 [INFO] (12476) Uncompressing....Do you want to decompress this package: [exchangis-eureka_0.5.0.RELEASE_1.tar.gz]? (Y/N)y2021-10-25 15:16:22.691 [INFO] (12476) Uncompress package: [exchangis-eureka_0.5.0.RELEASE_1.tar.gz] to modules directoryDo you want to decompress this package: [exchangis-executor_0.5.0.RELEASE_1.tar.gz]? (Y/N)y2021-10-25 15:16:24.798 [INFO] (12476) Uncompress package: [exchangis-executor_0.5.0.RELEASE_1.tar.gz] to modules directoryDo you want to decompress this package: [exchangis-gateway_0.5.0.RELEASE_1.tar.gz]? (Y/N)y2021-10-25 15:16:31.947 [INFO] (12476) Uncompress package: [exchangis-gateway_0.5.0.RELEASE_1.tar.gz] to modules directoryDo you want to decompress this package: [exchangis-service_0.5.0.RELEASE_1.tar.gz]? (Y/N)y2021-10-25 15:16:35.029 [INFO] (12476) Uncompress package: [exchangis-service_0.5.0.RELEASE_1.tar.gz] to modules directory2021-10-25 15:16:36.537 [INFO] (12476) ####### Finish To Umcompress Packages ######Scan modules directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/../modules] to find server under exchangis2021-10-25 15:16:36.542 [INFO] (12476) ####### Start To Install Modules ######2021-10-25 15:16:36.545 [INFO] (12476) Module servers could be installed: [exchangis-eureka] [exchangis-executor] [exchangis-gateway] [exchangis-service]Do you want to confiugre and install [exchangis-eureka]? (Y/N)y2021-10-25 15:16:37.676 [INFO] (12476) Install module server: [exchangis-eureka]2021-10-25 15:16:37.706 [INFO] (12527) Start to build directory2021-10-25 15:16:37.709 [INFO] (12527) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../logs].2021-10-25 15:16:37.779 [INFO] (12527) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../conf] has been exist2021-10-25 15:16:37.782 [INFO] (12527) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../data].Do you want to confiugre and install [exchangis-executor]? (Y/N)y2021-10-25 15:16:38.529 [INFO] (12476) Install module server: [exchangis-executor]2021-10-25 15:16:38.558 [INFO] (12565) Start to build directory2021-10-25 15:16:38.561 [INFO] (12565) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../logs].2021-10-25 15:16:38.596 [INFO] (12565) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../conf] has been exist2021-10-25 15:16:38.599 [INFO] (12565) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../data].Do you want to confiugre and install [exchangis-gateway]? (Y/N)y2021-10-25 15:16:39.291 [INFO] (12476) Install module server: [exchangis-gateway]2021-10-25 15:16:39.317 [INFO] (12603) Start to build directory2021-10-25 15:16:39.320 [INFO] (12603) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../logs].2021-10-25 15:16:39.354 [INFO] (12603) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../conf] has been exist2021-10-25 15:16:39.356 [INFO] (12603) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../data].Do you want to confiugre and install [exchangis-service]? (Y/N)y2021-10-25 15:16:39.991 [INFO] (12476) Install module server: [exchangis-service]2021-10-25 15:16:40.017 [INFO] (12641) Start to build directory2021-10-25 15:16:40.020 [INFO] (12641) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../logs].2021-10-25 15:16:40.056 [INFO] (12641) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../conf] has been exist2021-10-25 15:16:40.059 [INFO] (12641) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../data].2021-10-25 15:16:40.099 [INFO] (12641) Scan out mysql command, so begin to initalize the databaseDo you want to initalize database with sql: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/exchangis-init.sql]? (Y/N)yPlease input the db host(default: 127.0.0.1):Please input the db port(default: 3306):Please input the db username(default: root):Please input the db password(default: ): Cby123..Please input the db name(default: exchangis)mysql: [Warning] Using a password on the command line interface can be insecure.2021-10-25 15:16:55.665 [INFO] (12476) ####### Finish To Install Modules ######[root@localhost bin]#[root@localhost bin]# ./start-all.sh2021-10-25 15:18:22.181 [INFO] (12691) Try To Start Modules In Order2021-10-25 15:18:22.189 [INFO] (12699) ####### Begin To Start Module: [exchangis-eureka] ######2021-10-25 15:18:22.199 [INFO] (12707) load environment variables2021-10-25 15:18:22.717 [INFO] (12707) /root/jdk/jdk1.8.0_141//bin/java2021-10-25 15:18:22.721 [INFO] (12707) Waiting EXCHANGIS-EUREKA to start complete ...2021-10-25 15:18:22.994 [INFO] (12707) EXCHANGIS-EUREKA start success2021-10-25 15:18:23.003 [INFO] (13009) ####### Begin To Start Module: [exchangis-gateway] ######2021-10-25 15:18:23.012 [INFO] (13017) load environment variables2021-10-25 15:18:23.493 [INFO] (13017) /root/jdk/jdk1.8.0_141//bin/java2021-10-25 15:18:23.497 [INFO] (13017) Waiting EXCHANGIS-GATEWAY to start complete ...2021-10-25 15:18:24.081 [INFO] (13017) EXCHANGIS-GATEWAY start success2021-10-25 15:18:24.091 [INFO] (13321) ####### Begin To Start Module: [exchangis-service] ######2021-10-25 15:18:24.099 [INFO] (13329) load environment variables2021-10-25 15:18:24.933 [INFO] (13329) /root/jdk/jdk1.8.0_141//bin/java2021-10-25 15:18:24.936 [INFO] (13329) Waiting EXCHANGIS-SERVICE to start complete ...2021-10-25 15:18:26.398 [INFO] (13329) EXCHANGIS-SERVICE start success2021-10-25 15:18:26.410 [INFO] (13634) ####### Begin To Start Module: [exchangis-executor] ######2021-10-25 15:18:26.423 [INFO] (13643) load environment variables2021-10-25 15:18:27.677 [INFO] (13643) /root/jdk/jdk1.8.0_141//bin/java2021-10-25 15:18:27.681 [INFO] (13643) Waiting EXCHANGIS-EXECUTOR to start complete ...2021-10-25 15:18:28.441 [INFO] (13643) EXCHANGIS-EXECUTOR start success[root@localhost bin]#**登陆拜访
**
注册核心：http://192.168.1.161:8500/
拜访地址：http://192.168.1.161:9503/
账号：admin
明码：admin
Linux运维交换社区
Linux运维交换社区，互联网新闻以及技术交换。
43篇原创内容
公众号
本文应用 文章同步助手 同步