我的项目简介
Exchangis 是一个轻量级的、高扩展性的数据交换平台，反对对结构化及无结构化的异构数据源之间的数据传输，在应用层上具备数据权限管控、节点服务高可用和多租户资源隔离等业务个性，而在数据层上又具备传输架构多样化、模块插件化和组件低耦合等架构特点。
Exchangis 的传输替换能力依赖于其底层聚合的传输引擎，其顶层对各类数据源定义对立的参数模型，每种传输引擎对参数模型进行映射配置，转化为引擎的输出模型。每聚合一种引擎，都将减少 Exchangis 一类个性，对某类引擎的个性强化，都是对 Exchangis 个性的欠缺。默认聚合以及强化 Alibaba 的 DataX 传输引擎。
外围特点
数据源治理
以绑定我的项目的形式共享本人的数据源；
设置数据源对外权限，控制数据的流入和流出。
多传输引擎反对
传输引擎可横向扩大；
以后版本残缺聚合了离线批量引擎 DataX、局部聚合了大数据批量导数引擎 SQOOP
近实时工作管控
疾速抓取传输工作日志以及传输速率等信息，实时敞开工作；
可依据带宽情况对工作进行动静限流
反对无结构化传输
DataX 框架革新，独自构建二进制流快速通道，实用于无数据转换的纯数据同步场景。
工作状态自检
监控长时间运行的工作和状态异样工作，及时开释占用的资源并收回告警。
架构设计
环境筹备
根底软件装置
MySQL (5.5+) 必选，对应客户端能够选装, Linux 服务上若装置 mysql 的客户端能够通过部署脚本疾速初始化数据库
JDK (1.8.0\_141) 必选
Maven (3.6.1+) 必选
SQOOP (1.4.6) 可选，如果想要 SQOOP 做传输引擎，能够装置 SQOOP，SQOOP 装置依赖 Hive,Hadoop 环境，这里就不开展来讲
Python (2.x) 可选，次要用于调度执行底层 DataX 的启动脚本，默认的形式是以 Java 子过程形式执行 DataX，用户能够抉择以 Python 形式来做自定义的革新
mysql 数据库装置
[root@localhost ~]# mkdir mysql
[root@localhost ~]# cd mysql
[root@localhost mysql]# wget https://downloads.mysql.com/archives/get/p/23/file/mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar
[root@localhost mysql]# tar xvf mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar
[root@localhost mysql]# yum install ./*.rpm
[root@localhost mysql]# systemctl start mysqld.service
[root@localhost mysql]#
[root@localhost mysql]# systemctl enable mysqld.service
[root@localhost mysql]#
[root@localhost mysql]#
[root@localhost mysql]# sudo grep 'temporary password' /var/log/mysqld.log
2021-10-25T06:57:46.569037Z 1 [Note] A temporary password is generated for root@localhost: (l5aFfIxfNuu
[root@localhost mysql]#
[root@localhost mysql]#
[root@localhost mysql]#
[root@localhost mysql]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.35 MySQL Community Server (GPL)
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Cby123..';
Query OK, 0 rows affected (0.00 sec)
mysql>
mysql>
mysql>
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql>
mysql>
mysql> update user set host='%' where user ='root';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_mixed_case_count=0;
Query OK, 0 rows affected (0.01 sec)
mysql> set global validate_password_number_count=3;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_special_char_count=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_length=3;
Query OK, 0 rows affected (0.00 sec)
mysql>jdk 装置
[root@localhost ~]# mkdir jdk
[root@localhost ~]# cd jdk
[root@localhost jdk]#
[root@localhost jdk]# tar xf jdk-8u141-linux-x64.tar.gz
[root@localhost jdk]#
[root@localhost jdk]# ll
total 181172
drwxr-xr-x. 8 10 143 255 Jul 12 2017 jdk1.8.0_141
-rw-r--r--. 1 root root 185516505 Jul 25 2017 jdk-8u141-linux-x64.tar.gz
[root@localhost jdk]#
[root@localhost jdk]# vim /etc/profile
[root@localhost jdk]# tail -n 3 /etc/profile
export JAVA_HOME=/root/jdk/jdk1.8.0_141/
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
[root@localhost jdk]#
[root@localhost jdk]# source /etc/profile
[root@localhost jdk]# java -version
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
[root@localhost jdk]#maven 装置
[root@localhost ~]#
[root@localhost ~]# mkdir maven
[root@localhost ~]# cd maven
[root@localhost maven]# wget https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz
[root@localhost maven]#
[root@localhost maven]# tar xf apache-maven-3.6.1-bin.tar.gz
[root@localhost maven]# ll
total 8924
drwxr-xr-x. 6 root root 99 Oct 25 15:08 apache-maven-3.6.1
-rw-r--r--. 1 root root 9136463 Sep 4 2019 apache-maven-3.6.1-bin.tar.gz
[root@localhost maven]# vim /etc/profile
[root@localhost maven]#
[root@localhost maven]# tail -n 3 /etc/profile
export MAVEN_HOME=/root/maven/apache-maven-3.6.1
export PATH=$MAVEN_HOME/bin:$PATH:$HOME/bin
[root@localhost maven]#
[root@localhost maven]# source /etc/profile
[root@localhost maven]# mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /root/maven/apache-maven-3.6.1
Java version: 1.8.0_141, vendor: Oracle Corporation, runtime: /root/jdk/jdk1.8.0_141/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.31.1.el7.x86_64", arch: "amd64", family: "unix"
[root@localhost maven]#Exchangis 装置
[root@localhost ~]#
[root@localhost ~]# mkdir Exchangis
[root@localhost ~]# cd Exchangis
[root@localhost Exchangis]# wget https://github.com/WeBankFinTech/Exchangis/releases/download/release-0.5.0/wedatasphere-exchangis-0.5.0.RELEASE.tar.gz
[root@localhost Exchangis]# ll
total 552904
-rw-r--r--. 1 root root 566172217 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE.tar.gz
[root@localhost Exchangis]# tar xf wedatasphere-exchangis-0.5.0.RELEASE.tar.gz
[root@localhost Exchangis]# ll
total 552904
drwxr-xr-x. 6 root root 91 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE
-rw-r--r--. 1 root root 566172217 Oct 25 15:14 wedatasphere-exchangis-0.5.0.RELEASE.tar.gz
[root@localhost Exchangis]# cd wedatasphere-exchangis-0.5.0.RELEASE
[root@localhost wedatasphere-exchangis-0.5.0.RELEASE]# ll
total 20
drwxrwxrwx. 2 root root 120 Oct 29 2020 bin
drwxrwxrwx. 4 root root 32 May 12 2020 docs
drwxrwxrwx. 4 root root 57 May 12 2020 images
-rwxrwxrwx. 1 root root 11357 Oct 29 2020 LICENSE
drwxr-xr-x. 2 root root 198 Oct 25 15:14 packages
-rwxrwxrwx. 1 root root 4582 Oct 29 2020 README.md
[root@localhost wedatasphere-exchangis-0.5.0.RELEASE]# cd bin/
[root@localhost bin]# ./install.sh
2021-10-25 15:16:19.723 [INFO] (12476) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/../modules].
2021-10-25 15:16:19.728 [INFO] (12476) ####### Start To Uncompress Packages ######
2021-10-25 15:16:19.730 [INFO] (12476) Uncompressing....
Do you want to decompress this package: [exchangis-eureka_0.5.0.RELEASE_1.tar.gz]? (Y/N)y
2021-10-25 15:16:22.691 [INFO] (12476) Uncompress package: [exchangis-eureka_0.5.0.RELEASE_1.tar.gz] to modules directory
Do you want to decompress this package: [exchangis-executor_0.5.0.RELEASE_1.tar.gz]? (Y/N)y
2021-10-25 15:16:24.798 [INFO] (12476) Uncompress package: [exchangis-executor_0.5.0.RELEASE_1.tar.gz] to modules directory
Do you want to decompress this package: [exchangis-gateway_0.5.0.RELEASE_1.tar.gz]? (Y/N)y
2021-10-25 15:16:31.947 [INFO] (12476) Uncompress package: [exchangis-gateway_0.5.0.RELEASE_1.tar.gz] to modules directory
Do you want to decompress this package: [exchangis-service_0.5.0.RELEASE_1.tar.gz]? (Y/N)y
2021-10-25 15:16:35.029 [INFO] (12476) Uncompress package: [exchangis-service_0.5.0.RELEASE_1.tar.gz] to modules directory
2021-10-25 15:16:36.537 [INFO] (12476) ####### Finish To Umcompress Packages ######
Scan modules directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/../modules] to find server under exchangis
2021-10-25 15:16:36.542 [INFO] (12476) ####### Start To Install Modules ######
2021-10-25 15:16:36.545 [INFO] (12476) Module servers could be installed:
[exchangis-eureka] [exchangis-executor] [exchangis-gateway] [exchangis-service]
Do you want to confiugre and install [exchangis-eureka]? (Y/N)y
2021-10-25 15:16:37.676 [INFO] (12476) Install module server: [exchangis-eureka]
2021-10-25 15:16:37.706 [INFO] (12527) Start to build directory
2021-10-25 15:16:37.709 [INFO] (12527) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../logs].
2021-10-25 15:16:37.779 [INFO] (12527) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../conf] has been exist
2021-10-25 15:16:37.782 [INFO] (12527) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-eureka/bin/../data].
Do you want to confiugre and install [exchangis-executor]? (Y/N)y
2021-10-25 15:16:38.529 [INFO] (12476) Install module server: [exchangis-executor]
2021-10-25 15:16:38.558 [INFO] (12565) Start to build directory
2021-10-25 15:16:38.561 [INFO] (12565) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../logs].
2021-10-25 15:16:38.596 [INFO] (12565) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../conf] has been exist
2021-10-25 15:16:38.599 [INFO] (12565) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-executor/bin/../data].
Do you want to confiugre and install [exchangis-gateway]? (Y/N)y
2021-10-25 15:16:39.291 [INFO] (12476) Install module server: [exchangis-gateway]
2021-10-25 15:16:39.317 [INFO] (12603) Start to build directory
2021-10-25 15:16:39.320 [INFO] (12603) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../logs].
2021-10-25 15:16:39.354 [INFO] (12603) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../conf] has been exist
2021-10-25 15:16:39.356 [INFO] (12603) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-gateway/bin/../data].
Do you want to confiugre and install [exchangis-service]? (Y/N)y
2021-10-25 15:16:39.991 [INFO] (12476) Install module server: [exchangis-service]
2021-10-25 15:16:40.017 [INFO] (12641) Start to build directory
2021-10-25 15:16:40.020 [INFO] (12641) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../logs].
2021-10-25 15:16:40.056 [INFO] (12641) Directory or file: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../conf] has been exist
2021-10-25 15:16:40.059 [INFO] (12641) Creating directory: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/modules/exchangis-service/bin/../data].
2021-10-25 15:16:40.099 [INFO] (12641) Scan out mysql command, so begin to initalize the database
Do you want to initalize database with sql: [/root/Exchangis/wedatasphere-exchangis-0.5.0.RELEASE/bin/exchangis-init.sql]? (Y/N)y
Please input the db host(default: 127.0.0.1):
Please input the db port(default: 3306):
Please input the db username(default: root):
Please input the db password(default:): Cby123..
Please input the db name(default: exchangis)
mysql: [Warning] Using a password on the command line interface can be insecure.
2021-10-25 15:16:55.665 [INFO] (12476) ####### Finish To Install Modules ######
[root@localhost bin]#
[root@localhost bin]# ./start-all.sh
2021-10-25 15:18:22.181 [INFO] (12691) Try To Start Modules In Order
2021-10-25 15:18:22.189 [INFO] (12699) ####### Begin To Start Module: [exchangis-eureka] ######
2021-10-25 15:18:22.199 [INFO] (12707) load environment variables
2021-10-25 15:18:22.717 [INFO] (12707) /root/jdk/jdk1.8.0_141//bin/java
2021-10-25 15:18:22.721 [INFO] (12707) Waiting EXCHANGIS-EUREKA to start complete ...
2021-10-25 15:18:22.994 [INFO] (12707) EXCHANGIS-EUREKA start success
2021-10-25 15:18:23.003 [INFO] (13009) ####### Begin To Start Module: [exchangis-gateway] ######
2021-10-25 15:18:23.012 [INFO] (13017) load environment variables
2021-10-25 15:18:23.493 [INFO] (13017) /root/jdk/jdk1.8.0_141//bin/java
2021-10-25 15:18:23.497 [INFO] (13017) Waiting EXCHANGIS-GATEWAY to start complete ...
2021-10-25 15:18:24.081 [INFO] (13017) EXCHANGIS-GATEWAY start success
2021-10-25 15:18:24.091 [INFO] (13321) ####### Begin To Start Module: [exchangis-service] ######
2021-10-25 15:18:24.099 [INFO] (13329) load environment variables
2021-10-25 15:18:24.933 [INFO] (13329) /root/jdk/jdk1.8.0_141//bin/java
2021-10-25 15:18:24.936 [INFO] (13329) Waiting EXCHANGIS-SERVICE to start complete ...
2021-10-25 15:18:26.398 [INFO] (13329) EXCHANGIS-SERVICE start success
2021-10-25 15:18:26.410 [INFO] (13634) ####### Begin To Start Module: [exchangis-executor] ######
2021-10-25 15:18:26.423 [INFO] (13643) load environment variables
2021-10-25 15:18:27.677 [INFO] (13643) /root/jdk/jdk1.8.0_141//bin/java
2021-10-25 15:18:27.681 [INFO] (13643) Waiting EXCHANGIS-EXECUTOR to start complete ...
2021-10-25 15:18:28.441 [INFO] (13643) EXCHANGIS-EXECUTOR start success
[root@localhost bin]#** 登陆拜访
**
注册核心：http://192.168.1.161:8500/
拜访地址：http://192.168.1.161:9503/
账号：admin
明码：admin
Linux 运维交换社区
Linux 运维交换社区，互联网新闻以及技术交换。
43 篇原创内容
公众号
本文应用 文章同步助手 同步