聊聊flink的logback配置

序本文主要研究一下flink的logback配置client端pom文件配置<dependencies> <!– Add the two required logback dependencies –> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-core</artifactId> <version>1.2.3</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> <!– Add the log4j -> sfl4j (-> logback) bridge into the classpath Hadoop is logging to log4j! –> <dependency> <groupId>org.slf4j</groupId> <artifactId>log4j-over-slf4j</artifactId> <version>1.7.15</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.7.1</version> <exclusions> <exclusion> <groupId>log4j</groupId> <artifactId></artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.7.1</version> <exclusions> <exclusion> <groupId>log4j</groupId> <artifactId></artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.7.1</version> <exclusions> <exclusion> <groupId>log4j</groupId> <artifactId></artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency></dependencies>添加logback-core、logback-classic及log4j-over-slf4j依赖，之后对flink-java、flink-streaming-java_2.11、flink-clients_2.11等配置log4j及slf4j-log4j12的exclusions；最后通过mvn dependency:tree查看是否还有log4j12，以确认下是否都全部排除了服务端配置添加logback-classic.jar、logback-core.jar、log4j-over-slf4j.jar到flink的lib目录下(比如/opt/flink/lib)移除flink的lib目录下(比如/opt/flink/lib)log4j及slf4j-log4j12的jar(比如log4j-1.2.17.jar及slf4j-log4j12-1.7.15.jar)如果要自定义logback的配置的话，可以覆盖flink的conf目录下的logback.xml、logback-console.xml或者logback-yarn.xmlflink-daemon.shflink-release-1.7.1/flink-dist/src/main/flink-bin/bin/flink-daemon.sh#!/usr/bin/env bash################################################################################# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements. See the NOTICE file# distributed with this work for additional information# regarding copyright ownership. The ASF licenses this file# to you under the Apache License, Version 2.0 (the# “License”); you may not use this file except in compliance# with the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an “AS IS” BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.################################################################################# Start/stop a Flink daemon.USAGE=“Usage: flink-daemon.sh (start|stop|stop-all) (taskexecutor|zookeeper|historyserver|standalonesession|standalonejob) [args]“STARTSTOP=$1DAEMON=$2ARGS=("${@:3}”) # get remaining arguments as arraybin=dirname "$0"bin=cd "$bin"; pwd. “$bin”/config.shcase $DAEMON in (taskexecutor) CLASS_TO_RUN=org.apache.flink.runtime.taskexecutor.TaskManagerRunner ;; (zookeeper) CLASS_TO_RUN=org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer ;; (historyserver) CLASS_TO_RUN=org.apache.flink.runtime.webmonitor.history.HistoryServer ;; (standalonesession) CLASS_TO_RUN=org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint ;; (standalonejob) CLASS_TO_RUN=org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint ;; () echo “Unknown daemon ‘${DAEMON}’. $USAGE.” exit 1 ;;esacif [ “$FLINK_IDENT_STRING” = "” ]; then FLINK_IDENT_STRING="$USER"fiFLINK_TM_CLASSPATH=constructFlinkClassPathpid=$FLINK_PID_DIR/flink-$FLINK_IDENT_STRING-$DAEMON.pidmkdir -p “$FLINK_PID_DIR”# Log files for daemons are indexed from the process ID’s position in the PID# file. The following lock prevents a race condition during daemon startup# when multiple daemons read, index, and write to the PID file concurrently.# The lock is created on the PID directory since a lock file cannot be safely# removed. The daemon is started with the lock closed and the lock remains# active in this script until the script exits.command -v flock >/dev/null 2>&1if [[ $? -eq 0 ]]; then exec 200<"$FLINK_PID_DIR" flock 200fi# Ascending ID depending on number of lines in pid file.# This allows us to start multiple daemon of each type.id=$([ -f “$pid” ] && echo $(wc -l < “$pid”) || echo “0”)FLINK_LOG_PREFIX="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-${DAEMON}-${id}-${HOSTNAME}“log="${FLINK_LOG_PREFIX}.log"out="${FLINK_LOG_PREFIX}.out"log_setting=("-Dlog.file=${log}” “-Dlog4j.configuration=file:${FLINK_CONF_DIR}/log4j.properties” “-Dlogback.configurationFile=file:${FLINK_CONF_DIR}/logback.xml”)JAVA_VERSION=$(${JAVA_RUN} -version 2>&1 | sed ’s/.version “(.).(.)..”/\1\2/; 1q’)# Only set JVM 8 arguments if we have correctly extracted the versionif [[ ${JAVA_VERSION} =~ ${IS_NUMBER} ]]; then if [ “$JAVA_VERSION” -lt 18 ]; then JVM_ARGS="$JVM_ARGS -XX:MaxPermSize=256m" fificase $STARTSTOP in (start) # Rotate log files rotateLogFilesWithPrefix “$FLINK_LOG_DIR” “$FLINK_LOG_PREFIX” # Print a warning if daemons are already running on host if [ -f “$pid” ]; then active=() while IFS=’’ read -r p || [[ -n “$p” ]]; do kill -0 $p >/dev/null 2>&1 if [ $? -eq 0 ]; then active+=($p) fi done < “${pid}” count="${#active[@]}" if [ ${count} -gt 0 ]; then echo “[INFO] $count instance(s) of $DAEMON are already running on $HOSTNAME.” fi fi # Evaluate user options for local variable expansion FLINK_ENV_JAVA_OPTS=$(eval echo ${FLINK_ENV_JAVA_OPTS}) echo “Starting $DAEMON daemon on host $HOSTNAME.” $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} “${log_setting[@]}” -classpath “manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"” ${CLASS_TO_RUN} “${ARGS[@]}” > “$out” 200<&- 2>&1 < /dev/null & mypid=$! # Add to pid file if successful start if [[ ${mypid} =~ ${IS_NUMBER} ]] && kill -0 $mypid > /dev/null 2>&1 ; then echo $mypid >> “$pid” else echo “Error starting $DAEMON daemon.” exit 1 fi ;; (stop) if [ -f “$pid” ]; then # Remove last in pid file to_stop=$(tail -n 1 “$pid”) if [ -z $to_stop ]; then rm “$pid” # If all stopped, clean up pid file echo “No $DAEMON daemon to stop on host $HOSTNAME.” else sed $d “$pid” > “$pid.tmp” # all but last line # If all stopped, clean up pid file [ $(wc -l < “$pid.tmp”) -eq 0 ] && rm “$pid” “$pid.tmp” || mv “$pid.tmp” “$pid” if kill -0 $to_stop > /dev/null 2>&1; then echo “Stopping $DAEMON daemon (pid: $to_stop) on host $HOSTNAME.” kill $to_stop else echo “No $DAEMON daemon (pid: $to_stop) is running anymore on $HOSTNAME.” fi fi else echo “No $DAEMON daemon to stop on host $HOSTNAME.” fi ;; (stop-all) if [ -f “$pid” ]; then mv “$pid” “${pid}.tmp” while read to_stop; do if kill -0 $to_stop > /dev/null 2>&1; then echo “Stopping $DAEMON daemon (pid: $to_stop) on host $HOSTNAME.” kill $to_stop else echo “Skipping $DAEMON daemon (pid: $to_stop), because it is not running anymore on $HOSTNAME.” fi done < “${pid}.tmp” rm “${pid}.tmp” fi ;; () echo “Unexpected argument ‘$STARTSTOP’. $USAGE.” exit 1 ;;esac使用flink-daemon.sh启动的flink使用的logback配置文件是logback.xmlflink-console.shflink-release-1.7.1/flink-dist/src/main/flink-bin/bin/flink-console.sh#!/usr/bin/env bash################################################################################# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements. See the NOTICE file# distributed with this work for additional information# regarding copyright ownership. The ASF licenses this file# to you under the Apache License, Version 2.0 (the# “License”); you may not use this file except in compliance# with the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an “AS IS” BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.################################################################################# Start a Flink service as a console application. Must be stopped with Ctrl-C# or with SIGTERM by kill or the controlling process.USAGE=“Usage: flink-console.sh (taskexecutor|zookeeper|historyserver|standalonesession|standalonejob) [args]“SERVICE=$1ARGS=("${@:2}”) # get remaining arguments as arraybin=dirname "$0"bin=cd "$bin"; pwd. “$bin”/config.shcase $SERVICE in (taskexecutor) CLASS_TO_RUN=org.apache.flink.runtime.taskexecutor.TaskManagerRunner ;; (historyserver) CLASS_TO_RUN=org.apache.flink.runtime.webmonitor.history.HistoryServer ;; (zookeeper) CLASS_TO_RUN=org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer ;; (standalonesession) CLASS_TO_RUN=org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint ;; (standalonejob) CLASS_TO_RUN=org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint ;; () echo “Unknown service ‘${SERVICE}’. $USAGE.” exit 1 ;;esacFLINK_TM_CLASSPATH=constructFlinkClassPathlog_setting=("-Dlog4j.configuration=file:${FLINK_CONF_DIR}/log4j-console.properties” “-Dlogback.configurationFile=file:${FLINK_CONF_DIR}/logback-console.xml”)JAVA_VERSION=$(${JAVA_RUN} -version 2>&1 | sed ’s/.version “(.).(.)..”/\1\2/; 1q’)# Only set JVM 8 arguments if we have correctly extracted the versionif [[ ${JAVA_VERSION} =~ ${IS_NUMBER} ]]; then if [ “$JAVA_VERSION” -lt 18 ]; then JVM_ARGS="$JVM_ARGS -XX:MaxPermSize=256m" fifiecho “Starting $SERVICE as a console application on host $HOSTNAME.“exec $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} “${log_setting[@]}” -classpath “manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"” ${CLASS_TO_RUN} “${ARGS[@]}“使用flink-console.sh启动的flink使用的logback配置文件是logback-console.xmlyarn-session.shflink-release-1.7.1/flink-dist/src/main/flink-bin/yarn-bin/yarn-session.sh#!/usr/bin/env bash################################################################################# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements. See the NOTICE file# distributed with this work for additional information# regarding copyright ownership. The ASF licenses this file# to you under the Apache License, Version 2.0 (the# “License”); you may not use this file except in compliance# with the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an “AS IS” BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.################################################################################bin=dirname "$0"bin=cd "$bin"; pwd# get Flink config. “$bin”/config.shif [ “$FLINK_IDENT_STRING” = "” ]; then FLINK_IDENT_STRING="$USER"fiJVM_ARGS="$JVM_ARGS -Xmx512m"CC_CLASSPATH=manglePathList $(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPATHSlog=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-yarn-session-$HOSTNAME.loglog_setting="-Dlog.file="$log” -Dlog4j.configuration=file:"$FLINK_CONF_DIR”/log4j-yarn-session.properties -Dlogback.configurationFile=file:"$FLINK_CONF_DIR”/logback-yarn.xml"export FLINK_CONF_DIR$JAVA_RUN $JVM_ARGS -classpath “$CC_CLASSPATH” $log_setting org.apache.flink.yarn.cli.FlinkYarnSessionCli -j “$FLINK_LIB_DIR”/flink-dist*.jar “$@“使用yarn-session.sh启动的flink使用的logback配置文件是logback-yarn.xml小结client端使用logback的话，要在pom文件添加logback-core、logback-classic及log4j-over-slf4j依赖，之后对flink-java、flink-streaming-java_2.11、flink-clients_2.11等配置log4j及slf4j-log4j12的exclusions；最后通过mvn dependency:tree查看是否还有log4j12，以确认下是否都全部排除了服务端使用logback的话，要在添加logback-classic.jar、logback-core.jar、log4j-over-slf4j.jar到flink的lib目录下(比如/opt/flink/lib)；移除flink的lib目录下(比如/opt/flink/lib)log4j及slf4j-log4j12的jar(比如log4j-1.2.17.jar及slf4j-log4j12-1.7.15.jar)；如果要自定义logback的配置的话，可以覆盖flink的conf目录下的logback.xml、logback-console.xml或者logback-yarn.xml使用flink-daemon.sh启动的flink使用的logback配置文件是logback.xml；使用flink-console.sh启动的flink使用的logback配置文件是logback-console.xml；使用yarn-session.sh启动的flink使用的logback配置文件是logback-yarn.xmldocUsing Logback instead of Log4j