关于数据库:前端响应超时APIserver-服务内存不足碰见这类-DolphinScheduler-资源中心相关问题怎么办

4次阅读

共计 21465 个字符，预计需要花费 54 分钟才能阅读完成。

作者 | 刘森卡特加特大数据工程师 
 最近，有些用户小伙伴反映在应用 Apache DolphinScheduler 资源核心时会遇到问题，社区小伙伴整顿了一些常见问题，心愿帮大家解决当务之急。
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>[WARN] 2023-04-25 03:02:58.463 +0000 org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver:[434] - Failure in @ExceptionHandler org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler#exceptionHandler(Exception, HandlerMethod) java.lang.IllegalStateException: Could not resolve parameter [1] in public org.apache.dolphinscheduler.api.utils.Result org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler.exceptionHandler(java.lang.Exception,org.springframework.web.method.HandlerMethod): No suitable resolver at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:176) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:146) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428) at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75) at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:142) at org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80) at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1330) at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1141) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1087) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:517) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)</pre>
 问题起因：前端响应超时 
 解决办法：
 须要批改以下 4 个文件。
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>/api-server/ui/assets/service.33119ae9.js /api-server/ui/assets/service.33119ae9.js.gz

/ui/assets/service.33119ae9.js /ui/assets/service.33119ae9.js.gz</pre>
 并找到以下配置，而后将 15e3 批改为 15e5 或更高版本。
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>baseURL:"/dolphinscheduler",timeout:15e3</pre>
1. 这里 service.33119ae9. 并不是固定的名称，只有找到 service..js* 的文件批改就行 
2. 如果应用 docker 部署的 dolphinscheduler。是没有 /ui/assets/service.33119ae9.js 和 /ui/assets/service.33119ae9.js.gz 文件，疏忽就行 
3..gz 文件应用 vim 就能够关上 
4. 重启 api 服务 
5. 革除浏览器缓存 
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>[WARN] 2023-04-25 03:05:50.855 +0000 org.eclipse.jetty.server.HttpChannel:[776] - /dolphinscheduler/resources org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Failed to parse multipart servlet request; nested exception is java.lang.OutOfMemoryError: Java heap space at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:517) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)     at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)     at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)     at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)     at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)     at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)     at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)     at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)     at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)</pre>
 问题起因：
api-server 服务内存不足 
 解决办法：
 更改 bin/dolphinscheduler-daemon.sh 的 api-server 项内存大小参数 
Docker 部署解决办法：
1. 进入 Docker 实例 
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>docker exec -it bbd60fe717af /bin/bash</pre>
2. 批改 /opt/dolphinscheduler/bin/start.sh
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>vim /opt/dolphinscheduler/bin/start.sh</pre>
3. 内存参数从‘-Xms1g -Xmx1g -Xmn512m’批改为‘-Xms2g -Xmx2g -Xmn1g’。具体数值依据须要资源批改 
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>#!/bin/bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. #


BIN_DIR=$(dirname&nbsp;<span class="hljs-variable" style="color: #d19a66; line-height: 26px;">$0</span>)
DOLPHINSCHEDULER_HOME=<span class="hljs-variable" style="color: #d19a66; line-height: 26px;">${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/..;&nbsp;pwd)}</span><br><br><span class="hljs-built_in" style="color: #e6c07b; line-height: 26px;">source</span>&nbsp;<span class="hljs-string" style="color: #98c379; line-height: 26px;">"<span class="hljs-variable"style="color: #d19a66; line-height: 26px;">$DOLPHINSCHEDULER_HOME</span>/conf/dolphinscheduler_env.sh"</span>
JAVA_OPTS=<span class="hljs-variable" style="color: #d19a66; line-height: 26px;">${JAVA_OPTS:-"-server&nbsp;-Duser.timezone=${SPRING_JACKSON_TIME_ZONE}</span> -Xms2g -Xmx2g -Xmn1g -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof<span class="hljs-string"style="color: #98c379; line-height: 26px;">"}
if [["</span><span class="hljs-variable"style="color: #d19a66; line-height: 26px;">$DOCKER</span><span class="hljs-string"style="color: #98c379; line-height: 26px;">"&nbsp;==&nbsp;"</span><span class="hljs-literal"style="color: #56b6c2; line-height: 26px;">true</span><span class="hljs-string"style="color: #98c379; line-height: 26px;">"&nbsp;]];&nbsp;then<br>&nbsp;&nbsp;JAVA_OPTS="</span><span class="hljs-variable"style="color: #d19a66; line-height: 26px;">${JAVA_OPTS}</span> -XX:-UseContainerSupport<span class="hljs-string"style="color: #98c379; line-height: 26px;">"
fi

$JAVA_HOME/bin/java $JAVA_OPTS \ -cp "$DOLPHINSCHEDULER_HOME/conf":"$DOLPHINSCHEDULER_HOME/libs/*" \ org.apache.dolphinscheduler.api.ApiApplicationServer </pre>
3.12 BUG
<ul data-tool=”mdnice 编辑器 ” style=”margin-top: 8px; margin-bottom: 8px; padding-left: 25px; color: black; list-style-type: disc;”>
<li><section style=”margin-top: 5px; margin-bottom: 5px; line-height: 26px; text-align: left; color: rgb(1,1,1); font-weight: 500;”> 修复工作组可能被多个工作抢走的问题 #13115</section></li></ul>
 如果设置工作组，多个工作会抢占工作组不开释，导致 master 始终重试开释，无奈继续执行工作 
 解决办法：
1. 降级到 3.13
2. 勾销工作组 
<pre class=”custom” data-tool=”mdnice 编辑器 ” style=”margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;”><span style=”display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;”>[INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-357 - Begin to handle state event, TaskStateEvent(processInstanceId=357, taskInstanceId=4553, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=357-4553, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-357 - Handle task instance wait task group event, taskInstanceId: 4553 [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-356 - Begin to handle state event, TaskStateEvent(processInstanceId=356, taskInstanceId=4554, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=356-4554, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-356 - Handle task instance wait task group event, taskInstanceId: 4554 [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-360 - Begin to handle state event, TaskStateEvent(processInstanceId=360, taskInstanceId=4613, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=360-4613, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-360 - Handle task instance wait task group event, taskInstanceId: 4613 [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-355 - Begin to handle state event, TaskStateEvent(processInstanceId=355, taskInstanceId=4549, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=355-4549, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-355 - Handle task instance wait task group event, taskInstanceId: 4549</pre>
</section>