乐趣区

ambari-自定义服务

jiezi

Ambari 自定义服务学习

集成 kafka-manager 到 ambari

存入的地址:

cd /var/lib/ambari-server/resources/stacks/HDP/2.5/services

最后的文件结构 

KAFKA_MANAGER
├── configuration
│   └── kafka-manager-env.xml
├── metainfo.xml
├── package
│   ├── archive.zip
│   └── scripts
│       └── master.py
└── quicklinks
    └── quicklinks.json

1. 写 metainfo.xml

<?xml version="1.0"?>
<metainfo>
  <schemaVersion>2.0</schemaVersion>
  <services>
    <service>
      <!-- Internal name for service (must be unique) -->
      <name>KAFKA_MANAGER</name>
      <!-- display name in Ambari UI -->
      <displayName>KAFKA_MANAGER</displayName>
      <!-- Description of service - will be displayed when user clicks add service -->
      <comment>the kafka's manager</comment>
      <!-- Version of service -->
      <version>1.3.2.1</version>
      <components>
        <!-- In this case, there is only one master component -->
        <component>
          <name>KAKFA_MASTER</name>
          <displayName>KAFKA_MANAGER</displayName>
          <category>MASTER</category>
          <!-- how many of these components are allowed in a cluster -->
          <cardinality>1</cardinality>
          <!-- reference to (and details of) what script is to be used to install/stop/start/config the service -->
          <commandScript>
            <script>scripts/master.py</script>
            <scriptType>PYTHON</scriptType>
            <timeout>600</timeout>
          </commandScript>
        </component>
      </components>
      <!-- 需要安装的包，依赖的 操作系统 -->
      <osSpecifics>
        <osSpecific>
          <osFamily>any</osFamily>
          <packages></packages>
        </osSpecific>
      </osSpecifics>
      <!-- 配置文件 -->
      <configuration-dependencies>
        <config-type>kafka-manager-env</config-type>
      </configuration-dependencies>
      <!-- 修改配置后，是否需要重启服务 -->
      <restartRequiredAfterChange>false</restartRequiredAfterChange>
      <!-- 快速链接 -->
      <quickLinksConfigurations>
        <quickLinksConfiguration>
          <fileName>quicklinks.json</fileName>
          <default>true</default>
        </quickLinksConfiguration>
      </quickLinksConfigurations>
    </service>
  </services>
</metainfo>

2. 写脚本 master.py

#!/usr/bin/env python
from resource_management import *

class Master(Script):
    # 安装生命周期，本例子安装不做任何事情，使用安装好的包
    def install(self, env):
        #self.install_packages(env)
        print 'Install the KAFKA Master'
        
    # 停止动作，kill -9 pid，并删除 pid 文件
    def stop(self, env):
        Execute('ps -ef | grep kafka-manager | grep -v grep | awk  \'{print $2}\'| xargs kill -9')
        Execute('rm -f /opt/kafka-manager-1.3.2.1/RUNNING_PID')
        print 'Stop the KAFKA Master'
        
    # 启动动作，使用 shell 命令 nohup 后台启动  
    def start(self, env):
        Execute('nohup /opt/kafka-manager-1.3.2.1/bin/kafka-manager -Dconfig.file=/opt/kafka-manager-1.3.2.1/conf/application.conf -Dhttp.port=6888 > /opt/kafka-manager-1.3.2.1/kafka-manager.log 2>&1 &')

    # 检查状态脚本，使用 ambari 自带方法，参数给 pid 文件
    def status(self, env):
        check_process_status('/opt/kafka-manager-1.3.2.1/RUNNING_PID')
        
    # config 文件
    def configure(self, env):
        print 'Configure the KAFKA Master';

if __name__ == "__main__":
    Master().execute()

3. 写 quicklinks.json 文件 

{
  "name": "default",
  "description": "default quick links configuration",
  "configuration": {
    "protocol":
    {
      "type":"https",
      "checks":[
        {
          "property":"ssl_enable",
          "desired":"true",
          "site":"kafka-manager-env"
        }
      ]
    },

    "links": [
      {
        "name": "manager_ui",
        "label": "Kafka Manager",
        "requires_user_name": "false",
        "component_name": "KAKFA_MASTER",
        "url":"%@://%@:%@/",
        "port":{
          "http_property": "kafka.server.port",
          "http_default_port": "6888",
          "https_property": "kafka.server.port",
          "https_default_port": "6888",
          "regex": "^(\\d+)$",
          "site": "kafka-manager-env"
        }
      }
    ]
  }
}

4. 写配置文件 

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<configuration>
  <property>
    <name>kafka.server.port</name>
    <value>6888</value>
    <display-name>HTTP Post</display-name>
    <description>Webserver listens on this port</description>
  </property>
</configuration>

5. 重启 ambari-server

ambari-server restart

6. 在 WEB 上 ADD SERVICE

可以看到 KAFKA_MANAGER 在我们的 service 列表了

点击下一步，添加 KAFKA_MANAGER SERVICE

并试着去点击 STOP，Restart All，Quicklinks 功能是否正常

验证没有问题

集成 Java 程序

Java 程序文件结构，lib 包太多不列出，把相关 jar 包全部放入 lib 包下，暂时只支持 Xml 文件格式 

javaSpringboot/
├── config
│   ├── application.yml
│   └── java-config.xml
├── lib
├── log
│   └── stdio.log
└── javaControl.py

其中 javaControl.py 是自己写的 python 脚本，用来方便启动和关闭 java 程序
python 脚本参考博客网站地址
博客作者写的还带一个程序挂了，有重启的功能，这里省略 

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import commands
import os
import sys
import subprocess

_programName = "springboot"  # 自定义程序名称
_programClassName = "com.sharing.App"  # java 程序 main Class
_pythonName = sys.argv[0]  # 获得脚本名称
_workPlace = os.getcwd() + os.sep  # 获得工作目录 / 当前目录
_pidFileName = "pid.txt"  # 程序 pid 写入文件


# 介绍脚本使用方法入口
def getProgramHelpIntroduce():
    print "请使用 %s help 查看，如何使用脚本操作程序" % _pythonName


# 脚本使用说明书
def getProgramHelpCommand():
    print "脚本提供三个命令如下："
    print "启动程序命令: %s start" % _pythonName
    print "关闭程序命令: %s stop" % _pythonName
    print "重启程序命令: %s restart" % _pythonName


# pid 文件是否存在
def pidIsExist():
    return os.path.exists(_workPlace + _pidFileName)


# 开启程序
def startProgram():
    if pidIsExist():
        print "程序正在运行，请先关闭程序！"
        getProgramHelpIntroduce()
    else:
        print "程序开始启动..."
        os.system(
            "exec -a %s java -Xmx1024m -cp config/:lib/* %s >> log/stdio.log 2>&1 & echo $! > %s &" % (_programName, _programClassName, _workPlace + _pidFileName))
        print "程序运行成功！"


# 关闭程序
def stopProgram():
    if not pidIsExist():
        print "程序没有运行！"
        os.system("rm -f %s" % _workPlace + _pidFileName)
        getProgramHelpIntroduce()
    else:
        pid = int(commands.getoutput("cat %s" % (_workPlace + _pidFileName)))
        os.system("kill %s" % pid)
        os.system("rm -f %s" % _workPlace + _pidFileName)
        print "程序关闭成功！"


if __name__ == "__main__":
    if len(sys.argv) == 2:
        args = sys.argv[1]
    else:
        args = raw_input("Please Enter Your Command:")

    logPath = _workPlace + "log"

    if not os.path.exists(logPath):
        os.mkdir(logPath)

    if args == "start":
        startProgram()
    elif args == "stop":
        stopProgram()
    elif args == "restart":
        stopProgram()
        startProgram()
    elif args == "help":
        getProgramHelpCommand()
    else:
        getProgramHelpIntroduce()

接下来做 ambari 集成 java，java-config 是从上面的 java 程序拷贝过来的
文件结构 

JAVA_TEST/
├── configuration
│   └── java-config.xml
├── metainfo.xml
├── package
│   └── scripts
│       ├── master.py
│       └── params.py

metainfo.xml

<?xml version="1.0"?>
<metainfo>
  <schemaVersion>2.0</schemaVersion>
  <services>
    <service>
      <!-- Internal name for service (must be unique) -->
      <name>JAVA_TEST</name>
      <!-- display name in Ambari UI -->
      <displayName>JAVA_TEST</displayName>
      <!-- Description of service - will be displayed when user clicks add service -->
      <comment>a java application test</comment>
      <!-- Version of service -->
      <version>1.0.0.test</version>
      <components>
        <!-- In this case, there is only one master component -->
        <component>
          <name>JAVA_MASTER</name>
          <displayName>JAVA_MAIN</displayName>
          <category>MASTER</category>
          <!-- how many of these components are allowed in a cluster -->
          <cardinality>1</cardinality>
          <!-- reference to (and details of) what script is to be used to install/stop/start/config the service -->
          <commandScript>
            <script>scripts/master.py</script>
            <scriptType>PYTHON</scriptType>
            <timeout>600</timeout>
          </commandScript>
        </component>
      </components>
      <osSpecifics>
        <osSpecific>
          <osFamily>any</osFamily>
          <packages></packages>
        </osSpecific>
      </osSpecifics>
      <configuration-dependencies>
        <config-type>java-config</config-type>
      </configuration-dependencies>
      <restartRequiredAfterChange>false</restartRequiredAfterChange>
      <quickLinksConfigurations>
        <quickLinksConfiguration>
          <fileName>quicklinks.json</fileName>
          <default>true</default>
        </quickLinksConfiguration>
      </quickLinksConfigurations>
    </service>
  </services>
</metainfo>

params.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from resource_management.libraries.script.script import Script

config = Script.get_config()

java_config = config['configurations']['java-config']

master.py

#!/usr/bin/env python
from resource_management import *
import os


class Master(Script):
    def install(self, env):
        self.install_packages(env)

    #To stop the service, use the linux service stop command and pipe output to log file
    def stop(self, env):
        os.chdir('/usr/local/share/applications/javaSpringboot')
        Execute('./javaControl.py stop')
       
    #To start the service, use the linux service start command and pipe output to log file
    def start(self, env):
        import params
        self.configure(env)
        os.chdir('/usr/local/share/applications/javaSpringboot')
        Execute('./javaControl.py start')

    #To get status of the, use the linux service status command
    def status(self, env):
        check_process_status('/usr/local/share/applications/javaSpringboot/pid.txt')
        
    def configure(self, env):
        import params
        env.set_params(params)
        XmlConfig("java-config",
                  conf_dir="/usr/local/share/applications/javaSpringboot/config",
                  configurations=params.java_config,
                  owner="root",
                  group="root",
                  mode=0644
                  )

if __name__ == "__main__":
    Master().execute()

集成 tomcat

ambari tomcat

集成中踩过的坑

1. ambari-server 重启会报错 数据库检查失败 

2017-10-16 11:21:52,736  INFO - ******************************* Check database started *******************************
2017-10-16 11:21:56,029  INFO - Checking for configs not mapped to any cluster
2017-10-16 11:21:56,048  INFO - Checking for configs selected more than once
2017-10-16 11:21:56,050  INFO - Checking for hosts without state
2017-10-16 11:21:56,051  INFO - Checking host component states count equals host component desired states count
2017-10-16 11:21:56,053  INFO - Checking services and their configs
2017-10-16 11:21:58,035 ERROR - Service(s): KAFKA_MANAGER, from cluster cx has no config(s) in serviceconfig table!

对于 ambari 配置表的思考
ambari Server 配置修改流程

2. quicklinks 无法显示

检查 metainfo.xml 是否正确配置快速链接，并检查 quicklinks.json 文件是否正确

ambari 不会报错，只是不显示 quicklinks

资源链接

  • ambari 自定义服务 步骤说明（一、二、三、四）CSDN
  • ambari 自定义服务文档
  • ambari 自定义服务相关脚本 github 源码
  • python2.7 入门教程 廖雪峰
  • ambari 服务配置相关
  • ambari 自定义神器
退出移动版