Debezium监控系统搭建

Kafka&Zookeeper监控

开启Zookeeper监控

修改zk-Server.sh文件

• JMXPORT端口

-Dcom.sun.management.jmxremote.port=$JMXPORT

• JMXAUTH

-Dcom.sun.management.jmxremote.authenticate=$JMXAUTH

• JMXSSL

-Dcom.sun.management.jmxremote.ssl=$JMXSSL

• JMXLOG4J

-Dzookeeper.jmx.log4j.disable=$JMXLOG4J

修改zkEnv.sh文件

JMXPORT=21811
JMXSSL=false
JMXAUTH=false
JMXLOG4J=false

开启Kafka监控

查看kafka-run-class.sh

• JMX_PORT

-Dcom.sun.management.jmxremote.port=$JMX_PORT

• KAFKA_JMX_OPTS

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

修改kafka-start-server.sh

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
    # export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

开启Kafka Connect JMX

• 下载jmx_prometheus_javaagent-0.3.1.jar

• 配置config.yml

startDelaySeconds: 0
ssl: false
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:19092/jmxrmi
lowercaseOutputName: false
lowercaseOutputLabelNames: false
rules:
- pattern : "kafka.connect<type=connect-worker-metrics>([^:]+):"
  name: "kafka_connect_connect_worker_metrics_$1"
- pattern : "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([^:]+)"
  name: "kafka_connect_connect_metrics_$2"
  labels:
    client: "$1"
- pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^,]+), key=([^>]+)><>RowsScanned"
  name: "debezium_metrics_RowsScanned"
  labels:
    plugin: "$1"
    name: "$3"
    context: "$2"
    table: "$4"
- pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^>]+)>([^:]+)"
  name: "debezium_metrics_$4"
  labels:
    plugin: "$1"
    name: "$3"
    context: "$2"

修改connect-distributed.sh

# 修改KAFKA_HEAP_OPTS环境变量
  export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
# 添加JMX_PORT
 export JMX_PORT="19092"
# 添加KAFKA_OPTS
 export KAFKA_OPTS="-javaagent:/opt/cloudera/parcels/CDH/lib/kafka/monitor/jmx_prometheus_javaagent-0.3.1.jar=8080:/opt/cloudera/parcels/CDH/lib/kafka/monitor/config.yml"

配置监控大盘

配置prometheus

• 配置prometheus.yml文件

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'debezium'
    static_configs:
    #  export KAFKA_OPTS="-javaagent:/opt/cloudera/parcels/CDH/lib/kafka/monitor/jmx_prometheus_javaagent-0.3.1.jar=8080:/opt/cloudera/parcels/CDH/lib/kafka/monitor/config.yml"
      - targets: [这里填写的是kafkaconnect的ip地址和jmx暴露的web端口，如上就是8080]
      

配置grafana

• 导入dashboard uid

• MySQL dashboard uid:11523,https://grafana.com/grafana/dashboards/11523

• debezium dashboard uid:13546