kafka日志收集

安装JDK

如果你有安装java11 版本的.. 可以换成java8 新版的java11 可能有兼容问题

dpkg --list | grep jdk
apt-get --purge remove openjdk-11-jdk 

安装java jdk 8

sudo apt-get update
dpkg -l | grep jdk
sudo apt-get purge openjdk-11-jre-headless
sudo apt-get install openjdk-8-jdk

查看java版本

java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

安装zookeeper和zookeeperd

zookeeperd是用来启动zookeeper的

sudo apt-get update
sudo apt-get install zookeeper
sudo apt-get install zookeeperd
sudo systemctl enable zookeeper
sudo systemctl start zookeeper

安装kafka

增加kafka用户

sudo useradd kafka
sudo mkdir /var/log/kafka
sudo chown kafka:nogroup /var/log/kafka -R

安装

sudo wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.1.0/kafka_2.12-2.1.0.tgz
tar -zxvf kafka_2.12-2.1.0.tgz -C /usr/local/kafka

启动kafka

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties > /var/log/kafka/kafka.log

设置systemctl脚本

暂时未验证成功

[Unit]
Description=High-available, distributed message broker
After=network.target
[Service]
User=kafka
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties > /var/log/kafka/kafka.log
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target

模拟消费端脚本

0.9版本以后用这条命令

/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic nginx_log --from-beginning