LB
Lin's Blog
Blog

本文档基于实际部署过程整理，涵盖 Elasticsearch、Kibana、Logstash、Docker 容器日志采集、本机 /var/log 采集，以及 Kibana Data View 与常见问题。

1. 架构说明

《搭建ELK日志系统,完整的部署流程,Agent接入,日志配置,索引等等》文章正文配图 — STARBUCKET BLOG

组件 建议端口 作用
Elasticsearch 9200 / 9300 存储与检索
Kibana 5601 可视化查询
Logstash 5044（Beats 输入） 解析并写入 ES
filebeat — 采集 Docker 容器日志
filebeat-host — 采集本机 /var/log

版本建议：Elastic Stack 8.19.x（各组件大版本保持一致）。

2. 系统准备（Ubuntu 24.04）

在 ELK 服务器执行：

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget gnupg apt-transport-https ca-certificates

# 关闭 swap（ES 建议）
sudo swapoff -a
# 永久关闭：注释 /etc/fstab 中的 swap 行

# 内核参数
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elasticsearch.conf
sudo sysctl --system

# 文件描述符
echo '* soft nofile 65536' | sudo tee -a /etc/security/limits.conf
echo '* hard nofile 65536' | sudo tee -a /etc/security/limits.conf

添加 Elastic 8.x 仓库：

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt update

防火墙 / 安全组放行：

端口 用途
9200 Elasticsearch HTTP
5601 Kibana
5044 Logstash Beats 输入（业务机必须能访问） 
sudo ufw allow 9200/tcp
sudo ufw allow 5601/tcp
sudo ufw allow 5044/tcp

3. 安装 Elasticsearch

sudo apt install -y elasticsearch

首次安装请保存输出的 elastic 密码。丢失时可重置：

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

编辑 /etc/elasticsearch/elasticsearch.yml（单机示例）：

cluster.name: devops-logs
node.name: es-node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node

# 单机必须用 single-node，且不要同时配置：
# cluster.initial_master_nodes   ← 与 single-node 互斥，保留会启动失败

堆内存（按机器内存调整，8G 机器建议 2G）：

echo '-Xms2g' | sudo tee /etc/elasticsearch/jvm.options.d/heap.options
echo '-Xmx2g' | sudo tee -a /etc/elasticsearch/jvm.options.d/heap.options

启动：

sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch --no-pager

curl -k -u elastic:你的密码 https://localhost:9200

常见启动失败

报错 处理
cluster.initial_master_nodes is not allowed when discovery.type is single-node 删除/注释 cluster.initial_master_nodes
max virtual memory areas 设置 vm.max_map_count=262144
内存不足 / OOM 减小 -Xms/-Xmx

详细日志：

sudo tail -n 100 /var/log/elasticsearch/*.log

4. 安装 Kibana

sudo apt install -y kibana

生成 enrollment token 并配置：

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
sudo /usr/share/kibana/bin/kibana-setup
# 粘贴 token

或手动编辑 /etc/kibana/kibana.yml：

server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "http://ELK服务器IP:5601"

sudo systemctl enable --now kibana

浏览器访问：http://ELK服务器IP:5601，使用 elastic 登录。

5. 安装 Logstash

sudo apt install -y logstash

sudo mkdir -p /etc/logstash/certs
sudo cp /etc/elasticsearch/certs/http_ca.crt /etc/logstash/certs/
sudo chown -R logstash:logstash /etc/logstash/certs

5.1 Pipeline 配置

创建 /etc/logstash/conf.d/beats-pipeline.conf：

input {
  beats {
    port => 5044
  }
}

filter {
  # ---------- 本机日志（filebeat-host）----------
  if [fields][log_source] == "host" {
    mutate {
      add_field => { "[fields][service]" => "host" }
    }
  }

  # ---------- 容器日志（filebeat）----------
  if ![fields][log_source] or [fields][log_source] != "host" {
    if ![fields][service] {
      mutate { add_field => { "[fields][service]" => "docker" } }
    }
    if [container][name] {
      mutate { replace => { "[fields][service]" => "%{[container][name]}" } }
    }
    mutate {
      add_field => { "[fields][log_source]" => "container" }
      gsub => [ "[fields][service]", "[^A-Za-z0-9_.-]", "_" ]
    }
  }

  # nginx（可选）
  if [fields][service] == "nginx" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status:int} %{NUMBER:bytes:int} "%{DATA:referrer}" "%{DATA:user_agent}"'
      }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
    }
  }

  # 应用 JSON（可选）
  if [fields][service] == "app" {
    json {
      source => "message"
      target => "app"
      skip_on_invalid_json => true
    }
  }
}

output {
  if [fields][log_source] == "host" {
    elasticsearch {
      hosts => ["https://localhost:9200"]
      user => "elastic"
      password => "你的密码"
      ssl_enabled => true
      ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
      index => "logs-host-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["https://localhost:9200"]
      user => "elastic"
      password => "你的密码"
      ssl_enabled => true
      ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
      index => "logs-docker-%{[fields][service]}-%{+YYYY.MM.dd}"
    }
  }
}

注意：索引名使用 %{[fields][service]} 时，Docker 日志必须保证有 service（可用 container.name 填充），否则会出现异常索引名。

启动：

sudo systemctl enable --now logstash
sudo journalctl -u logstash -f
ss -lntp | grep 5044

6. 业务机：采集 Docker 容器日志（filebeat）

目录建议：/opt/service/filebeat/

6.1 docker-compose.yml

services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.19.19   # 版本尽量和 ES 一致
    container_name: filebeat
    user: root
    restart: unless-stopped
    volumes:
      - ./config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./data:/usr/share/filebeat/data
      - ./logs:/usr/share/filebeat/logs
      # 宿主机日志挂进来（按实际路径改）
      - /var/log/nginx:/var/log/nginx:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro   # 采容器日志时需要

6.2 config/filebeat.yml

filebeat.inputs:
  - type: container
    enabled: true
    paths:
      - /var/lib/docker/containers/*/*.log
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"
      # 排除 Filebeat 自身，避免噪声
      - drop_event.when.or:
          - equals.container.name: "filebeat"
          - equals.container.name: "filebeat-host"
          - contains.container.image.name: "filebeat"
    fields:
      env: prod
      log_source: container
    fields_under_root: false

processors:
  - add_host_metadata: ~

output.logstash:
  hosts: ["ELK服务器IP:5044"]

logging.level: info
logging.to_stderr: true
logging.to_files: false

启动：

cd /opt/service/filebeat
chmod go-w config/filebeat.yml
docker-compose up -d
docker-compose logs -f --tail=50

成功标志：

Connection to ...5044) established
Harvester started for paths: [/var/lib/docker/containers/*/*.log]

6.3 Kibana 查询容器日志

container.name: "opms"
container.name: "opms" and message: "error"

表格建议只展示：@timestamp、container.name、message。

日志正文在字段 message 中。

7. 业务机：采集本机 /var/log（filebeat-host）

与容器采集分开，使用独立收集器与独立索引。

目录建议：/opt/service/filebeat-host/

7.1 docker-compose.yml

services:
  filebeat-host:
    image: docker.elastic.co/beats/filebeat:8.19.19
    container_name: filebeat-host
    user: root
    restart: unless-stopped
    volumes:
        - ./config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
        - ./data:/usr/share/filebeat/data
        - ./logs:/usr/share/filebeat/logs
    volumes:
        - ./config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
        - ./data:/usr/share/filebeat/data
        - /var/log:/var/log:ro

7.2 config/filebeat.yml

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/syslog
      - /var/log/auth.log
      - /var/log/elk-host-test.log
      - /var/log/nginx/*.log
    ignore_older: 72h
    close_inactive: 1m
    # 仅测试新日志时可打开：
    # tail_files: true
    fields:
      env: prod
      log_source: host
      service: system
    fields_under_root: false

processors:
  - add_host_metadata: ~

output.logstash:
  hosts: ["ELK服务器IP:5044"]
  bulk_max_size: 2048
  timeout: 30s

logging.level: info
logging.to_stderr: true
logging.to_files: false

启动：

cd /opt/service/filebeat-host
chmod go-w config/filebeat.yml
docker-compose up -d
docker logs filebeat-host --tail=50

7.3 写入测试日志

echo "HOST-LOG-TEST $(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  | sudo tee -a /var/log/elk-host-test.log

验证：

# 业务机
docker logs filebeat-host --tail=30 2>&1 | grep -E 'Harvester|established|Failed|elk-host'

# ELK 机
curl -k -u elastic:密码 'https://localhost:9200/_cat/indices/logs-host*?v'
curl -k -u elastic:密码 'https://localhost:9200/logs-host-*/_search?q=HOST-LOG-TEST&pretty'

8. 测试用持续打日志容器（可选）

elk/log-generator/docker-compose.yml：

services:
  log-generator:
    image: python:3.12-alpine
    container_name: log-generator
    restart: unless-stopped
    command:
      - python
      - -u
      - -c
      - |
        import time
        from datetime import datetime, timezone
        i = 0
        while True:
            i += 1
            ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
            print(f"ELK-TEST heartbeat seq={i} ts={ts} service=log-generator", flush=True)
            time.sleep(2)

cd /opt/service/log-generator   # 或仓库 elk/log-generator
docker-compose up -d

Kibana 搜索：

container.name: "log-generator" and message: "ELK-TEST"

9. Kibana Data View

前提

必须先有匹配的索引，否则会报：

名称必须与一个或多个数据流、索引或索引别名匹配。

不需要为了建 Data View 而重装 Filebeat/Logstash；只要 ES 里已有对应索引即可。

查看现有索引：

curl -k -u elastic:密码 'https://localhost:9200/_cat/indices/logs*?v'

创建步骤

  1. Stack Management → Data Views → Create data view
用途 Name（显示名） Index pattern
容器日志 logs-docker logs-docker-*
本机日志 logs-host logs-host-*
兼容旧索引 logs logs-*
  1. Timestamp field：@timestamp
  2. 保存后到 Discover，左上角切换 Data View

Observability 菜单

若使用左侧「日志 / 探险者」：

  1. 进入 背景设定（Settings）
  2. 配置日志索引为 logs-docker-* 或 logs-host-*
  3. 再回到日志页面查看

常用搜索

# 容器
container.name: "opms"
message: "ELK-TEST"

# 本机
fields.log_source: "host"
message: "HOST-LOG-TEST"
log.file.path: "/var/log/syslog"

界面展示字段建议：@timestamp、container.name（或 log.file.path）、message。

10. 索引与生命周期（可选）

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "20gb", "max_age": "1d" }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

按天索引也可直接用 delete-by-age 的 ILM，或定期清理旧索引。

11. 验证清单

# ES
curl -k -u elastic:密码 https://localhost:9200/_cluster/health?pretty
curl -k -u elastic:密码 'https://localhost:9200/_cat/indices/logs*?v'

# Logstash 5044
ss -lntp | grep 5044
# 业务机
nc -vz -w 5 ELK服务器IP 5044

# Filebeat 连通
docker logs filebeat --tail=20 2>&1 | grep established
docker logs filebeat-host --tail=20 2>&1 | grep established
检查项 期望
ES 健康 green 或单机 yellow
5044 业务机 nc 成功
Filebeat Connection ... established
索引 存在 logs-docker-* / logs-host-*
Kibana Discover 能搜到测试关键字

12. 常见问题

现象 原因与处理
ES 启动失败：initial_master_nodes + single-node 删除 cluster.initial_master_nodes
Filebeat i/o timeout :5044 安全组/防火墙放行 5044；确认 Logstash 在监听
connection reset by peer Logstash 压力过大或写 ES 失败；缩小 /var/log 路径，加 ignore_older；查 journalctl -u logstash
采不到 Docker 日志 挂载 /var/lib/docker/containers 与 docker.sock
采不到本机日志 挂载 /var/log；paths 包含目标文件
Kibana 无数据 / 无法建 Data View 先确认 _cat/indices 有匹配索引
字段太多难读 表格只加 message 等必要列；drop filebeat 自身容器
Unable to load ingest pipelines 警告 走 Logstash 时可忽略

13. 安全建议

  • 勿将 elastic 密码提交到 Git 或发到公开聊天
  • 生产环境使用专用账号（非 elastic 超级用户）给 Logstash
  • 限制 9200/5601/5044 安全组来源 IP
  • 定期轮换密码：
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Comments & discussion

The first comment in each thread opens a topic. Signed-in readers can keep the conversation going under that topic.

No comments yet. Sign in to start a topic.

Start a new topic

Sign in to start a topic or join the discussion.