1、Filebeat概述
Filebeat是用于转发和集中日志数据的轻量级传送程序。作为服务器上的代理安装，Filebeat监视您指定的日志文件或位置，收集日志事件，并将其转发给[Elasticsearch]或 [Logstash]进行索引。
Filebeat的工作方式如下：启动Filebeat时，它将启动一个或多个输入，这些输入将在为日志数据指定的位置中查找。对于Filebeat所找到的每个日志，Filebeat都会启动收集器。每个收割机都读取单个日志以获
取新内容，并将新日志数据发送到libbeat，libbeat将聚集事件并将聚集的数据发送到为Filebeat配置的输出。
2、在Kubernetes上运行Filebeat
将Filebeat部署为DaemonSet，以确保集群的每个节点上都有一个正在运行的实例。Docker日志主机文件夹（/var/lib/docker/containers）安装在Filebeat容器上。Filebeat会开始输入文件，并在文件出现在
文件夹中后立即开始收集它们。
这里使用官方提供的方式部署：
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.5/deploy/kubernetes/filebeat-kubernetes.yaml
3、设置
默认情况下，Filebeat将事件发送到现有的Elasticsearch部署（如果存在）。要指定其他目标，请在清单文件中更改以下参数：
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: changeme
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
输出到logstash：
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
# Mounted `filebeat-inputs` configmap:
path: ${path.config}/inputs.d/*.yml
# Reload inputs configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
# To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# hints.enabled: true
processors:
- add_cloud_metadata:
#cloud.id: ${ELASTIC_CLOUD_ID}
#cloud.auth: ${ELASTIC_CLOUD_AUTH}
#output.elasticsearch:
#hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
#username: ${ELASTICSEARCH_USERNAME}
#password: ${ELASTICSEARCH_PASSWORD}
output.logstash:
hosts: ["192.168.0.104:5044"]
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: log #设置类型为log
paths:
- /var/lib/docker/containers/*/*.log
#fields:
#app: k8s
#type: docker-log
fields_under_root: true
json.keys_under_root: true
json.overwrite_keys: true
encoding: utf-8
fields.sourceType: docker-log #索引名格式
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:6.5.4 #提前准备好镜像，需要***下载
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat创建并运行：
能看到上面日志，则表示启动成功。
4、排错
如果没启动成功，查看logstash的日志，报错如下
[2019-12-20T19:53:14,049][ERROR][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dev-%{[fields][sourceType]}-2019-12-20", :_type=>"doc", :routing=>nil},
#<LogStash::Event:0x4c8737db>], :response=>{"index"=>{"_index"=>"dev-%{[fields][sourceType]}-2019-12-20", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name
[dev-%{[fields][sourceType]}-2019-12-20], must be lowercase", "index_uuid"=>"_na_", "index"=>"dev-%{[fields][sourceType]}-2019-12-20"}}}}
[2019-12-20T19:53:14,049][ERROR][logstash.outputs.原因是logstash中output的index不能有大写：
我原来的logstash的conf文件
output {
elasticsearch {
hosts => ["localhost:9200"]
index => '%{platform}-%{[fields][sourceType]}-%{+YYYY-MM-dd}'
template => "/opt/logstash-6.5.2/config/af-template.json"
template_overwrite => true
}
}修改后的
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "k8s-log-%{+YYYY.MM.dd}"
template => "/opt/logstash-6.5.2/config/af-template.json"
template_overwrite => true
}
}完美结束！无坑