最终目的是：将K8S的Pod产生的日志收集到Elastic Search中，从而能通过搜索关键词快速定位问题。
部署ES和Kibana
参考https://www.elastic.co/guide/en/cloud-on-k8s/2.1/k8s-overview.html
先部署CRD，和Operator，用于操作cluster。
之后通过自己定义cluster，创建EScluster，Kibana服务。
Elasticsearch yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart-pvc
namespace: elastic-system
spec:
image: coreharbor.bdap.com/library/elasticsearch:8.2.0
nodeSets:
- config:
node.store.allow_mmap: false
count: 1
name: default
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
version: 8.2.0
|
Kibana yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
| apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: quickstart
namespace: elastic-system
spec:
config:
xpack.fleet.agentPolicies:
- id: new-eck-fleet-server
is_default_fleet_server: true
monitoring_enabled:
- logs
- metrics
name: fleet server policy
package_policies:
- id: fleet_server-1
name: fleet_server-1
package:
name: fleet_server
- id: container-log-eck-agent
is_default: true
monitoring_enabled:
- logs
- metrics
name: container log elastic agent
package_policies:
- id: elastic_agent-1
name: elastic_agent-1
package:
name: elastic_agent
unenroll_timeout: 900
xpack.fleet.agents.elasticsearch.hosts:
- <https://quickstart-pvc-es-http.elastic-system.svc:9200>
xpack.fleet.agents.fleet_server.hosts:
- <https://fleet-server-quickstart-agent-http.elastic-system.svc:8220>
xpack.fleet.packages:
- name: elastic_agent
version: latest
- name: fleet_server
version: latest
count: 1
elasticsearchRef:
name: quickstart-pvc
http:
tls:
selfSignedCertificate:
disabled: true
image: coreharbor.bdap.com/elasticsearch/kibana:8.2.0
version: 8.2.0
|
部署Elastic Agent
Agent的yaml配置，【注意】mount的路径配置，对应一个k8s中的daemonst，收集每个机器上的log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic-agent-quickstart
namespace: elastic-system
spec:
daemonSet:
podTemplate:
spec:
automountServiceAccountToken: true
containers:
- name: agent
volumeMounts:
- mountPath: "/home/docker/lib/docker/containers"
name: varlibdockercontainers
- mountPath: "/var/log/containers"
name: varlogcontainers
- mountPath: "/var/log/pods"
name: varlogpods
securityContext:
runAsUser: 0
serviceAccountName: elastic-agent
volumes:
- hostPath:
path: "/home/docker/lib/docker/containers"
name: varlibdockercontainers
- hostPath:
path: "/var/log/containers"
name: varlogcontainers
- hostPath:
path: "/var/log/pods"
name: varlogpods
fleetServerRef:
name: fleet-server-quickstart
image: elastic/elastic-agent:8.2.0
kibanaRef:
name: quickstart
mode: fleet
version: 8.2.0
|
这三个路径的关系：/var/log/containers/.log为软连接，指向/var/log/pods///.log软连接，再指向/var/lib/docker/containers//.log，为了让日志中包含pod名字，我们使用需要软连接，勾选Use Symlinks，且Kubernetes container log path为：
1
| /var/log/containers/*${kubernetes.container.id}.log
|
【注意】由于裸机上的软连接最终指向【已经修改过路径的】/home/docker/lib/docker/containers，因此，在container中也要mount到一样的路径下，而非默认的/var/lib。
Fleet Server yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server-quickstart
namespace: elastic-system
spec:
deployment:
podTemplate:
spec:
automountServiceAccountToken: true
securityContext:
runAsUser: 0
serviceAccountName: elastic-agent
replicas: 1
elasticsearchRefs:
- name: quickstart-pvc
fleetServerEnabled: true
image: elastic/elastic-agent:8.2.0
kibanaRef:
name: quickstart
mode: fleet
version: 8.2.0
|
tips:
之后对于elastic agent的各种配置，直接在kibana网页进行修改，且将配置存储到了数据库中。yaml中的后续修改可能不起作用。
比如：
- 在integrations中kubernetes的proxy设置里，将localhost修改为：
- integrations中System的设置有默认server.example.com域名，可以选择关闭
如果要删除现有policy以及agent，重新通过yaml进行部署的话，需要修改新部署的policy对应id，原因是修改过的配置被web服务存储到了数据库中，因此出现冲突。
最终效果：
