本篇文章为你整理了Beats 入门实战(2)()的详细内容,包含有 Beats 入门实战(2),希望能帮助你了解 Beats 入门实战(2)。
本文主要介绍 Filebeat 的相关概念及基本使用,相关的环境及软件信息如下:CentOS7.9、FileBeat 8.2.2、Logstash 8.2.2、Elasticsearch 8.2.2。
1、Filebeat 简介
1.1、Filebeat 总览
Filebeat 是用于转发和集中日志数据的轻量级传送程序。 作为服务器上的代理安装,Filebeat 监视你指定的日志文件或位置,收集日志事件,并将它们转发到Elasticsearch或Logstash以进行索引。
Filebeat 具有如下的一些特性:
正确处理日志旋转:针对每隔一个时间段生产一个新的日志的案例,Filebeat 可以帮我们正确地处理新生产的日志,并重新启动对新生成日志的处理
背压敏感:如果日志生成的速度过快,从而导致 Filebeat 生产的速度超过 Elasticsearch 处理的速度,那么 Filebeat 可以自动调节处理的速度,以达到 Elasticsearch 可以处理的范围内
“至少一次”保证:每个日志生成的事件至少被处理一次
结构化日志:可以处理结构化的日志数据数据
多行事件:如果一个日志有多行信息,也可以被正确处理,比如错误信息往往是多行数据
条件过滤:可以有条件地过滤一些事件
Filebeat 的工作方式如下:启动 Filebeat 时,它将启动一个或多个输入,这些输入将在为日志数据指定的位置中查找。 对于 Filebeat 所找到的每个日志,Filebeat 都会启动收集器(havester)。 每个收集器都读取一个日志以获取新内容,并将新日志数据发送到 libbeat。libbeat 会汇总事件,并将汇总的数据发送到为 Filebeat 配置的输出。
从上面有可以看出来在 spooler 里有一些缓存,这个可以用于重新发送以确保至少一次的事件消费,同时也可以用于背压敏感。一旦 Filebeat 生成的事件的速度超过 Elasticsearch 能够处理的极限,这个缓存可以用于存储一些事件。每个 Filebeat 可以配置多个 input,并且每个 input 可以配置来采集一个或多个文件路径的文件。 就像上面的图显示的那样,Filebeat 支持多种输入方式。Filbeat 支持如下的一些输出:
Elasticsearch
Logstash
Kafka
Redis
Console
Cloud
1.2、Filebeat 模块
一个 Filebeat 模块通常由如下的部分组成:
Filebeat 模块简化了常见日志格式的收集,解析和可视化。文件集包含以下内容:
Filebeat 输入配置,其中包含在其中查找日志文件的默认路径。 这些默认路径取决于操作系统。 Filebeat 配置还负责在需要时将多行事件缝合在一起。
Elasticsearch Ingest Node 管道定义,用于解析日志行。
字段定义,用于为每个字段配置具有正确的 Elasticsearch 类型,它们还包含每个字段的简短说明。
示例 Kibana 仪表板(如果有)可用于可视化日志文件。
Filebeat 会根据你的环境自动调整这些配置,并将它们加载到相应的 Elastic Stack 组件中。
一个典型的模块(例如,对于 Nginx 日志)由一个或多个文件集(对于 Nginx,访问和错误日志)组成,比如,Nginx 模块解析 NGINX HTTP 服务器创建的访问和错误日志。它在幕后执行如下的一些任务:
设置日志文件的默认路径(你可以更改)
确保每个多行日志事件都作为单个事件发送
使用 ingest node 来解析和处理日志行
将数据塑造成适合在 Kibana 中进行可视化的结构
部署仪表板以可视化日志数据
针对其它的 Beats 模块来说,基本和 Filebeat 一样。目前针对 Elasticsearch 所提供的模块来说,有非常多的模块可以供使用:
关于 Filbeat 的模块信息,可查看官网文档:https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html。
2、Filebeat 安装
根据环境下载对应的安装包:https://www.elastic.co/cn/downloads/beats/filebeat,这里选择 Linux x86_64 版本;下载完成后在服务器上解压即可:
tar zxvf filebeat-8.2.2-linux-x86_64.tar.gz
3、Filebeat 使用
这里使用 Filebeat 来收集 Nginx 访问日志并发送到 Logstash 和 Elasticsearch 中。
3.1、按照普通日志文件方式收集 Nginx 访问日志
3.1.1、配置
修改配置文件 filebeat.yml,配置输入、输出,开始可以先把输出配成 console,调试没问题后再配置输出到Logstash 和 Elasticsearch。
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: true
paths:
- /home/hadoop/app/nginx-1.8.0/logs/access.log
output.console:
pretty: true
enable: true
3.1.2、启动
./filebeat -e -c filebeat.yml
3.1.3、验证
访问 Nginx,日志文件/home/hadoop/app/nginx-1.8.0/logs/access.log 会输出日志:
10.49.196.1 - - [16/Sep/2022:15:35:20 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
filebeat 的控制台打印采集的日志信息如下:
{
"@timestamp": "2022-09-16T07:35:25.797Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.2.2",
"pipeline": "filebeat-8.2.2-nginx-access-pipeline"
"event": {
"timezone": "+08:00",
"module": "nginx",
"dataset": "nginx.access"
"fileset": {
"name": "access"
"ecs": {
"version": "1.12.0"
"agent": {
"type": "filebeat",
"version": "8.2.2",
"ephemeral_id": "5f0443a4-e5c6-43c0-88d2-4ad7b946c4a7",
"id": "197bfd49-e03a-416e-b53f-4ac143b94fa5",
"name": "pxc2"
"message": "10.49.196.1 - - [16/Sep/2022:15:35:20 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"",
"input": {
"type": "log"
"host": {
"id": "a91a3eb89492470398ed77cd24a29299",
"containerized": false,
"ip": [
"10.49.196.11",
"fe80::2011:ba1b:b32b:b63a"
"mac": [
"00:0c:29:97:29:ce"
"hostname": "pxc2",
"name": "pxc2",
"architecture": "x86_64",
"os": {
"version": "7 (Core)",
"family": "redhat",
"name": "CentOS Linux",
"kernel": "3.10.0-957.el7.x86_64",
"codename": "Core",
"type": "linux",
"platform": "centos"
"log": {
"offset": 16308,
"file": {
"path": "/home/hadoop/app/nginx-1.8.0/logs/access.log"
"service": {
"type": "nginx"
}
View Code
3.1.4、处理器
默认情况下配置了如下处理器:
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
如果觉得输出的日志过多,可以把这些处理器去掉并添加去除字段的处理器:
processors:
#- add_host_metadata:
# when.not.contains.tags: forwarded
#- add_cloud_metadata: ~
#- add_docker_metadata: ~
#- add_kubernetes_metadata: ~
- drop_fields:
fields: ["agent.id", "log"]
3.1.5、配置输出到 Logstash
output.logstash:
hosts: ["localhost:5044"]
Logstash 接受到的日志信息如下:
{
"tags" = [
[0] "beats_input_codec_plain_applied"
"event" = {
"timezone" = "+08:00",
"original" = "10.49.196.1 - - [16/Sep/2022:15:51:38 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"",
"dataset" = "nginx.access",
"module" = "nginx"
"log" = {
"offset" = 16866,
"file" = {
"path" = "/home/hadoop/app/nginx-1.8.0/logs/access.log"
"@version" = "1",
"fileset" = {
"name" = "access"
"message" = "10.49.196.1 - - [16/Sep/2022:15:51:38 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"",
"@timestamp" = 2022-09-16T07:51:45.515Z,
"agent" = {
"name" = "pxc2",
"type" = "filebeat",
"ephemeral_id" = "9a6a130b-04c6-4bbf-a228-60e1f237f5a9",
"version" = "8.2.2",
"id" = "197bfd49-e03a-416e-b53f-4ac143b94fa5"
"ecs" = {
"version" = "1.12.0"
"host" = {
"name" = "pxc2"
"service" = {
"type" = "nginx"
"input" = {
"type" = "log"
}
View Code
3.1.6、配置输出到Elasticsearch
output.elasticsearch:
hosts: ["localhost:9200"]
Elasticsearch 会自动创建名为 filebeat-8.2.2 的数据流,数据流中保存的日志信息如下:
{
"_index": ".ds-filebeat-8.2.2-2022.09.16-000001",
"_id": "8qxKRYMBPnCOyxVi1GuP",
"_version": 1,
"_score": 1,
"_source": {
"agent": {
"name": "pxc2",
"id": "197bfd49-e03a-416e-b53f-4ac143b94fa5",
"ephemeral_id": "a6b1d9a5-8485-4391-92be-6a6ae530a5cd",
"type": "filebeat",
"version": "8.2.2"
"nginx": {
"access": {
"remote_ip_list": [
"10.49.196.1"
"log": {
"file": {
"path": "/home/hadoop/app/nginx-1.8.0/logs/access.log"
"offset": 17052
"source": {
"address": "10.49.196.1",
"ip": "10.49.196.1"
"fileset": {
"name": "access"
"url": {
"path": "/",
"original": "/"
"input": {
"type": "log"
"@timestamp": "2022-09-16T07:54:24.000Z",
"ecs": {
"version": "1.12.0"
"_tmp": {},
"related": {
"ip": [
"10.49.196.1"
"service": {
"type": "nginx"
"host": {
"name": "pxc2"
"http": {
"request": {
"method": "GET"
"response": {
"status_code": 304,
"body": {
"bytes": 0
"version": "1.1"
"event": {
"ingested": "2022-09-16T07:54:34.439241175Z",
"original": "10.49.196.1 - - [16/Sep/2022:15:54:24 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"",
"timezone": "+08:00",
"created": "2022-09-16T07:54:33.213Z",
"kind": "event",
"module": "nginx",
"category": [
"web"
"type": [
"access"
"dataset": "nginx.access",
"outcome": "success"
"user_agent": {
"original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
"os": {
"name": "Windows",
"version": "10",
"full": "Windows 10"
"name": "Chrome",
"device": {
"name": "Other"
"version": "105.0.0.0"
"fields": {
"event.category": [
"web"
"user_agent.os.full": [
"Windows 10"
"user_agent.original.text": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
"url.original.text": [
"source.address": [
"10.49.196.1"
"user_agent.os.name.text": [
"Windows"
"user_agent.os.version": [
"10"
"user_agent.os.name": [
"Windows"
"traefik.access.user_agent.name": [
"Chrome"
"service.type": [
"nginx"
"agent.type": [
"filebeat"
"event.module": [
"nginx"
"http.request.method": [
"GET"
"related.ip": [
"10.49.196.1"
"traefik.access.user_agent.original": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
"source.ip": [
"10.49.196.1"
"agent.name": [
"pxc2"
"host.name": [
"pxc2"
"user_agent.version": [
"105.0.0.0"
"http.response.status_code": [
"http.version": [
"1.1"
"event.kind": [
"event"
"event.timezone": [
"+08:00"
"event.outcome": [
"success"
"user_agent.original": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
"event.original": [
"10.49.196.1 - - [16/Sep/2022:15:54:24 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\""
"fileset.name": [
"access"
"nginx.access.remote_ip_list": [
"10.49.196.1"
"input.type": [
"log"
"log.offset": [
17052
"user_agent.name": [
"Chrome"
"agent.hostname": [
"pxc2"
"http.response.body.bytes": [
"traefik.access.user_agent.os_name": [
"Windows"
"user_agent.os.full.text": [
"Windows 10"
"event.ingested": [
"2022-09-16T07:54:34.439Z"
"url.original": [
"@timestamp": [
"2022-09-16T07:54:24.000Z"
"url.path": [
"agent.id": [
"197bfd49-e03a-416e-b53f-4ac143b94fa5"
"ecs.version": [
"1.12.0"
"event.type": [
"access"
"log.file.path": [
"/home/hadoop/app/nginx-1.8.0/logs/access.log"
"event.created": [
"2022-09-16T07:54:33.213Z"
"agent.ephemeral_id": [
"a6b1d9a5-8485-4391-92be-6a6ae530a5cd"
"agent.version": [
"8.2.2"
"user_agent.device.name": [
"Other"
"event.dataset": [
"nginx.access"
}
View Code
3.2、使用 Nginx 模块收集 Nginx 访问日志
Filebeat 中 Nginx 模块可以针对 Nginx 日志进行解析,简化了我们处理日志的过程。
3.2.1、配置
A、filebeat.yml 文件配置
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: false
paths:
- /home/hadoop/app/nginx-1.8.0/logs/access.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.elasticsearch:
hosts: ["localhost:9200"]
B、modules.d/nginx.yml 文件配置
- module: nginx
access:
enabled: true
var.paths: ["/home/hadoop/app/nginx-1.8.0/logs/access.log"]
error:
enabled: false
ingress_controller:
enabled: false
3.2.2、启用 Nginx 模块并启动 Filebeat
./filebeat modules enable nginx
./filebeat -e -c filebeat.yml
3.2.3、验证
访问 Nginx 后,Elasticsearch 会自动创建名为 filebeat-8.2.2 的数据流,数据流中保存的日志信息如下:
{
"_index": ".ds-filebeat-8.2.2-2022.09.16-000001",
"_id": "-qxgRYMBPnCOyxVikGvO",
"_version": 1,
"_score": 1,
"_source": {
"agent": {
"name": "pxc2",
"id": "197bfd49-e03a-416e-b53f-4ac143b94fa5",
"type": "filebeat",
"ephemeral_id": "c7505853-b6de-46b0-abb0-7727dfb37d4b",
"version": "8.2.2"
"nginx": {
"access": {
"remote_ip_list": [
"10.49.196.1"
"log": {
"file": {
"path": "/home/hadoop/app/nginx-1.8.0/logs/access.log"
"offset": 17424
"source": {
"address": "10.49.196.1",
"ip": "10.49.196.1"
"fileset": {
"name": "access"
"url": {
"path": "/",
"original": "/"
"input": {
"type": "log"
"@timestamp": "2022-09-16T08:18:09.000Z",
"ecs": {
"version": "1.12.0"
"_tmp": {},
"related": {
"ip": [
"10.49.196.1"
"service": {
"type": "nginx"
"host": {
"name": "pxc2"
"http": {
"request": {
"method": "GET"
"response": {
"status_code": 304,
"body": {
"bytes": 0
"version": "1.1"
"event": {
"ingested": "2022-09-16T08:18:19.574911507Z",
"original": "10.49.196.1 - - [16/Sep/2022:16:18:09 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"",
"timezone": "+08:00",
"created": "2022-09-16T08:18:18.539Z",
"kind": "event",
"module": "nginx",
"category": [
"web"
"type": [
"access"
"dataset": "nginx.access",
"outcome": "success"
"user_agent": {
"original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
"os": {
"name": "Windows",
"version": "10",
"full": "Windows 10"
"name": "Chrome",
"device": {
"name": "Other"
"version": "105.0.0.0"
"fields": {
"event.category": [
"web"
"user_agent.os.full": [
"Windows 10"
"user_agent.original.text": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
"url.original.text": [
"source.address": [
"10.49.196.1"
"user_agent.os.name.text": [
"Windows"
"user_agent.os.version": [
"10"
"user_agent.os.name": [
"Windows"
"traefik.access.user_agent.name": [
"Chrome"
"service.type": [
"nginx"
"agent.type": [
"filebeat"
"event.module": [
"nginx"
"http.request.method": [
"GET"
"related.ip": [
"10.49.196.1"
"traefik.access.user_agent.original": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
"source.ip": [
"10.49.196.1"
"agent.name": [
"pxc2"
"host.name": [
"pxc2"
"user_agent.version": [
"105.0.0.0"
"http.response.status_code": [
"http.version": [
"1.1"
"event.kind": [
"event"
"event.timezone": [
"+08:00"
"event.outcome": [
"success"
"user_agent.original": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
"event.original": [
"10.49.196.1 - - [16/Sep/2022:16:18:09 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\""
"fileset.name": [
"access"
"nginx.access.remote_ip_list": [
"10.49.196.1"
"input.type": [
"log"
"log.offset": [
17424
"user_agent.name": [
"Chrome"
"agent.hostname": [
"pxc2"
"http.response.body.bytes": [
"traefik.access.user_agent.os_name": [
"Windows"
"user_agent.os.full.text": [
"Windows 10"
"event.ingested": [
"2022-09-16T08:18:19.574Z"
"url.original": [
"@timestamp": [
"2022-09-16T08:18:09.000Z"
"url.path": [
"agent.id": [
"197bfd49-e03a-416e-b53f-4ac143b94fa5"
"ecs.version": [
"1.12.0"
"event.type": [
"access"
"log.file.path": [
"/home/hadoop/app/nginx-1.8.0/logs/access.log"
"event.created": [
"2022-09-16T08:18:18.539Z"
"agent.ephemeral_id": [
"c7505853-b6de-46b0-abb0-7727dfb37d4b"
"agent.version": [
"8.2.2"
"user_agent.device.name": [
"Other"
"event.dataset": [
"nginx.access"
}
View Code
可以看到日志信息已经被解析出各个单独的有用字段,如:source.ip,url.path,user_agent.name 等等。
参考:https://blog.csdn.net/UbuntuTouch/article/details/104432643
以上就是Beats 入门实战(2)()的详细内容,想要了解更多 Beats 入门实战(2)的内容,请持续关注盛行IT软件开发工作室。
郑重声明:本文由网友发布,不代表盛行IT的观点,版权归原作者所有,仅为传播更多信息之目的,如有侵权请联系,我们将第一时间修改或删除,多谢。