-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple Harvested of log file path got started for the same log file name within the span of 7 seconds #39567
Comments
@vikas271019 Please share your Filebeat configuration. And please enclose it in triple backticks (```) so the formatting is correctly preserved. |
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
Events: |
Hi Team, |
Hi All, |
We did lot of improvement on K8S environment in the last versions. |
--Thanks for your reply, as you specifically mentioned lot of improvements on K8S environment in the latest version of fliebeat. kindly confirm based on what all factors you suspect this issue is related to K8S environment. |
Hi Team, |
Hi @pierrehilbert |
We are using filebeat 8.4.3.
The problem that we are facing is, multiple harvester is being created for a single log file, within a span of 7 seconds as result of which duplicate logs being generated from filebeat.
For example, please find the below set of logs flow from log producer to filebeat
Log Producer microservice logs -test-controller
Single logs being printed in log generator microservice container logs with message = "auto,1.64"
{"timestamp":"2024-05-10T05:54:51.350+02:00","service_id":"test-controller","message":"auto,1.64","metadata":{"category":"HA-in-service-performance","namespace":"test","pod_name":"test-controller-6f565dd5b9-qv2w2","application_id":"test884"},"severity":"info","version":"1.1.0","facility":"security/authorization messages","extra_data":{"test_stream":"dsp","in_service_performance":{"version": "1.0.0","originating_service_name": "test-controller","originating_service_version": "1.239.7-1","originating_pod_name": "test-controller-6f565dd5b9-qv2w2","event_type":"small-local-restart","reporting_service_version": "1.239.7-1"}}}
But in the configured output we are seeing the logs being seen twice, sometime 4 times. while we are analysing the beats logs we could see two harvesters being created. please find the filebeat logs as mentioned below for the same file path being started twice(test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log)
{"log.level":"info","@timestamp":"2024-05-10T04:56:47.874+0200","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":310},"message":"Harvester started for paths: [/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log]","service.name":"filebeat","input_id":"e271da1d-390e-48ac-be82-4be7d7e644bc","source_file":"/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log","state_id":"native::143563266-64515","finished":false,"os_id":"14266-64515","harvester_id":"e9f36c6a-d5c1-4ddc-958f-d254da5b6ea6","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-05-10T04:56:54.160+0200","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":310},"message":"Harvester started for paths: [/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log]","service.name":"filebeat","input_id":"71aa7599-3723-4e6c-a3d2-f9ac0a1007ec","source_file":"/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log","state_id":"native::14266-64515","finished":false,"os_id":"14266-64515","harvester_id":"2e86644e-e9c7-4f68-b859-b225711f9220","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-05-10T04:56:54.281+0200","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":337},"message":"Reader was closed. Closing.","service.name":"filebeat","input_id":"71aa7599-3723-4e6c-a3d2-f9ac0a1007ec","source_file":"/var/log/containers/test-controller-6f565dd5b9-qv2w2-d76e6f05091b6747ecac5660ab65b32c51368729a48640afbc86484cf3c05d15.log","state_id":"native::14266-64515","finished":false,"os_id":"143563266-64515","harvester_id":"2e86644e-e9c7-4f68-b859-b225711f9220","ecs.version":"1.6.0"}
Hence we wanted to understand this is causing duplicate logs being sent to the configured output? Why multiple harvester is being created? is there any way we can avoid it?
NOTE: there are no restarts or disturbances caused to the registry file.
The text was updated successfully, but these errors were encountered: