Delay in sending logs with fielbeat 8.12.1

[sasikiranvaddi]

There has been a noticeable delay observed when sending logs to using filebat 8.12.1

filebeat configuration used

filebeat.inputs:

• type: stdin
  json.keys_under_root: true
  processors:

• drop_event:
  when:
  !java.util.LinkedHashMap
  or:
  - !java.util.LinkedHashMap
  equals: !java.util.LinkedHashMap
  exclude: Hello
  - !java.util.LinkedHashMap
  equals: !java.util.LinkedHashMap
  exclude2: Hello Testing

• drop_fields:
  fields: ["input","offset","log","@metadata","host","beat","prospector","source"]
  output.console:
  pretty: false

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

Hello @sasikiranvaddi
Have you seeing the same delay without the multiple exclusion filters? Is it only for the first events or every events?
Would like to understand if this is linked to the new defaults we introduced in 8.12: https://www.elastic.co/fr/blog/using-elastic-agent-performance-presets-in-8-12

[sasikiranvaddi]

Hi @pierrehilbert,
Yes we fine tuned flush timeout as specified https://github.com/elastic/beats/blob/v8.12.1/CHANGELOG.asciidoc, by setting queue.mem.flush.timeout: 1s in output section on monday and observed the logs are receiving faster. We are trying to perform some more tests to confirm that it has improved.

[pierrehilbert]

Keep me honest here but it doesn't seem to be a problem with the exclusion filters then but with the new default you have compare to the behavior you were expecting.
@faec do you have some recommendation to share here to help them getting the expected behavior?

[sasikiranvaddi]

Hi @pierrehilbert,
After a quick test, the behavior of delayed events are observed from 8.12.1 even without including exclusion/inclusion filters. After adding queue.mem.flush.timeout then the events are sent faster to the output source.
I'll update the title accordingly.
Just to add, in 8.11 we haven't noticed any delay. Probably the new change of updating default value of queue.mem.flush.timeout from 1s to 10s might have caused this delay

[pierrehilbert]

Yes this is definitively related to this parameter that we changed to improve overall performances observed by the users.

[sasikiranvaddi]

Thank you @pierrehilbert,
We will keep this Bug open for few days once our tests are concluded and if the results found to be positive then we can close the Bug.

[faec]

Yes, it sounds to me like this is caused by the presets changes. If you want to stick with a general preset you can try preset: latency, but you can also just customize your performance parameters as before. If latency is at a premium you might even consider queue.mem.flush.timeout: 0, which will send events without any delay.

[sasikiranvaddi]

Hi @faec,
Got your point. Thank you for the info.

[sasikiranvaddi]

Hi @faec, @pierrehilbert ,
The test results looks promising. Thank you for the support.
I will be closing the ticket.