perf(core): Introduce concurrency control for main mode

[ivov]

This PR introduces concurrency control for main mode. This limits how many production executions are allowed to run at the same time in main mode, preventing event loop overload and so improving reliability.

Concurrency control applies only in main mode and only to production executions (i.e. in webhook and trigger modes), not to executions in manual, retry, error, integrated, cli or internal modes.

Summary of changes

Queuing

• Implement ConcurrencyControlService.
• Reuse Adi's work in ConcurrencyQueue.
• Block or let through executions at ActiveExecutions.
• On releasing an enqueued execution, reset startedAt timestamp.
• On cancelling an enqueued execution, remove from queue.
• On retrying an enqueued execution, block retry attempt.
• On deleting an enqueued execution via Public API, remove from queue.
• On deleting an enqueued execution via Internal API, remove from queue.
• On instance startup post crash, exclude enqueued executions from crash recovery.
• On instance startup post crash, run all enqueued executions.
• Serve executions prioritizing running over new
• Ensure queue mode is unaffected by concurrency control.

Envs, logs, tests

• Add new env var N8N_CONCURRENCY_PRODUCTION_LIMIT.
• Add logs on throttling and releasing executions.
• Add tests for ConcurrencyControlService and ConcurrencyQueue.
• Add telemetry to monitor throttling frequency.

Discussion: https://www.notion.so/n8n/Concurrency-limits-for-main-mode-8b642df5e98a479cadd1cfc4950d27a1

Follow-up to: #8458

[luizeof]

@ivov It would be great if we could configure a worker to only run workflows that have certain tags.

This way we can configure a service for the "critical" tag, another for the "mailing" tag, etc...

[ivov]

@ivov It would be great if we could configure a worker to only run workflows that have certain tags.

This way we can configure a service for the "critical" tag, another for the "mailing" tag, etc...

Thanks for your feedback! fyi @jcn8n

[krynble]

LGTM

[github-actions]

⚠️ Some Cypress E2E specs are failing, please fix them before merging