-
Notifications
You must be signed in to change notification settings - Fork 557
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
馃悶 Setting --oci-max-parallelism 2
causes deadlocks
#6894
Comments
--oci-max-parallelism 2
can cause deadlocks--oci-max-parallelism 2
causes deadlocks
The team over here at Motion have seen this happen to us as well on 0.10.3 |
I think this is expected - suppose the following module structure:
I think with modules, this effectively means that Not sure what the right fix is here:
I don't think there's a clear way forwards here, but to me this kind of highlights that we need a better way of constraining resources - |
Adding this to the v0.12.0 milestone since by that release we should probably either:
|
x-linking to #6465 as well. |
While we are keeping this default so that we don't break existing users, setting this value is something that we want to move away from. The problem is that this setting limits how many operations can run in parallel. It is still possible for a single operation to max out all available cores. It is also known for a value of `2` to cause deadlocks, i.e. dagger#6894 For now, we just allow this to be disabled with either `--set engine.args=''` or by explicitly setting this value to an empty list. This started as dagger#7395 which turned out to be too big of a change. We since scaled back the initial ambition & are taking a smaller step towards eventually phasing this out. FWIW, all the Dagger Engines that we run inside the Dagger infra do not use the `--oci-max-parallelism` option. This also removes the option from tekton-dagger-task docs example. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>
While we are keeping this default so that we don't break existing users, setting this value is something that we want to move away from. The problem is that this setting limits how many operations can run in parallel. It is still possible for a single operation to max out all available cores. It is also known for a value of `2` to cause deadlocks, i.e. #6894 For now, we just allow this to be disabled with either `--set engine.args=''` or by explicitly setting this value to an empty list. This started as #7395 which turned out to be too big of a change. We since scaled back the initial ambition & are taking a smaller step towards eventually phasing this out. FWIW, all the Dagger Engines that we run inside the Dagger infra do not use the `--oci-max-parallelism` option. This also removes the option from tekton-dagger-task docs example. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>
What is the issue?
Setting
--oci-max-parallelism 2
on the Engine causes deadlocks, both with & without modules.This is what that looks like from the CLI / user perspective:
dagger-0.10.2-deadlock.mp4
Attaching Dagger Engine v0.10.2 logs +
SIGQUIT
. Thank you @vito for https://vito.github.io/swirly/ and the tips!FWIW:
dagger-0.10.2-sigquit-swirly.mp4
cc @sipsma @vito @jedevc @aluzzardi
Dagger version
Confirmed on 0.9.11 & 0.10.2
Steps to reproduce
No response
Log output
No response
The text was updated successfully, but these errors were encountered: