Missing credentials in ExecStartPost=/ExecStop*= when ReadWritePaths= is used

[mrc0mmand]

I came across this when debugging test-execute in the coverage run - when ReadWritePaths= (and probably friends) is used, the service is missing credentials in ExecStartPost= and ExecStop*= stages:

# SPDX-License-Identifier: LGPL-2.1-or-later
[Unit]
Description=Test for SetCredential=

[Service]
ExecStart=sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
ExecStartPost=sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
ExecStop=sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
ExecStopPost=sh -x -c 'test "$$(cat %d/test-execute.set-credential)" = "hoge"'
Type=oneshot
SetCredential=test-execute.set-credential:hoge
ReadWritePaths=/mnt

# systemctl start exec-set-credential
Job for exec-set-credential.service failed because the control process exited with error code.
See "systemctl status exec-set-credential.service" and "journalctl -xeu exec-set-credential.service" for details.
# journalctl -e -o short-monotonic --no-hostname
...
[ 7434.726476] systemd[1]: Starting Test for SetCredential=...
[ 7434.732271] sh[262661]: ++ cat /run/credentials/exec-set-credential.service/test-execute.set-credential
[ 7434.732983] sh[262659]: + test hoge = hoge
[ 7434.742345] sh[262664]: ++ cat /run/credentials/exec-set-credential.service/test-execute.set-credential
[ 7434.743267] sh[262664]: cat: /run/credentials/exec-set-credential.service/test-execute.set-credential: No such file or directory
[ 7434.743406] sh[262663]: + test '' = hoge
[ 7434.744353] systemd[1]: exec-set-credential.service: Control process exited, code=exited, status=1/FAILURE
[ 7434.750558] sh[262667]: ++ cat /run/credentials/exec-set-credential.service/test-execute.set-credential
[ 7434.751156] sh[262667]: cat: /run/credentials/exec-set-credential.service/test-execute.set-credential: No such file or directory
[ 7434.751319] sh[262666]: + test '' = hoge
[ 7434.751794] systemd[1]: exec-set-credential.service: Control process exited, code=exited, status=1/FAILURE
[ 7434.751823] systemd[1]: exec-set-credential.service: Failed with result 'exit-code'.
[ 7434.752704] systemd[1]: Failed to start Test for SetCredential=.
[ 7434.753599] systemd[1]: run-credentials-exec\x2dset\x2dcredential.service.mount: Deactivated successfully.
...

The original error I was trying to reproduce (from the coverage run) is:

[    6.169854] (test-execute-root)[767]: exec-set-credential.service: Will spawn child (service_enter_start): sh
[    6.170048] (test-execute-root)[767]: exec-set-credential.service: Passing 0 fds to service
[    6.170217] (test-execute-root)[767]: exec-set-credential.service: About to execute: sh -x -c "test \"\$\$(cat /run/credentials/exec-set-credential.service/test-execute.set-credential)\" = \"hoge\""
[    6.170407] (test-execute-root)[767]: Serializing sd-executor-state to memfd.
[    6.170620] (test-execute-root)[767]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
[    6.170851] (test-execute-root)[767]: exec-set-credential.service: Forked sh as 841 (without CLONE_INTO_CGROUP)
[    6.171118] (test-execute-root)[767]: Closing set fd 25 (socket:[9648])
[    6.171346] (test-execute-root)[767]: Closing set fd 26 (socket:[9646])
[    6.171578] (test-execute-root)[767]: exec-set-credential.service: Changed dead -> start
[    6.177047] (sh)[841]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
[    6.177637] (sh)[841]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
[    6.177843] (sh)[841]: exec-set-credential.service: Failed to set up credentials: Read-only file system
[    6.178040] (sh)[841]: exec-set-credential.service: Failed at step CREDENTIALS spawning sh: Read-only file system
[    6.186056] (test-execute-root)[767]: Received SIGCHLD from PID 841 ((sh)).
[    6.186235] (test-execute-root)[767]: Child 841 ((sh)) died (code=exited, status=243/CREDENTIALS)
[    6.186429] (test-execute-root)[767]: exec-set-credential.service: Child 841 belongs to exec-set-credential.service.
[    6.186620] (test-execute-root)[767]: exec-set-credential.service: Main process exited, code=exited, status=243/CREDENTIALS
[    6.186841] (test-execute-root)[767]: exec-set-credential.service: Will spawn child (service_enter_stop_post): sh
[    6.187032] (test-execute-root)[767]: exec-set-credential.service: About to execute: sh -x -c "test \"\$\$(cat /run/credentials/exec-set-credential.service/test-execute.set-credential)\" = \"hoge\""
[    6.187217] (test-execute-root)[767]: Serializing sd-executor-state to memfd.
[    6.187376] (test-execute-root)[767]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
[    6.187530] (test-execute-root)[767]: exec-set-credential.service: Forked sh as 842 (without CLONE_INTO_CGROUP)
[    6.187701] (test-execute-root)[767]: Closing set fd 27 (socket:[9646])
[    6.187864] (test-execute-root)[767]: Closing set fd 26 (socket:[9648])
[    6.188027] (test-execute-root)[767]: exec-set-credential.service: Changed start -> stop-post
[    6.188193] (test-execute-root)[767]: exec-set-credential.service: Control group is empty.
[    6.190942] (sh)[842]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
[    6.191286] (sh)[842]: Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
[    6.191458] (sh)[842]: exec-set-credential.service: Executing: sh -x -c "test \"\$(cat /run/credentials/exec-set-credential.service/test-execute.set-credential)\" = \"hoge\""
[    6.191647] (test-execute-root)[767]: Got handoff timestamp event for PID 842.
[    6.202756] test-execute[767]:         Stan
[    6.202756] test-execute[843]: ++ cat /run/credentials/exec-set-credential.service/test-execute.set-credential
[    6.203511] test-execute[843]: cat: /run/credentials/exec-set-credential.service/test-execute.set-credential: No such file or directory
[    6.203720] test-execute[842]: + test '' = hoge
[    6.203987] (test-execute-root)[767]: Received SIGCHLD from PID 842 (sh).
[    6.204235] (test-execute-root)[767]: Child 842 (sh) died (code=exited, status=1/FAILURE)
[    6.204488] (test-execute-root)[767]: exec-set-credential.service: Child 842 belongs to exec-set-credential.service.
[    6.204750] (test-execute-root)[767]: exec-set-credential.service: Control process exited, code=exited, status=1/FAILURE
[    6.205077] (test-execute-root)[767]: exec-set-credential.service: Got final SIGCHLD for state stop-post.
[    6.205340] (test-execute-root)[767]: exec-set-credential.service: Failed with result 'exit-code'.
[    6.205653] (test-execute-root)[767]: exec-set-credential.service: Service will not restart (restart setting)
[    6.205921] (test-execute-root)[767]: exec-set-credential.service: Changed stop-post -> failed
[    6.206180] (test-execute-root)[767]: exec-set-credential.service: Unit entered failed state.
[    6.206437] (test-execute-root)[767]: src/test/test-execute.c:355:test_exec_credentials: exec-set-credential.service: can_unshare=yes: exit status 243, expected 0
[    6.213069] systemd-coredump[844]: Process 767 ((test-execute-r) of user 0 terminated abnormally with signal 6/ABRT, processing...

Which feels quite similar.

[YHNdnzj]

Hmm? ExecStop= and friends should not be able to access credentials in the first place? I'm quite surprised that the test used to pass?

But I'll check why this fails for ExecStartPost=

[YHNdnzj]

Hmm, I have little idea on this specific problem...

However, while debugging this, I've discovered that Type=simple + the use of credentials is racy, since we don't wait for sd-executor for ExecStart= to finish before starting ExecStartPost=, and setup_credentials_internal depends on the state of the final dir. I think it's better to imply Type=exec when creds are used. @poettering WDYT?

[mrc0mmand]

Hmm? ExecStop= and friends should not be able to access credentials in the first place? I'm quite surprised that the test used to pass?

But I'll check why this fails for ExecStartPost=

It's been that way for a while, see https://github.com/systemd/systemd/blob/main/test/test-execute/exec-set-credential.service.