Support for Configuration Reload in journald

[smreichling]

So far as I am able to tell, making a change to any of journald's configuration files (/etc/systemd/journald.conf, for example) requires subsequently running systemctl restart systemd-journald in order for the changes to take effect. However, restarting the journald service appears to break logging for all other running services until those services are also restarted (journalctl -flu <service_name> shows no new log lines post journald restart).

I'd like to request support for reloading journald configuration without restarting the process and breaking logging for other systemd services. I see this feature as rather critical. Without it, it's necessary for me to take servers out of service just to change the logging configuration.

[evverx]

However, restarting the journald service appears to break logging

@smreichling , which version of systemd are you using?

See 13790add4bf648fed8163

journald: allow restarting journald without losing stream connections
Making use of the fd storage capability of the previous commit, allow
restarting journald by serilizing stream state to /run, and pushing open
fds to PID 1.

(Since v219)

[smreichling]

@evverx I had been on 208, but I just tried upgrading to 219 and I still have the same problem.

Here's what I did: I wrote a dummy python service called log-tester which just prints to stdout 5 times a second in an endless loop. To test the upgrade, I ran:

1. journalctl -flu log-tester -- see a constant stream of output, all appears to be working.
2. systemctl restart systemd-journald
3. journalctl -flu log-tester -- stream of output has stopped.
4. systemctl restart log-tester
5. journalctl -flu log-tester -- stream of output has resumed.

I repeated these same steps several times with the same results each time. I confirmed the upgrade by running systemctl --version and journalctl --version, both show 219.

[evverx]

Hm,

• v225: I can reproduce it.
• v226: I can reproduce it.
• v227: I can't reproduce it.
• v228: I can't reproduce it (maybe, this commit is a fix)

So, something went wrong between v219..v226 (my git bisect is too slow:))
Fixed: a5bd3c3

Try v227 or v228.

[poettering]

@smreichling 219 is too old. We only track issues with the two most current systemd versions upstream, and afaics you are describing an issue in much older systemd versions that got fixed since. Closing hence.

[lewicki-pk]

Hi,
I have just reproduced this issue on systemd 230 and then again on systemd 232.

I'm using systemd provided by yocto system so it has some extra patches applied. If you're familiar with Yocto project than you can read the recipe here http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/systemd/systemd_232.bb?h=master and also check the patches here: http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/systemd/systemd

My setup info:
I have a system where my systemd-journal uses only volatile storage and I have rsyslog configured to store logs in the persistent directory.

I thought I can toggle ForwardToSyslog= parameter in journald.conf file in order to enable/disable persistent storage for logs.

Unfortunately this does not work and I need to restart all of the services whose logs I want to have in my syslog files.

Steps to reproduce the issue:

1. Change the option ForwardToSyslog from yes to no.
2. Restart the systemd-journal service (systemctl restart systemd-journald)
3. Check if syslog files are no longer written to (in my case they are not appended).
4. Change the option ForwardToSyslog back to no.
5. Restart the systemd-journal service again (systemctl restart systemd-journald)
6. Syslog files are still not appended.
7. After restarting only one of running services you can observe that the logs from that service show up in syslog files and logs from other services do not.

Can you reopen this issue?

Extra question that would help me workaround my problem:
Is there an option to restart all running services in systemd?

[evverx]

4. Change the option ForwardToSyslog back to no.

Do you mean "back to yes"?

6. Syslog files are still not appended.

Does journalctl -fu some-service print anything?

[evverx]

Can you reopen this issue?

This issue is not about ForwardToSyslog. It's about journalctl -fu some-service. Please create a new issue.

[evverx]

btw we can lose some logs due to FileDescriptorStoreMax=1024.

systemd-journald.service: Cannot store more fds than FileDescriptorStoreMax=1024, closing remaining.

systemctl restart systemd-journald causes write(1, "Woo\n", 4) = -1 EPIPE (Broken pipe) so I have to restart some services.

[keszybz]

@lewicki-pk can you please open a new issue? This is most likely unrelated, because the transport for syslog files is different.

Is there an option to restart all running services in systemd?

No.

[lewicki-pk]

Ok. I have opened a new issue #5586

Thanks

[yuchengwu]

I know 219 is not supported, just providing a clue. Not sure it is related to os environment because I can reproduce on one environment but not on another, both using systemd 219

executing the following 2 steps

1. systemctl restart systemd-journald
2. systemctl status kubelet # check kubelet daemon status
   I found kubelet not restart on this env

[root@tdc-tester01 _output]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@xxx _output]# uname -a
Linux xxx 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@xxx _output]# systemctl --version 
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

but restarted on that env once we restarting systemd-journald