Redirect run.sh stdout and stderr to log files

[abagali1]

Description

Currently site process logs are only viewable by right clicking on the file sidebar and choosing "View Logs". This feature is not well known and writing to log files was the standard under director 3.0. We should redirect site process logs to a file again.

Things to consider

A simple output redirection (>out.log, 2>error.log, etc) would redirect stdout and stderr to files and docker service logs will not longer be able to read the output. docker service logs is good for administrative debugging so we should try to keep this feature

[ghost]

First, I don't think keeping the logs accessible via docker service logs is terribly important. We can just go in and read the logfile the same way the user does.

Redirecting the logs could be done easily from inside the sh script that launches the site's run.sh. The tricky part would be keeping the "Show Logs" feature in place as-is.

The reason this feature is important (honestly, the difference is so important that I've been shocked by how many people opened the logfile directly instead) is that it continuously streams data from the logs; the rough equivalent of a tail -f. We would need to add separate methods to be able to stream data like that from a file instead of docker service logs. And we would want this to be a generalized solution that we can do with any file, so we can revive Director 3.0's "Show as Log" option on any file.

Here's what would need to be added to support that (once the output redirection is put in place):

1. Add a tail action to the files helper to read continuously from a file. The idea would be that it opens the file and reads to the end, printing out chunks of text as it reads them. Once it reaches the end, it enters a loop where it sleep()s briefly, then tries to read from the file to see if anything has been added. (This is what tail often does. We can't use inotify to watch for changes because the Docker container might be modifying the logs on a different appserver.)
2. Add a websocket route on the orchestrator that launches the tail files helper and streams its output.
3. Add a websocket consumer on the manager that similarly propagates the information up.
4. Update the JS to handle this (most likely only minor tweaks).

[ghost]

Something else I thought of while writing my previous comment: We need to rotate the logfile. Some sites might write thousands of lines of log output; at some point the logfile will get so long that the browser can't load it. That means we can't do a simple redirection; we need another program to process the logs.

Additionally, this would slightly complicate the tail action in the files helper, since it would need to check if the file had been renamed so it could seamlessly transition and begin watching the new file instead.