cccb-website-team/kik-website
Watch
0
0
Fork
You've already forked kik-website
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Restart the shell server only when it has to, and record failed starts

Browse source 
Every outage since 28 Oct began at the cleanup job's routine restart of a
working server: the start failed and kept failing until a reboot. The job
now leaves a working server alone and restarts it only when it does not
listen, is older than its configuration, or holds open a program or the
runtime that has to be repaired.

A failed start left no trace anywhere, so windows now records the history
of the scheduled tasks and the start and end of every process, and the job
writes the processes and the state of the server's task to
failed-starts.log when the server does not come up. The security log grows
to a gigabyte, because the scanners of the remote desktop fill a hundred
megabytes in a week.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zebreus 2026-11-06 00:17:08 +01:00
commit 727f8f6e9c
1 changed files with 92 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

16
server/README.md
View file

@ -69,10 +69,11 @@ Three scheduled tasks run as SYSTEM, all at startup, and the user cannot stop an
the server only starts it again when it failed, never when it was stopped, so a cleanup job that is killed while
it has the server stopped would otherwise take the site down with it.
The cleanup job also restarts the server when nobody is connected, and when the server does not listen at all:
it comes back by itself if it died, it takes over a server that was left behind, and it picks up the programs that
were just restored. A session that is open keeps the port, so a server that is started now could not bind it, and
the job never disturbs a session.
The cleanup job only restarts the server when it has to: when it does not listen at all, when it is older than its
configuration, or when the protected copy or the runtime has to be repaired - those cannot be replaced while the
server holds them open. A working server is left alone, because every start can fail, and failed starts have kept
the site down for hours. The job also never restarts the server while somebody is connected: a session that is open
keeps the port, so a server that is started then could not bind it.
The server is only down for the short part of the job that needs it down: the protected copy cannot be replaced
while the server holds it open. Everything else - the home directory, the four texts, the permissions of the
@ -90,7 +91,12 @@ at the next run of the cleanup job, which happens within half an hour.
The server logs to `C:\ProgramData\kik-restore\sshd.log`. The log is appended to, not overwritten, because it is
the only trace of what the server did: a server that is restarted would otherwise take the evidence of an outage
with it. It is rotated when it grows past a hundred megabytes, which takes years. The event logs of windows are
made that big as well, so that they keep the history of the box itself for just as long.
made bigger as well, the security log to a gigabyte: the scanners of the remote desktop alone fill a hundred
megabytes of it in a week.
A start of the server that fails leaves nothing in that log. So windows records the history of the scheduled tasks
and the start and end of every process, and when the server does not come up, the cleanup job writes the processes
and the state of the server's task to `C:\ProgramData\kik-restore\failed-starts.log`.
Only keys may log in over SSH. The box stands in the open internet and both of its accounts have a password.