Every outage since 28 Oct began at the cleanup job's routine restart of a
working server: the start failed and kept failing until a reboot. The job
now leaves a working server alone and restarts it only when it does not
listen, is older than its configuration, or holds open a program or the
runtime that has to be repaired.
A failed start left no trace anywhere, so windows now records the history
of the scheduled tasks and the start and end of every process, and the job
writes the processes and the state of the server's task to
failed-starts.log when the server does not come up. The security log grows
to a gigabyte, because the scanners of the remote desktop fill a hundred
megabytes in a week.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The sshd log is rotated at a hundred megabytes instead of one, and the event
logs of windows are made that big as well: they are the only record of what
the box itself did, and twenty megabytes are not much with all the connections.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The cleanup job stops the server, puts the files back and starts it again. On
a slow disk that took minutes, and on 2026-10-28 the job was killed while the
server was down: the task of the server only starts it again when it failed,
never when it was stopped, so the site was down until the box was power-cycled.
The job now puts the home directory, the four texts and the permissions of the
installation back while the server runs, and only stops it for the protected
copy, which cannot be replaced while the server holds it open.
A new task, "KiK watch the server", starts the server every two minutes when
nothing listens on port 22 and the cleanup job is not running.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The task that runs the server ignores a start while it thinks it is still
running, so the cleanup job could stop the server and then fail to start it
again: the site had no shell for forty minutes. The start is now tried three
times, with the processes of the task killed in between.
The job now only counts the established connections when it decides whether
anybody is connected - the server leaves the processes of finished sessions
behind, and they kept it from ever restarting the server. Those processes are
killed with the server, because they hold the runtime and the protected
programs open.
The pacman calls are bounded, so a database lock cannot keep the server down,
and they run before the server is stopped. The exit code of a program started
this way is only available when its handle is taken while it runs, and the
pattern for the files to overwrite has to be quoted, or the runtime expands
it.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The server logs to a file that was overwritten on every start, so an outage left no
trace. It is appended to now, and rotated when it grows past a megabyte.
The cleanup job restarts the server when nobody is connected and when the server does
not listen at all: it comes back by itself if it died, it takes over a server that was
left behind, and it picks up the programs that were just restored. A session that is
open keeps the port and the runtime, so the server is not restarted then.
A deployment does not restart the server. It runs in a session, and that session keeps
the port, so a server that starts during a deployment cannot bind it. The deployment
only writes the configuration; the cleanup job applies it.
The runtime of the installation is not protected any more. A program and the shell
that starts it have to use the same runtime DLL, or the program does not run at all,
so the login path now uses the one in /usr/bin, next to the user's own programs. What
is protected instead is that it is the one the package says it is: pacman checks the
files of msys2-runtime and the cleanup job puts the package back if it was changed.
Only keys may log in over SSH now, and Remote Desktop needs network level
authentication. Only the administrator may log in over it: the user of the website is
in no group that may.
Co-Authored-By: Claude Opus <noreply@anthropic.com>
The root user now owns the whole MSYS2 installation and can run pacman. The
login path the journey depends on - the sshd, the login shell zsh and cat - is
kept as a protected copy in C:\msys2\opt\kik that only SYSTEM and the
administrators may write, and the sshd runs as SYSTEM from there (the new
KiK sshd task; the msys2-sshd service is disabled). Replacing the user's own
/usr/bin/sshd therefore cannot touch the running server.
opt/kik is two levels below C:\msys2, so its runtime resolves the same
installation root and the login shell and the user's tools share one cygwin
runtime. A separate second installation does not work: two runtimes in one
process tree corrupt each other's file descriptors and pacman cannot lock its
database.
/etc/passwd and /etc/ssh are read-only to the user, /opt/kik is first on PATH
so cat README survives even if the user deletes their own /usr/bin/cat, and
the restore task rebuilds the protected copy and re-applies the permissions
every 30 minutes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The user could empty the authorized_keys and break the login of the
website, change the shell configuration to break the prompt, and write
startup files that zsh reads on every login. The shell configuration and
the key are now read-only for the user, and the shell reads its
configuration from ~/.zsh, so the files in the home directory are
ignored.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The user could delete the whole home directory, which takes the website
down with it: sshd cannot log in without it. The home directory and the
files that have to stay in it now belong to the administrators, so the
user may change them but not remove them. Their own files stay theirs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MSYS2 only copies /etc/skel into a home directory that does not exist yet
when the user logs in. The cleanup always creates the home first, so the
default files never arrived.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The resources were copied from home/ and admin/, which held the same
files a second time. Everything that goes to the server is in resources/
now, and the generator only writes the four text files.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The keys of the people who deploy are in the repository now: in home/ for
the root user and in admin/ for the Administrator. The resources mirror
both home directories, and the deploy script no longer reads any private
key.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The personal key of the deployer goes into the authorized_keys of the
root user together with the key of the website, and the deployment makes
sure it is in the authorized_keys of the Administrator.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The key that the website logs in with was written into the authorized_keys
of the Administrator, which would have given everyone who has the website
administrator rights on the box. It only gets access as the root user.
The deploy script logs in as Administrator with the personal key of the
deployer instead, from the agent or from -i.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The key that the website logs in with lives in the home directory of the
user, so a deleted home directory took it away and the site could not
connect any more. The key is now one of the copies the cleanup restores,
and it is merged into the authorized_keys of the user, so that keys they
added stay.
The installer writes the same key for the Administrator, and the deploy
script checks that it is the key the website uses.
Co-Authored-By: Claude Code <noreply@anthropic.com>
It logs in as Administrator, whose shell is a Windows one, copies the
scripts and the resources to the server and runs the installer there. It
uses the key the website uses and checks the host key against the one
the website pins.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The shell configuration was copied from the server by hand. It now has
its own directory, which make_files.py copies into the resources, so
that the resources are the whole set of files the cleanup restores and
nothing has to be copied by hand.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The cleanup job now restores the whole home directory instead of only
the four text files: the directory itself, the shell configuration that
gives the prompt, and the files. Anything that is not a regular file is
replaced, so the files stay text files.
The README in server/ describes the security model: the box is a
disposable VPS that may be compromised, the root user may do whatever
they want, and only the prompt, the home directory and the four files
are kept working.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The README and Code of Conduct files move to server/resources, the
scripts that build and deploy them to server/scripts. The new
install_restore_task.ps1 installs a scheduled task as SYSTEM that puts
the files back every 30 minutes, so the shell server's root user can
neither stop it nor change the copies it restores from.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The terminal versions of the pages for the home directory of the shell
server's root user, so each page's top command (cat README, cat README.en,
cat CODE_OF_CONDUCT, cat CODE_OF_CONDUCT.en) works in the web terminal.
Generated by server/make_files.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- OSC 8 hyperlinks and plain URLs are clickable on the screen and in the
page history. Links to a section of the current page scroll there,
others open in a new tab. Links show their target on hover and invert
in their own color, like the page's links.
- The terminal fills the page column (the fit addon kept room for a
hidden scrollbar).
- Lines in the page history sit exactly on the terminal's cell grid,
also for wide and fallback-font characters, and no longer wrap.
- The first command is sent as soon as the shell's line editor is ready,
instead of guessing from a pause in its output.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The build is flat (pages, loader.js, terminal.js, logo.png) and only uses
relative URLs, so the site works wherever it is hosted, e.g. at
https://berlin.ccc.de/kik/.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The glow used to be green for all text, including red or blue terminal
output. The explicit red glows are no longer needed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lines scrolling off the terminal screen are converted from xterm.js's
cells to styled page text: palette, 256 and true colors, background,
bold (bright), italic, dim, underline, strikethrough, inverse and
invisible, without trailing spaces and keeping empty lines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The key is no longer served as public/root_key. It is stored as base64
of its rot13 and decoded where the connection is made.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The bottom prompt is now a real text input. Sending the first command
lazy-loads an xterm.js terminal that continues right below it and
connects to wss://web.nowhere.wirs.ing with a minimal in-browser SSH
client (curve25519, strict kex, pinned ed25519 host key,
chacha20-poly1305, re-keying) using public/root_key.
The page is the scrollback: lines scrolling off the window-sized
terminal screen become page text, the screen grows with the lines in
use, and the wheel scrolls the page except in full-screen apps.
Typing anywhere on the page goes to the prompt, later to the terminal.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
