The cleanup job wrote the user's grant on C:\msys2 on every run, which goes through all 30000 files and kept the only processor busy for half a minute every half hour. It is now only written when it is not there yet; a run takes 17 seconds instead of up to two minutes. Process creation events now carry the command line, so what a visitor ran is known even when they deleted their shell history along with the installation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
128 lines
8.5 KiB
Markdown
128 lines
8.5 KiB
Markdown
# The shell server
|
|
|
|
The website's terminal connects to a shell server, whose `root` user logs in at `wss://web.nowhere.wirs.ing` and
|
|
gets a shell on the box.
|
|
|
|
## The security model
|
|
|
|
The box is an isolated, disposable VPS at a random hoster. We assume that it can be compromised and that this is
|
|
acceptable: the worst case is that the gimmick on the website stops working. Nothing else is on the box. The
|
|
website and the websocket bridge that carries the connection run on other servers, so a compromised shell server
|
|
cannot get at them.
|
|
|
|
The `root` user on the box is a normal Windows account without administrator rights. It may do a lot of things
|
|
there, including things we did not plan for. We do not try to contain it and we do not try to undo what it does.
|
|
We only keep the basic user journey working.
|
|
|
|
## What we keep working
|
|
|
|
Anything that breaks the following user journey is not acceptable:
|
|
|
|
- The user can connect with the SSH key as `root`.
|
|
- The prompt: `root@khaos-im-kiez:~#`, which `~/.zsh/.zshrc` sets.
|
|
- The home directory the user ends up in: `/home/root`. They cannot change where it is, and they cannot delete it.
|
|
- `cat README`, `cat README.en`, `cat CODE_OF_CONDUCT` and `cat CODE_OF_CONDUCT.en` always work. The four files
|
|
are always there and always regular text files.
|
|
|
|
So in summary running `ssh root@nowhere.wirs.ing cat README` must always succeed and display the contents of the file.
|
|
|
|
Anything thats not specified above (like the content of README for example) is NOT part of the user journey.
|
|
|
|
## What we do not keep working
|
|
|
|
As long as the user story above does not break, we do not guarantee the functionality of anything else on the server. However the cleanup task should still try to best-effort restore everything to a working condition.
|
|
|
|
## Packages
|
|
|
|
The root user owns the whole MSYS2 installation: they can install packages with pacman, and change and delete
|
|
everything in it. Protected is only what the sshd trusts, because the sshd runs as SYSTEM and a file it trusts that
|
|
the user wrote would hand them the box:
|
|
|
|
- the sshd, the login shell `zsh` and `cat`: a second copy in `C:\msys2\opt\kik`, which the server runs from and
|
|
whose `zsh` is the login shell, so the journey keeps working no matter what the user does to their own `/usr/bin`
|
|
- the runtime `/usr/bin/msys-2.0.dll`
|
|
- `/etc/passwd` and `/etc/group`, which decide which Windows account a login becomes - a changed one could make the
|
|
website's key log in as the Administrator
|
|
- `/etc/fstab`, `/etc/fstab.d` and `/etc/nsswitch.conf`, which decide which files those paths are and how accounts
|
|
are looked up
|
|
- `/etc/ssh`, the configuration and the host keys of the server
|
|
|
|
The user may read these at most. Their folders - `C:\msys2`, `etc`, `usr`, `usr\bin` and `opt` - belong to the
|
|
administrators and cannot be renamed, or the user could put a folder of their own in their place. The user has
|
|
*modify* rights on the installation, not full control: full control would also let them delete files whose own
|
|
permissions forbid it. That is how one visitor deleted the locked `/etc/passwd` along with the whole installation.
|
|
|
|
The copy has to be the *same* MSYS2 installation, not a second one: `C:\msys2\opt\kik` is two levels below
|
|
`C:\msys2`, so its runtime resolves the same root and the login shell and the user's own tools share one cygwin
|
|
runtime. A separate installation does not work - two runtimes in one process tree corrupt each other's file
|
|
descriptors and pacman cannot even lock its database. That is also why the runtime itself is not copied but locked
|
|
where it is: a program and the shell that starts it have to use the same runtime DLL. The price is that the user
|
|
cannot update `msys2-runtime`; a deployment brings a new one.
|
|
|
|
All protected files come from a pristine installation in `C:\ProgramData\kik-restore\msys2`, built by the
|
|
installer from the official MSYS2 base archive, whose hash is pinned in the installer, and the packages of the
|
|
site. Neither the tasks nor the installer ever run or trust anything from `C:\msys2`: a program the user replaced
|
|
would run as SYSTEM. For the same reason the Administrator's sessions only search the folders of Windows for
|
|
programs - they would otherwise inherit the server's path, which leads into `/usr/bin`. When the user wipes the installation - when pacman is gone - the cleanup job copies everything
|
|
that is missing back from the pristine one, so the next visitor finds a working system again.
|
|
|
|
## The cleanup tasks
|
|
|
|
Three scheduled tasks run as SYSTEM, all at startup, and the user cannot stop any of them:
|
|
|
|
- `KiK sshd` runs the sshd from `C:\msys2\opt\kik`. It replaces the installation's own `msys2-sshd` service, which
|
|
is disabled.
|
|
- `KiK restore shell files` runs every 30 minutes and puts the protected files back from the pristine
|
|
installation, re-applies the permissions, and puts the home directory, the shell configuration, the website's
|
|
key and the four texts back. Its second job is to best-effort revert other mockery and bring a fresh server to
|
|
the expected state, which includes copying back an installation the user wiped.
|
|
- `KiK watch the server` runs every two minutes and starts the server when nothing listens on port 22. The task of
|
|
the server only starts it again when it failed, never when it was stopped, so a cleanup job that is killed while
|
|
it has the server stopped would otherwise take the site down with it.
|
|
|
|
The cleanup job only restarts the server when it has to: when it does not listen at all, when it is older than its
|
|
configuration, or when a deployment brought a new protected copy or runtime - those cannot be replaced while the
|
|
server holds them open. A working server is left alone, because every start can fail, and failed starts have kept
|
|
the site down for hours. The job also never restarts the server while somebody is connected: a session that is open
|
|
keeps the port, so a server that is started then could not bind it.
|
|
|
|
The server is only down for the short part of the job that needs it down: the protected copy cannot be replaced
|
|
while the server holds it open. Everything else - the home directory, the four texts, the permissions of the
|
|
installation - is put back while the server runs, because on a slow disk that takes minutes and the site should
|
|
have a shell the whole time.
|
|
|
|
The task that runs the server ignores a start while it thinks it is still running, so a start that is dropped
|
|
leaves the site without a shell. The cleanup job therefore tries three times, killing the processes of the task
|
|
in between, before it gives up.
|
|
|
|
The deployment does not restart the server. It runs in a session, and that session keeps the port, so a server that
|
|
starts during a deployment cannot bind it. The new configuration and the restored programs therefore take effect
|
|
at the next run of the cleanup job, which happens within half an hour.
|
|
|
|
The server logs to `C:\ProgramData\kik-restore\sshd.log`. The log is appended to, not overwritten, because it is
|
|
the only trace of what the server did: a server that is restarted would otherwise take the evidence of an outage
|
|
with it. It is rotated when it grows past a hundred megabytes, which takes years. The event logs of windows are
|
|
made bigger as well, the security log to a gigabyte: the scanners of the remote desktop alone fill a hundred
|
|
megabytes of it in a week.
|
|
|
|
A start of the server that fails leaves nothing in that log. So windows records the history of the scheduled tasks
|
|
and the start and end of every process, with its command line, and when the server does not come up, the cleanup job writes the processes
|
|
and the state of the server's task to `C:\ProgramData\kik-restore\failed-starts.log`.
|
|
|
|
Only keys may log in over SSH. The box stands in the open internet and both of its accounts have a password.
|
|
|
|
The files that have to stay belong to the administrators, so the user cannot delete them, and they cannot change the
|
|
shell configuration or the key the website logs in with either. The shell reads its configuration from `~/.zsh`, so
|
|
the startup files the user could write in the home directory are ignored.
|
|
|
|
## Installing the cleanup on the server
|
|
|
|
To deploy the cleanup task an the server, run:
|
|
|
|
server/scripts/deploy.sh
|
|
|
|
The script logs in as `Administrator` - whose shell is PowerShell, which nothing in the installation can replace -
|
|
copies the scripts and the resources to the Administrator's home directory and runs the installer there.
|
|
|
|
When SSH is down, `server/scripts/rdp.sh` opens a remote desktop session as Administrator on a virtual display, with
|
|
a local directory shared into it. The password goes in `.password.txt` in the root of the repository.
|