b142a1c5d2
On 6 Nov a visitor deleted the whole MSYS2 installation, the locked /etc/passwd included, and the site was down for a week: the cleanup job repaired the runtime with pacman, which was gone too, and never tried to start the server again because a port-22 query without results counted as a failed query. The installer now builds a pristine installation from the official base archive (hash pinned) and the site's packages in the administrators' folder. Everything the SYSTEM server trusts comes from there and is kept in place and locked by the cleanup job: opt/kik, the runtime, passwd, group, nsswitch.conf, fstab, fstab.d and etc/ssh. The user gets modify rather than full control, so locked files can no longer be deleted, and the folders leading to them cannot be renamed. When pacman is gone the job copies the missing files back from the pristine installation. Nothing that runs as SYSTEM or Administrator runs a file the user can write any more: the job ran /usr/bin/pacman.exe as SYSTEM, the installer ran ldd and pacman from the installation, the Administrator's shell was a cmdshell.exe in /usr/local/bin, and the Administrator's sessions searched /usr/bin for programs. The Administrator's shell is PowerShell now, and deploy.sh speaks it. server/scripts/rdp.sh opens a headless remote desktop session for when SSH is down. The shell finds the CLANG64 programs and runs helix through winpty. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
128 lines
8.4 KiB
Markdown
128 lines
8.4 KiB
Markdown
# The shell server
|
|
|
|
The website's terminal connects to a shell server, whose `root` user logs in at `wss://web.nowhere.wirs.ing` and
|
|
gets a shell on the box.
|
|
|
|
## The security model
|
|
|
|
The box is an isolated, disposable VPS at a random hoster. We assume that it can be compromised and that this is
|
|
acceptable: the worst case is that the gimmick on the website stops working. Nothing else is on the box. The
|
|
website and the websocket bridge that carries the connection run on other servers, so a compromised shell server
|
|
cannot get at them.
|
|
|
|
The `root` user on the box is a normal Windows account without administrator rights. It may do a lot of things
|
|
there, including things we did not plan for. We do not try to contain it and we do not try to undo what it does.
|
|
We only keep the basic user journey working.
|
|
|
|
## What we keep working
|
|
|
|
Anything that breaks the following user journey is not acceptable:
|
|
|
|
- The user can connect with the SSH key as `root`.
|
|
- The prompt: `root@khaos-im-kiez:~#`, which `~/.zsh/.zshrc` sets.
|
|
- The home directory the user ends up in: `/home/root`. They cannot change where it is, and they cannot delete it.
|
|
- `cat README`, `cat README.en`, `cat CODE_OF_CONDUCT` and `cat CODE_OF_CONDUCT.en` always work. The four files
|
|
are always there and always regular text files.
|
|
|
|
So in summary running `ssh root@nowhere.wirs.ing cat README` must always succeed and display the contents of the file.
|
|
|
|
Anything thats not specified above (like the content of README for example) is NOT part of the user journey.
|
|
|
|
## What we do not keep working
|
|
|
|
As long as the user story above does not break, we do not guarantee the functionality of anything else on the server. However the cleanup task should still try to best-effort restore everything to a working condition.
|
|
|
|
## Packages
|
|
|
|
The root user owns the whole MSYS2 installation: they can install packages with pacman, and change and delete
|
|
everything in it. Protected is only what the sshd trusts, because the sshd runs as SYSTEM and a file it trusts that
|
|
the user wrote would hand them the box:
|
|
|
|
- the sshd, the login shell `zsh` and `cat`: a second copy in `C:\msys2\opt\kik`, which the server runs from and
|
|
whose `zsh` is the login shell, so the journey keeps working no matter what the user does to their own `/usr/bin`
|
|
- the runtime `/usr/bin/msys-2.0.dll`
|
|
- `/etc/passwd` and `/etc/group`, which decide which Windows account a login becomes - a changed one could make the
|
|
website's key log in as the Administrator
|
|
- `/etc/fstab`, `/etc/fstab.d` and `/etc/nsswitch.conf`, which decide which files those paths are and how accounts
|
|
are looked up
|
|
- `/etc/ssh`, the configuration and the host keys of the server
|
|
|
|
The user may read these at most. Their folders - `C:\msys2`, `etc`, `usr`, `usr\bin` and `opt` - belong to the
|
|
administrators and cannot be renamed, or the user could put a folder of their own in their place. The user has
|
|
*modify* rights on the installation, not full control: full control would also let them delete files whose own
|
|
permissions forbid it. That is how one visitor deleted the locked `/etc/passwd` along with the whole installation.
|
|
|
|
The copy has to be the *same* MSYS2 installation, not a second one: `C:\msys2\opt\kik` is two levels below
|
|
`C:\msys2`, so its runtime resolves the same root and the login shell and the user's own tools share one cygwin
|
|
runtime. A separate installation does not work - two runtimes in one process tree corrupt each other's file
|
|
descriptors and pacman cannot even lock its database. That is also why the runtime itself is not copied but locked
|
|
where it is: a program and the shell that starts it have to use the same runtime DLL. The price is that the user
|
|
cannot update `msys2-runtime`; a deployment brings a new one.
|
|
|
|
All protected files come from a pristine installation in `C:\ProgramData\kik-restore\msys2`, built by the
|
|
installer from the official MSYS2 base archive, whose hash is pinned in the installer, and the packages of the
|
|
site. Neither the tasks nor the installer ever run or trust anything from `C:\msys2`: a program the user replaced
|
|
would run as SYSTEM. For the same reason the Administrator's sessions only search the folders of Windows for
|
|
programs - they would otherwise inherit the server's path, which leads into `/usr/bin`. When the user wipes the installation - when pacman is gone - the cleanup job copies everything
|
|
that is missing back from the pristine one, so the next visitor finds a working system again.
|
|
|
|
## The cleanup tasks
|
|
|
|
Three scheduled tasks run as SYSTEM, all at startup, and the user cannot stop any of them:
|
|
|
|
- `KiK sshd` runs the sshd from `C:\msys2\opt\kik`. It replaces the installation's own `msys2-sshd` service, which
|
|
is disabled.
|
|
- `KiK restore shell files` runs every 30 minutes and puts the protected files back from the pristine
|
|
installation, re-applies the permissions, and puts the home directory, the shell configuration, the website's
|
|
key and the four texts back. Its second job is to best-effort revert other mockery and bring a fresh server to
|
|
the expected state, which includes copying back an installation the user wiped.
|
|
- `KiK watch the server` runs every two minutes and starts the server when nothing listens on port 22. The task of
|
|
the server only starts it again when it failed, never when it was stopped, so a cleanup job that is killed while
|
|
it has the server stopped would otherwise take the site down with it.
|
|
|
|
The cleanup job only restarts the server when it has to: when it does not listen at all, when it is older than its
|
|
configuration, or when a deployment brought a new protected copy or runtime - those cannot be replaced while the
|
|
server holds them open. A working server is left alone, because every start can fail, and failed starts have kept
|
|
the site down for hours. The job also never restarts the server while somebody is connected: a session that is open
|
|
keeps the port, so a server that is started then could not bind it.
|
|
|
|
The server is only down for the short part of the job that needs it down: the protected copy cannot be replaced
|
|
while the server holds it open. Everything else - the home directory, the four texts, the permissions of the
|
|
installation - is put back while the server runs, because on a slow disk that takes minutes and the site should
|
|
have a shell the whole time.
|
|
|
|
The task that runs the server ignores a start while it thinks it is still running, so a start that is dropped
|
|
leaves the site without a shell. The cleanup job therefore tries three times, killing the processes of the task
|
|
in between, before it gives up.
|
|
|
|
The deployment does not restart the server. It runs in a session, and that session keeps the port, so a server that
|
|
starts during a deployment cannot bind it. The new configuration and the restored programs therefore take effect
|
|
at the next run of the cleanup job, which happens within half an hour.
|
|
|
|
The server logs to `C:\ProgramData\kik-restore\sshd.log`. The log is appended to, not overwritten, because it is
|
|
the only trace of what the server did: a server that is restarted would otherwise take the evidence of an outage
|
|
with it. It is rotated when it grows past a hundred megabytes, which takes years. The event logs of windows are
|
|
made bigger as well, the security log to a gigabyte: the scanners of the remote desktop alone fill a hundred
|
|
megabytes of it in a week.
|
|
|
|
A start of the server that fails leaves nothing in that log. So windows records the history of the scheduled tasks
|
|
and the start and end of every process, and when the server does not come up, the cleanup job writes the processes
|
|
and the state of the server's task to `C:\ProgramData\kik-restore\failed-starts.log`.
|
|
|
|
Only keys may log in over SSH. The box stands in the open internet and both of its accounts have a password.
|
|
|
|
The files that have to stay belong to the administrators, so the user cannot delete them, and they cannot change the
|
|
shell configuration or the key the website logs in with either. The shell reads its configuration from `~/.zsh`, so
|
|
the startup files the user could write in the home directory are ignored.
|
|
|
|
## Installing the cleanup on the server
|
|
|
|
To deploy the cleanup task an the server, run:
|
|
|
|
server/scripts/deploy.sh
|
|
|
|
The script logs in as `Administrator` - whose shell is PowerShell, which nothing in the installation can replace -
|
|
copies the scripts and the resources to the Administrator's home directory and runs the installer there.
|
|
|
|
When SSH is down, `server/scripts/rdp.sh` opens a remote desktop session as Administrator on a virtual display, with
|
|
a local directory shared into it. The password goes in `.password.txt` in the root of the repository.
|