cccb-website-team/kik-website
Watch
0
0
Fork
You've already forked kik-website
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Rebuild the shell server from a pristine installation it cannot break

Browse source 
On 6 Nov a visitor deleted the whole MSYS2 installation, the locked
/etc/passwd included, and the site was down for a week: the cleanup job
repaired the runtime with pacman, which was gone too, and never tried to
start the server again because a port-22 query without results counted as
a failed query.

The installer now builds a pristine installation from the official base
archive (hash pinned) and the site's packages in the administrators'
folder. Everything the SYSTEM server trusts comes from there and is kept
in place and locked by the cleanup job: opt/kik, the runtime, passwd,
group, nsswitch.conf, fstab, fstab.d and etc/ssh. The user gets modify
rather than full control, so locked files can no longer be deleted, and
the folders leading to them cannot be renamed. When pacman is gone the
job copies the missing files back from the pristine installation.

Nothing that runs as SYSTEM or Administrator runs a file the user can
write any more: the job ran /usr/bin/pacman.exe as SYSTEM, the installer
ran ldd and pacman from the installation, the Administrator's shell was
a cmdshell.exe in /usr/local/bin, and the Administrator's sessions
searched /usr/bin for programs. The Administrator's shell is PowerShell
now, and deploy.sh speaks it.

server/scripts/rdp.sh opens a headless remote desktop session for when
SSH is down. The shell finds the CLANG64 programs and runs helix through
winpty.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zebreus 2026-11-14 02:19:41 +01:00
commit b142a1c5d2
1 changed files with 290 additions and 136 deletions
Download patch file Download diff file Expand all files Collapse all files

58
server/README.md
View file

@ -34,27 +34,37 @@ As long as the user story above does not break, we do not guarantee the function
## Packages
The root user owns the whole MSYS2 installation and can install packages with pacman. That means they can also
replace or delete the very programs the login needs. So the sshd, the login shell `zsh` and `cat` are kept as a
second, protected copy in `C:\msys2\opt\kik`, which only SYSTEM and the administrators may write. The sshd runs as
SYSTEM from that copy and the login shell is that copy's `zsh`, with `/opt/kik` first on the path, so the journey
keeps working no matter what the user does to their own `/usr/bin`. Replacing `/usr/bin/sshd` cannot touch the
running server, so the key on the website still never grants administrator access.
The root user owns the whole MSYS2 installation: they can install packages with pacman, and change and delete
everything in it. Protected is only what the sshd trusts, because the sshd runs as SYSTEM and a file it trusts that
the user wrote would hand them the box:
- the sshd, the login shell `zsh` and `cat`: a second copy in `C:\msys2\opt\kik`, which the server runs from and
whose `zsh` is the login shell, so the journey keeps working no matter what the user does to their own `/usr/bin`
- the runtime `/usr/bin/msys-2.0.dll`
- `/etc/passwd` and `/etc/group`, which decide which Windows account a login becomes - a changed one could make the
website's key log in as the Administrator
- `/etc/fstab`, `/etc/fstab.d` and `/etc/nsswitch.conf`, which decide which files those paths are and how accounts
are looked up
- `/etc/ssh`, the configuration and the host keys of the server
The user may read these at most. Their folders - `C:\msys2`, `etc`, `usr`, `usr\bin` and `opt` - belong to the
administrators and cannot be renamed, or the user could put a folder of their own in their place. The user has
*modify* rights on the installation, not full control: full control would also let them delete files whose own
permissions forbid it. That is how one visitor deleted the locked `/etc/passwd` along with the whole installation.
The copy has to be the *same* MSYS2 installation, not a second one: `C:\msys2\opt\kik` is two levels below
`C:\msys2`, so its runtime resolves the same root and the login shell and the user's own tools share one cygwin
runtime. A separate installation does not work - two runtimes in one process tree corrupt each other's file
descriptors and pacman cannot even lock its database.
descriptors and pacman cannot even lock its database. That is also why the runtime itself is not copied but locked
where it is: a program and the shell that starts it have to use the same runtime DLL. The price is that the user
cannot update `msys2-runtime`; a deployment brings a new one.
The runtime itself is not among the protected copies. A program and the shell that starts it have to use the same
runtime DLL, or the program does not run at all - a second copy of it in one process is refused. The login path
therefore uses the one in `/usr/bin`, next to the user's own programs, and the user may replace and upgrade it.
What is protected instead is that it is the one the package says it is: pacman checks the files of `msys2-runtime`,
and the cleanup job puts the package back if it was changed - while the server is stopped, before it starts it
again, because the server runs as SYSTEM and would otherwise run code the user of the website wrote.
`/etc/passwd` (the login shell and home directory) and `/etc/ssh` (the host keys and config) are read-only to the
user as well; everything else in the installation is theirs.
All protected files come from a pristine installation in `C:\ProgramData\kik-restore\msys2`, built by the
installer from the official MSYS2 base archive, whose hash is pinned in the installer, and the packages of the
site. Neither the tasks nor the installer ever run or trust anything from `C:\msys2`: a program the user replaced
would run as SYSTEM. For the same reason the Administrator's sessions only search the folders of Windows for
programs - they would otherwise inherit the server's path, which leads into `/usr/bin`. When the user wipes the installation - when pacman is gone - the cleanup job copies everything
that is missing back from the pristine one, so the next visitor finds a working system again.
## The cleanup tasks
@ -62,15 +72,16 @@ Three scheduled tasks run as SYSTEM, all at startup, and the user cannot stop an
- `KiK sshd` runs the sshd from `C:\msys2\opt\kik`. It replaces the installation's own `msys2-sshd` service, which
is disabled.
- `KiK restore shell files` runs every 30 minutes and rebuilds the protected copy from its trusted originals,
re-applies the permissions, and puts the home directory, the shell configuration, the website's key and the four
texts back. Its second job is to best-effort revert other mockery and bring a fresh server to the expected state.
- `KiK restore shell files` runs every 30 minutes and puts the protected files back from the pristine
installation, re-applies the permissions, and puts the home directory, the shell configuration, the website's
key and the four texts back. Its second job is to best-effort revert other mockery and bring a fresh server to
the expected state, which includes copying back an installation the user wiped.
- `KiK watch the server` runs every two minutes and starts the server when nothing listens on port 22. The task of
the server only starts it again when it failed, never when it was stopped, so a cleanup job that is killed while
it has the server stopped would otherwise take the site down with it.
The cleanup job only restarts the server when it has to: when it does not listen at all, when it is older than its
configuration, or when the protected copy or the runtime has to be repaired - those cannot be replaced while the
configuration, or when a deployment brought a new protected copy or runtime - those cannot be replaced while the
server holds them open. A working server is left alone, because every start can fail, and failed starts have kept
the site down for hours. The job also never restarts the server while somebody is connected: a session that is open
keeps the port, so a server that is started then could not bind it.
@ -110,5 +121,8 @@ To deploy the cleanup task an the server, run:
server/scripts/deploy.sh
The script logs in as `Administrator` - whose shell is a Windows one - copies the scripts and the resources to the
Administrator's home directory and runs the installer there.
The script logs in as `Administrator` - whose shell is PowerShell, which nothing in the installation can replace -
copies the scripts and the resources to the Administrator's home directory and runs the installer there.
When SSH is down, `server/scripts/rdp.sh` opens a remote desktop session as Administrator on a virtual display, with
a local directory shared into it. The password goes in `.password.txt` in the root of the repository.