Rebuild the shell server from a pristine installation it cannot break

On 6 Nov a visitor deleted the whole MSYS2 installation, the locked
/etc/passwd included, and the site was down for a week: the cleanup job
repaired the runtime with pacman, which was gone too, and never tried to
start the server again because a port-22 query without results counted as
a failed query.

The installer now builds a pristine installation from the official base
archive (hash pinned) and the site's packages in the administrators'
folder. Everything the SYSTEM server trusts comes from there and is kept
in place and locked by the cleanup job: opt/kik, the runtime, passwd,
group, nsswitch.conf, fstab, fstab.d and etc/ssh. The user gets modify
rather than full control, so locked files can no longer be deleted, and
the folders leading to them cannot be renamed. When pacman is gone the
job copies the missing files back from the pristine installation.

Nothing that runs as SYSTEM or Administrator runs a file the user can
write any more: the job ran /usr/bin/pacman.exe as SYSTEM, the installer
ran ldd and pacman from the installation, the Administrator's shell was
a cmdshell.exe in /usr/local/bin, and the Administrator's sessions
searched /usr/bin for programs. The Administrator's shell is PowerShell
now, and deploy.sh speaks it.

server/scripts/rdp.sh opens a headless remote desktop session for when
SSH is down. The shell finds the CLANG64 programs and runs helix through
winpty.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitignore

@@ -1,3 +1,6 @@
 node_modules/
 dist/
 .astro/
+
+# the Administrator password server/scripts/rdp.sh reads
+.password.txt

server/README.md

@@ -34,27 +34,37 @@ As long as the user story above does not break, we do not guarantee the function
 
 ## Packages
 
-The root user owns the whole MSYS2 installation and can install packages with pacman. That means they can also
+The root user owns the whole MSYS2 installation: they can install packages with pacman, and change and delete
-replace or delete the very programs the login needs. So the sshd, the login shell `zsh` and `cat` are kept as a
+everything in it. Protected is only what the sshd trusts, because the sshd runs as SYSTEM and a file it trusts that
-second, protected copy in `C:\msys2\opt\kik`, which only SYSTEM and the administrators may write. The sshd runs as
+the user wrote would hand them the box:
-SYSTEM from that copy and the login shell is that copy's `zsh`, with `/opt/kik` first on the path, so the journey
+
-keeps working no matter what the user does to their own `/usr/bin`. Replacing `/usr/bin/sshd` cannot touch the
+- the sshd, the login shell `zsh` and `cat`: a second copy in `C:\msys2\opt\kik`, which the server runs from and
-running server, so the key on the website still never grants administrator access.
+  whose `zsh` is the login shell, so the journey keeps working no matter what the user does to their own `/usr/bin`
+- the runtime `/usr/bin/msys-2.0.dll`
+- `/etc/passwd` and `/etc/group`, which decide which Windows account a login becomes - a changed one could make the
+  website's key log in as the Administrator
+- `/etc/fstab`, `/etc/fstab.d` and `/etc/nsswitch.conf`, which decide which files those paths are and how accounts
+  are looked up
+- `/etc/ssh`, the configuration and the host keys of the server
+
+The user may read these at most. Their folders - `C:\msys2`, `etc`, `usr`, `usr\bin` and `opt` - belong to the
+administrators and cannot be renamed, or the user could put a folder of their own in their place. The user has
+*modify* rights on the installation, not full control: full control would also let them delete files whose own
+permissions forbid it. That is how one visitor deleted the locked `/etc/passwd` along with the whole installation.
 
 The copy has to be the *same* MSYS2 installation, not a second one: `C:\msys2\opt\kik` is two levels below
 `C:\msys2`, so its runtime resolves the same root and the login shell and the user's own tools share one cygwin
 runtime. A separate installation does not work - two runtimes in one process tree corrupt each other's file
-descriptors and pacman cannot even lock its database.
+descriptors and pacman cannot even lock its database. That is also why the runtime itself is not copied but locked
+where it is: a program and the shell that starts it have to use the same runtime DLL. The price is that the user
+cannot update `msys2-runtime`; a deployment brings a new one.
 
-The runtime itself is not among the protected copies. A program and the shell that starts it have to use the same
+All protected files come from a pristine installation in `C:\ProgramData\kik-restore\msys2`, built by the
-runtime DLL, or the program does not run at all - a second copy of it in one process is refused. The login path
+installer from the official MSYS2 base archive, whose hash is pinned in the installer, and the packages of the
-therefore uses the one in `/usr/bin`, next to the user's own programs, and the user may replace and upgrade it.
+site. Neither the tasks nor the installer ever run or trust anything from `C:\msys2`: a program the user replaced
-What is protected instead is that it is the one the package says it is: pacman checks the files of `msys2-runtime`,
+would run as SYSTEM. For the same reason the Administrator's sessions only search the folders of Windows for
-and the cleanup job puts the package back if it was changed - while the server is stopped, before it starts it
+programs - they would otherwise inherit the server's path, which leads into `/usr/bin`. When the user wipes the installation - when pacman is gone - the cleanup job copies everything
-again, because the server runs as SYSTEM and would otherwise run code the user of the website wrote.
+that is missing back from the pristine one, so the next visitor finds a working system again.
-
-`/etc/passwd` (the login shell and home directory) and `/etc/ssh` (the host keys and config) are read-only to the
-user as well; everything else in the installation is theirs.
 
 ## The cleanup tasks
 
@@ -62,15 +72,16 @@ Three scheduled tasks run as SYSTEM, all at startup, and the user cannot stop an
 
 - `KiK sshd` runs the sshd from `C:\msys2\opt\kik`. It replaces the installation's own `msys2-sshd` service, which
   is disabled.
-- `KiK restore shell files` runs every 30 minutes and rebuilds the protected copy from its trusted originals,
+- `KiK restore shell files` runs every 30 minutes and puts the protected files back from the pristine
-  re-applies the permissions, and puts the home directory, the shell configuration, the website's key and the four
+  installation, re-applies the permissions, and puts the home directory, the shell configuration, the website's
-  texts back. Its second job is to best-effort revert other mockery and bring a fresh server to the expected state.
+  key and the four texts back. Its second job is to best-effort revert other mockery and bring a fresh server to
+  the expected state, which includes copying back an installation the user wiped.
 - `KiK watch the server` runs every two minutes and starts the server when nothing listens on port 22. The task of
   the server only starts it again when it failed, never when it was stopped, so a cleanup job that is killed while
   it has the server stopped would otherwise take the site down with it.
 
 The cleanup job only restarts the server when it has to: when it does not listen at all, when it is older than its
-configuration, or when the protected copy or the runtime has to be repaired - those cannot be replaced while the
+configuration, or when a deployment brought a new protected copy or runtime - those cannot be replaced while the
 server holds them open. A working server is left alone, because every start can fail, and failed starts have kept
 the site down for hours. The job also never restarts the server while somebody is connected: a session that is open
 keeps the port, so a server that is started then could not bind it.
@@ -110,5 +121,8 @@ To deploy the cleanup task an the server, run:
 
     server/scripts/deploy.sh
 
-The script logs in as `Administrator` - whose shell is a Windows one - copies the scripts and the resources to the
+The script logs in as `Administrator` - whose shell is PowerShell, which nothing in the installation can replace -
-Administrator's home directory and runs the installer there.
+copies the scripts and the resources to the Administrator's home directory and runs the installer there.
+
+When SSH is down, `server/scripts/rdp.sh` opens a remote desktop session as Administrator on a virtual display, with
+a local directory shared into it. The password goes in `.password.txt` in the root of the repository.

server/resources/home/.zsh/.zshrc

@@ -1,6 +1,8 @@
 # The protected copies of zsh and cat (and the server itself) live here; keeping them first means the basic
 # journey - the prompt and reading the four texts - survives even if the user breaks their own /usr/bin copies.
 path=(/opt/kik $path)
+# Programs of MSYS2's CLANG64 environment (pacman -S mingw-w64-clang-x86_64-...), after the MSYS2 ones.
+path+=/clang64/bin
 
 # History shared between sessions.
 HISTFILE=~/.zsh_history HISTSIZE=10000 SAVEHIST=10000
@@ -36,5 +38,6 @@ PROMPT='%F{green}%n@%m%f:%F{blue}%~%f${git_branch:+ %5F($git_branch)%f}%(?.. %F{
 alias ls='ls --color=auto' ll='ls -lah' grep='grep --color=auto'
 
 # Process monitor: htop does not exist for MSYS2 (its /proc only shows MSYS2 processes).
-# bottom is a native Windows program; winpty lets it draw in the pty, as Server 2016 has no ConPTY.
+# bottom and helix are native Windows programs; winpty lets them draw in the pty and read the keys, as Server 2016
-alias btm='winpty /ucrt64/bin/btm'
+# has no ConPTY.
+alias btm='winpty /ucrt64/bin/btm' hx='winpty /clang64/bin/hx'

server/scripts/deploy.sh

@@ -3,8 +3,8 @@
 #
 #   server/scripts/deploy.sh [-i KEYFILE] [-h HOST] [--uninstall]
 #
-# The job runs on the box as SYSTEM, never as the root user. The Administrator account sets it up; its shell is a
+# The job runs on the box as SYSTEM, never as the root user. The Administrator account sets it up; its shell is
-# Windows one, so the commands we run there are Windows commands.
+# PowerShell, so the commands we run there are PowerShell commands.
 #
 # The Administrator logs in with the personal key of the deployer, from the agent or from -i. The key of the website
 # never gets administrator rights: it only logs in as the root user.
@@ -55,13 +55,12 @@ ssh=(ssh "${options[@]}" "$USER@$HOST")
 scp=(scp "${options[@]}")
 
 if [ "$UNINSTALL" = 1 ]; then
-    "${ssh[@]}" "if exist $remote\\scripts\\install_restore_task.ps1 (powershell -NoProfile -ExecutionPolicy Bypass -File $remote\\scripts\\install_restore_task.ps1 -Uninstall) else (echo Nothing is installed.)"
+    "${ssh[@]}" "if (Test-Path $remote\\scripts\\install_restore_task.ps1) { powershell -NoProfile -ExecutionPolicy Bypass -File $remote\\scripts\\install_restore_task.ps1 -Uninstall } else { 'Nothing is installed.' }"
     exit
 fi
 
 # The installer looks for the resources next to itself, so both directories go to the server. What was there
 # before is removed, so that files that are gone from the repository do not stay on the box.
-"${ssh[@]}" "if exist $remote rmdir /s /q $remote"
+"${ssh[@]}" "Remove-Item -Recurse -Force -ErrorAction SilentlyContinue $remote; New-Item -ItemType Directory $remote | Out-Null"
-"${ssh[@]}" "mkdir $remote"
 "${scp[@]}" -r "$server/scripts" "$server/resources" "$USER@$HOST:$remotePath/"
 "${ssh[@]}" "powershell -NoProfile -ExecutionPolicy Bypass -File $remote\\scripts\\install_restore_task.ps1"

server/scripts/install_restore_task.ps1

@@ -1,11 +1,15 @@
 # Sets up the shell server so the website's root user can install packages with pacman, while the basic user
 # journey - logging in with the key, the prompt, the home directory and the four text files - cannot be broken.
 #
-# The whole MSYS2 installation belongs to the root user: they may write everywhere and run pacman. The login path
+# The whole MSYS2 installation belongs to the root user: they may change and delete everything in it and run pacman.
-# the journey depends on - the sshd, the login shell zsh and cat - is kept as a protected copy in C:\msys2\opt\kik
+# Only what the sshd, which runs as SYSTEM, trusts is protected: the sshd itself, the login shell zsh and cat (a copy
-# that only SYSTEM and the administrators may write. The sshd runs as SYSTEM from that copy (the scheduled task
+# in C:\msys2\opt\kik, the scheduled task "KiK sshd" runs it), the runtime, the account database, the mount table
-# "KiK sshd"), so replacing /usr/bin/sshd does not touch the running server. A second scheduled task,
+# and the ssh configuration. A second scheduled task, "KiK restore shell files", keeps those and the home directory
-# "KiK restore shell files", puts the home directory, the four texts and the protected copy back every 30 minutes.
+# with its four texts in place every 30 minutes.
+#
+# All of it comes from a pristine installation, built from the official MSYS2 archive in C:\ProgramData\kik-restore
+# where the user cannot reach it. Nothing the user can write is ever run or trusted by the tasks or by this script.
+# When the user wipes the installation, the restore task copies it back from the pristine one.
 #
 # opt/kik works because its MSYS2 runtime resolves the same installation root (C:\msys2) as /usr/bin, so the login
 # shell and the user's own tools share one cygwin runtime. A separate second installation cannot: two runtimes in
@@ -27,6 +31,7 @@ $kik = "$msys\opt\kik"
 $installDir = 'C:\ProgramData\kik-restore'
 $copiesDir = Join-Path $installDir 'files'
 $binDir = Join-Path $installDir 'bin'
+$pristine = Join-Path $installDir 'msys2'
 $sshdBat = Join-Path $installDir 'run-sshd.bat'
 $restoreTask = 'KiK restore shell files'
 $sshdTask = 'KiK sshd'
@@ -35,14 +40,19 @@ $service = 'msys2-sshd'
 
 # The programs the journey needs, kept out of the user's reach. sshd-session and sshd-auth are the per-connection
 # helpers OpenSSH 10 re-executes; zsh is the login shell; cat reads the four texts.
-$loginBinaries = "$msys\usr\bin\sshd.exe", "$msys\usr\lib\ssh\sshd-session.exe", "$msys\usr\lib\ssh\sshd-auth.exe",
+$loginBinaries = "$pristine\usr\bin\sshd.exe", "$pristine\usr\lib\ssh\sshd-session.exe", "$pristine\usr\lib\ssh\sshd-auth.exe",
-    "$msys\usr\bin\zsh.exe", "$msys\usr\bin\cat.exe"
+    "$pristine\usr\bin\zsh.exe", "$pristine\usr\bin\cat.exe"
 
 # The runtime is not among them: a program and the shell that starts it have to use the same runtime, or the
-# program does not run at all. The login path therefore uses the one in /usr/bin, and the user may replace it -
+# program does not run at all. The login path therefore uses the one in /usr/bin, next to the user's programs, which
-# pacman checks that it is the one the package says it is, and the cleanup job puts it back if it is not.
+# is locked instead.
 $runtimeFiles = '/usr/bin/msys-2.0.dll'
 
+# The pristine installation: the official base archive, whose hash is pinned here, and the packages of the site.
+$baseArchive = 'msys2-base-x86_64-20260927.sfx.exe'
+$baseHash = 'AD336CCCFDA47758B5E15CDA993FBBA421115CB0B126697DAEF1EE4DFE37209F'
+$packages = 'openssh zsh vim git tmux rsync wget curl jq python tree zip unzip man-db make diffutils patch procps-ng psmisc winpty mingw-w64-ucrt-x86_64-bottom'
+
 # The copies the task restores from and the script it runs both live in $installDir, which only SYSTEM and the
 # administrators may read or write.
 $restoreScript = @'
@@ -54,9 +64,11 @@ $kik = "$msys\opt\kik"
 $log = Join-Path $PSScriptRoot 'restore.log'
 $copies = Join-Path $PSScriptRoot 'files\home'
 $binDir = Join-Path $PSScriptRoot 'bin'
+# The pristine installation the protected files come from, which the user cannot reach.
+$pristine = Join-Path $PSScriptRoot 'msys2'
 
 # The account of the shell server's root user, whose id the files in its home directory belong to.
-$passwd = Get-Content "$msys\etc\passwd" -ErrorAction SilentlyContinue | Where-Object { $_ -like 'root:*' }
+$passwd = Get-Content "$pristine\etc\passwd" -ErrorAction SilentlyContinue | Where-Object { $_ -like 'root:*' }
 $sid = if ($passwd) { ($passwd -split ':')[4].Split(',')[-1] }
 
 $system = New-Object System.Security.Principal.SecurityIdentifier('S-1-5-18')
@@ -199,34 +211,57 @@ function Start-Sshd {
     return $false
 }
 
-# Runs a program and gives up after a while: pacman can wait for the lock of its database, and the server must not
+# A folder that belongs to the administrators and that the user may use but not delete or rename. Only DELETE is
-# stay down because of it. The exit code is returned, or -1 when it did not finish in time.
+# denied - the deny of icacls also takes SYNCHRONIZE, and without it the user cannot even change into the folder. The
-function Invoke-Bounded($program, $arguments, $seconds) {
+# permissions are only written when they are wrong, because writing them goes through every file below the folder.
-    $process = Start-Process -FilePath $program -ArgumentList $arguments -NoNewWindow -PassThru `
+function Set-Undeletable($path) {
-        -RedirectStandardOutput (Join-Path $env:TEMP 'kik-pacman.out') -RedirectStandardError (Join-Path $env:TEMP 'kik-pacman.err')
+    $user = New-Object System.Security.Principal.SecurityIdentifier($sid)
-    # the handle is taken while the process runs, otherwise its exit code is not available afterwards
+    $acl = Get-Acl -LiteralPath $path
-    $null = $process.Handle
+    $denied = @($acl.GetAccessRules($true, $false, [System.Security.Principal.SecurityIdentifier]) |
-    if (-not $process.WaitForExit($seconds * 1000)) {
+        Where-Object { $_.AccessControlType -eq 'Deny' -and $_.IdentityReference -eq $user })
-        Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue
+    if ($acl.GetOwner([System.Security.Principal.SecurityIdentifier]) -eq $admins -and $denied.Count -eq 1 -and
-        return -1
+        $denied[0].FileSystemRights -eq 'Delete' -and $denied[0].InheritanceFlags -eq 'None') { return }
+    $acl.SetOwner($admins)
+    [void]$acl.RemoveAccessRuleAll((New-Object System.Security.AccessControl.FileSystemAccessRule($user, 'Delete', 'Deny')))
+    $acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($user, 'Delete', 'None', 'None', 'Deny')))
+    Set-Acl -LiteralPath $path -AclObject $acl
+}
+
+# Puts a file or a folder of the installation back to the pristine one where it differs, and locks it: the user may
+# do no more with it than $userRights.
+function Restore-Protected($relative, $userRights) {
+    $source = Join-Path $pristine $relative
+    $target = Join-Path $msys $relative
+    $folder = (Get-Item -LiteralPath $source -Force).PSIsContainer
+    if ($folder) { New-Item -ItemType Directory -Force -Path $target | Out-Null }
+    foreach ($file in @(if ($folder) { Get-ChildItem -LiteralPath $source -Recurse -File -Force } else { Get-Item -LiteralPath $source -Force })) {
+        $copy = $msys + $file.FullName.Substring($pristine.Length)
+        if ((Get-FileHash -LiteralPath $copy -ErrorAction SilentlyContinue).Hash -ne (Get-FileHash -LiteralPath $file.FullName).Hash) {
+            New-Item -ItemType Directory -Force -Path (Split-Path -Parent $copy) | Out-Null
+            Copy-Item -LiteralPath $file.FullName -Destination $copy -Force
+            Write-Log "restored $($copy.Substring($msys.Length + 1) -replace '\\', '/')"
+        }
     }
-    return $process.ExitCode
+    Set-LockedAcl $target $folder $userRights
 }
 
 $netError = $null
-$connections = @(Get-NetTCPConnection -LocalPort 22 -ErrorAction SilentlyContinue -ErrorVariable netError)
+# All connections are asked for: asking for those of port 22 alone fails when there are none, which is exactly when
+# the server is down.
+$connections = @(Get-NetTCPConnection -ErrorAction SilentlyContinue -ErrorVariable netError | Where-Object LocalPort -eq 22)
 # An open session keeps the port, so the server is only stopped when there is none. A query that fails is not a
 # reason to disturb a session. The processes of the sessions are left behind by the server when a client goes away,
 # so they say nothing about whether anybody is connected.
 $busy = @($connections | Where-Object { $_.State -eq 'Established' }).Count -gt 0
 $mayStop = -not $netError -and -not $busy
-# The runtime of the installation belongs to the user's programs and is not protected: a program and the shell that
+
-# starts it have to use the same one, or the program does not run at all. What has to be protected is that it is the
+# A user who wipes the installation takes pacman with it and could not install anything again. When pacman is gone,
-# one the package says it is, because the server runs as SYSTEM. The check takes a while, so it runs while the
+# everything that is missing is copied back from the pristine installation; what the user changed or added stays.
-# server is up - the server must not wait for it.
+# Only then: copying back what is missing on every run would bring back the packages the user removed or upgraded,
-$runtimeBroken = $false
+# and pacman's database would list both versions.
-if ($mayStop) {
+if (-not (Test-Path -LiteralPath "$msys\usr\bin\pacman.exe")) {
-    $runtimeBroken = (Invoke-Bounded "$msys\usr\bin\pacman.exe" @('-Qkk', 'msys2-runtime') 120) -ne 0
+    robocopy $pristine $msys /E /XC /XN /XO /XJ /XD "$pristine\home" "$pristine\tmp" /R:1 /W:1 /NP /NFL /NDL /NJH /NJS | Out-Null
+    Write-Log "copied the installation back from the pristine one (robocopy exit $LASTEXITCODE)"
 }
 
 # The home directory and the four texts: the server does not hold them open, so they are put back while it runs.
@@ -275,18 +310,50 @@ foreach ($copy in Get-ChildItem -LiteralPath $copies -Recurse -File -Force) {
     }
 }
 
-# The whole installation belongs to the user, so pacman can write it. The home directory keeps its own protected
+# The whole installation belongs to the user, so pacman can write it: they may change and delete every file in it.
-# permissions because they are set without inheritance.
+# Modify rather than full control, which would also let them delete files whose own permissions do not allow it - the
-if ($sid) { icacls $msys /grant "*${sid}:(OI)(CI)F" | Out-Null }
+# protected ones below. The folders that lead to those belong to the administrators and cannot be renamed either: a
+# folder put in the place of one would hand the server files the user wrote. The home directory keeps its own
+# protected permissions because they are set without inheritance.
+if ($sid) {
+    icacls $msys /grant:r "*${sid}:(OI)(CI)M" | Out-Null
+    foreach ($folder in $msys, "$msys\etc", "$msys\usr", "$msys\usr\bin", "$msys\opt") {
+        New-Item -ItemType Directory -Force -Path $folder | Out-Null
+        Set-Undeletable $folder
+    }
+}
+
+# The files the server, which runs as SYSTEM, trusts besides its programs: the account database decides which account
+# a login becomes, the mount table and the lookup rules which files those are, and the ssh configuration and keys who
+# may log in. They are kept the pristine ones and the user may at most read them - a changed one would let the
+# website's key in as another account. The server does not hold them open, so this needs no restart.
+try {
+    Restore-Protected 'etc\passwd' 'Read'
+    Restore-Protected 'etc\group' 'Read'
+    Restore-Protected 'etc\nsswitch.conf' 'Read'
+    Restore-Protected 'etc\fstab' 'Read'
+    Restore-Protected 'etc\fstab.d' 'ReadAndExecute'
+    Restore-Protected 'etc\ssh' 'None'
+} catch {
+    Write-Log "the protected files are not fully in place: $_"
+}
+# the server does not start without the directory it drops its privileges into, and the shell wants /tmp, which is
+# not copied from the pristine installation with the rest
+New-Item -ItemType Directory -Force -Path "$msys\var\empty", "$msys\tmp" | Out-Null
+
+# The runtime is locked as well, so it only differs from the pristine one when a deployment brought a new one.
+$runtime = 'usr\bin\msys-2.0.dll'
+$runtimeOld = (Get-FileHash -LiteralPath "$msys\$runtime" -ErrorAction SilentlyContinue).Hash -ne (Get-FileHash -LiteralPath "$pristine\$runtime").Hash
 
 # The protected login path: the copy the sshd and the login shell run from. The user cannot write it, so even after
-# they replace or remove their own copies in /usr/bin the server and the journey keep working. The trusted originals
+# they replace or remove their own copies in /usr/bin the server and the journey keep working. The originals are in
-# are in bin\; anything missing from opt/kik or different from its original is put back. Replacing them is the only
+# bin\, taken from the pristine installation; anything missing from opt/kik or different from its original is put
-# work that needs the server to be stopped, because a running server holds its programs open.
+# back. Replacing them and the runtime is the only work that needs the server to be stopped, because a running server
+# holds them open.
 $stale = @(if (Test-Path -LiteralPath $binDir) {
     Get-ChildItem -LiteralPath $binDir -File -Force | Where-Object {
         $target = Join-Path $kik $_.Name
-        $_.Name -ne 'msys-2.0.dll' -and (-not (Test-Path -LiteralPath $target) -or (Get-Item -LiteralPath $target -Force).Length -ne $_.Length)
+        -not (Test-Path -LiteralPath $target) -or (Get-Item -LiteralPath $target -Force).Length -ne $_.Length
     }
 })
 # A deployment writes the configuration but cannot restart the server, so a server that is older than its
@@ -294,17 +361,11 @@ $stale = @(if (Test-Path -LiteralPath $binDir) {
 $listener = Get-NetTCPConnection -LocalPort 22 -State Listen -ErrorAction SilentlyContinue | Select-Object -First 1
 $outdated = $listener -and @(Get-Item -LiteralPath "$msys\etc\ssh\sshd_config", (Join-Path $PSScriptRoot 'run-sshd.bat') |
     Where-Object { $_.LastWriteTime -gt (Get-Process -Id $listener.OwningProcess).StartTime }).Count -gt 0
-$restart = $mayStop -and (-not $listener -or $outdated -or $runtimeBroken -or $stale.Count -gt 0)
+$restart = $mayStop -and (-not $listener -or $outdated -or $runtimeOld -or $stale.Count -gt 0)
 
 if ($restart) {
     Stop-Sshd
-    Remove-Item "$kik\msys-2.0.dll" -Force -ErrorAction SilentlyContinue
+    if ($runtimeOld) { Restore-Protected $runtime 'ReadAndExecute' }
-    if ($runtimeBroken) {
-        # the package is put back while the server is stopped: the file cannot be replaced while it is mapped.
-        # The pattern is quoted, or the runtime expands it to the files of the working directory.
-        Invoke-Bounded "$msys\usr\bin\pacman.exe" @('-S', '--noconfirm', '--overwrite', "'*'", 'msys2-runtime') 300 | Out-Null
-        Write-Log 'restored the runtime of the installation'
-    }
     if (-not (Test-Path -LiteralPath $kik)) { New-Item -ItemType Directory -Path $kik | Out-Null }
     foreach ($file in $stale) {
         try {
@@ -319,24 +380,13 @@ if ($restart) {
 
 # The permissions do not need the server to be stopped, so they are put back on every run.
 try {
+    Set-LockedAcl "$msys\$runtime" $false 'ReadAndExecute'
     if (Test-Path -LiteralPath $binDir) {
         New-Item -ItemType Directory -Force -Path "$kik\dev\shm", "$kik\dev\mqueue" | Out-Null
         Set-LockedAcl $kik $true 'ReadAndExecute'
         # /dev is scratch: the user may write it so POSIX shared memory works, the rest of opt/kik stays read-only
         if ($sid) { icacls "$kik\dev" /grant "*${sid}:(OI)(CI)M" | Out-Null }
     }
-
-    # The account database and the ssh keys and config the SYSTEM server reads: the user may read passwd (programs
-    # map ids through it) but change none of it, and has no business in /etc/ssh at all.
-    $rootShell = '/opt/kik/zsh.exe'
-    $pw = "$msys\etc\passwd"
-    $lines = Get-Content $pw
-    if ($lines | Where-Object { $_ -match '^root:' -and $_ -notmatch ':/opt/kik/zsh\.exe$' }) {
-        ($lines | ForEach-Object { if ($_ -match '^root:') { $_ -replace ':[^:]*$', ":$rootShell" } else { $_ } }) | Set-Content $pw -Encoding ascii
-        Write-Log 'set the root login shell to the protected zsh'
-    }
-    Set-LockedAcl $pw $false 'Read'
-    if (Test-Path "$msys\etc\ssh") { Set-LockedAcl "$msys\etc\ssh" $true 'None' }
 } catch {
     Write-Log "the protected login path is not fully in place: $_"
 }
@@ -375,43 +425,19 @@ if (-not ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdent
 $system = New-Object System.Security.Principal.SecurityIdentifier('S-1-5-18')
 $admins = New-Object System.Security.Principal.SecurityIdentifier('S-1-5-32-544')
 
-$passwd = Get-Content "$msys\etc\passwd" -ErrorAction SilentlyContinue | Where-Object { $_ -like 'root:*' }
+$rootAccount = Get-LocalUser -Name 'root' -ErrorAction SilentlyContinue
-if (-not $passwd) { throw "No root user in $msys\etc\passwd." }
+if (-not $rootAccount) { throw 'There is no local user root.' }
-$sid = ($passwd -split ':')[4].Split(',')[-1]
+$sid = $rootAccount.SID.Value
 
-# A previous install may have locked opt/kik, the account database and the ssh keys. The administrators keep
+# The DLLs a program of the pristine installation loads, as Windows paths. Its own ldd is asked: what runs here is
-# full control over them, so reset the permissions to inherited now and let this run write freely; the restore
+# only ever the pristine installation, never the one the user can write.
-# task locks everything again at the end.
-$prevEA = $ErrorActionPreference; $ErrorActionPreference = 'SilentlyContinue'
-foreach ($p in "$msys\opt\kik", "$msys\etc\ssh", "$msys\etc\passwd") {
-    if (Test-Path -LiteralPath $p) { icacls $p /reset /t /q | Out-Null }
-}
-Get-ChildItem "$msys\etc\ssh", "$msys\etc\passwd" -Force | ForEach-Object { try { $_.IsReadOnly = $false } catch {} }
-$ErrorActionPreference = $prevEA
-
-function Set-LockedAcl($path, $forDirectory, $userRights) {
-    $acl = Get-Acl -LiteralPath $path
-    $acl.SetAccessRuleProtection($true, $false)
-    foreach ($rule in @($acl.Access)) { [void]$acl.RemoveAccessRuleSpecific($rule) }
-    $acl.SetOwner($admins)
-    $inherit = if ($forDirectory) { 'ContainerInherit,ObjectInherit' } else { 'None' }
-    $acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($system, 'FullControl', $inherit, 'None', 'Allow')))
-    $acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($admins, 'FullControl', $inherit, 'None', 'Allow')))
-    if ($userRights -ne 'None') {
-        $user = New-Object System.Security.Principal.SecurityIdentifier($sid)
-        $acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($user, $userRights, $inherit, 'None', 'Allow')))
-    }
-    Set-Acl -LiteralPath $path -AclObject $acl
-}
-
-# The DLLs an MSYS2 program loads, as Windows paths under C:\msys2.
 function Get-Libraries($program) {
     $paths = @()
-    $posix = ($program -replace '^C:\\msys2', '') -replace '\\', '/'
+    $posix = $program.Substring($pristine.Length) -replace '\\', '/'
-    foreach ($line in & "$msys\usr\bin\ldd.exe" $posix 2>$null) {
+    foreach ($line in & "$pristine\usr\bin\ldd.exe" $posix 2>$null) {
         foreach ($m in [regex]::Matches($line, '/usr/[^\s]+\.dll')) { $paths += $m.Value }
     }
-    $paths | Select-Object -Unique | Where-Object { $_ -notin $runtimeFiles } | ForEach-Object { $msys + ($_ -replace '/', '\') }
+    $paths | Select-Object -Unique | Where-Object { $_ -notin $runtimeFiles } | ForEach-Object { $pristine + ($_ -replace '/', '\') }
 }
 
 if ($Uninstall) {
@@ -421,11 +447,13 @@ if ($Uninstall) {
         icacls $kik /reset /t /q 2>$null | Out-Null
         Remove-Item -LiteralPath $kik -Recurse -Force -ErrorAction SilentlyContinue
     }
-    # the installation's own cygrunsrv sshd takes over again
+    # the installation's own cygrunsrv sshd takes over again, and the protected files are the user's again
     $pw = "$msys\etc\passwd"
     (Get-Content $pw | ForEach-Object { if ($_ -match '^root:') { $_ -replace ':[^:]*$', ':/usr/bin/zsh' } else { $_ } }) | Set-Content $pw -Encoding ascii
-    icacls $pw /reset | Out-Null
+    foreach ($p in 'etc\passwd', 'etc\group', 'etc\nsswitch.conf', 'etc\fstab', 'etc\fstab.d', 'etc\ssh', 'usr\bin\msys-2.0.dll') {
-    if (Test-Path "$msys\etc\ssh") { icacls "$msys\etc\ssh" /reset /t /q 2>$null | Out-Null }
+        if (Test-Path "$msys\$p") { icacls "$msys\$p" /reset /t /q 2>$null | Out-Null }
+    }
+    foreach ($folder in $msys, "$msys\etc", "$msys\usr", "$msys\usr\bin", "$msys\opt") { icacls $folder /remove:d "*$sid" | Out-Null }
     Set-Service $service -StartupType Automatic -ErrorAction SilentlyContinue
     Start-Service $service -ErrorAction SilentlyContinue
     Write-Host "Removed the tasks and $installDir; the $service service serves ssh again."
@@ -449,7 +477,8 @@ foreach ($owner in $system, $admins) {
 $acl.SetOwner($admins)
 Set-Acl -LiteralPath $installDir -AclObject $acl
 
-foreach ($item in Get-ChildItem -LiteralPath $installDir -Recurse -Force) {
+# The folders this script empties must not be links; the pristine installation is only ever written by this script.
+foreach ($item in @(Get-ChildItem -LiteralPath $installDir -Force) + @(Get-ChildItem -LiteralPath $copiesDir, $binDir -Recurse -Force)) {
     if ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) { throw "$($item.FullName) is a link, refusing to install." }
 }
 Remove-Item -LiteralPath $copiesDir -Recurse -Force
@@ -466,8 +495,35 @@ if (-not ($keys | Where-Object { ($_ -split '\s+')[1] -eq ($required[0] -split '
     New-Item -ItemType Directory -Force -Path (Split-Path -Parent $adminKeys) | Out-Null
     [IO.File]::WriteAllText($adminKeys, (($keys + $required) -join "`n") + "`n")
 }
-[IO.File]::WriteAllText((Join-Path $installDir 'restore.ps1'), $restoreScript)
+
-[IO.File]::WriteAllText((Join-Path $installDir 'watch.ps1'), $watchScript)
+# The pristine installation: the official base archive, checked against its pinned hash, and the packages of the
+# site, built where the user cannot reach it. It is built again only when the base or the packages change.
+$built = Join-Path $installDir 'msys2.built'
+if ((Get-Content -LiteralPath $built -ErrorAction SilentlyContinue) -ne "$baseHash $packages") {
+    $archive = Join-Path $installDir $baseArchive
+    if ((Get-FileHash -LiteralPath $archive -ErrorAction SilentlyContinue).Hash -ne $baseHash) {
+        [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
+        Invoke-WebRequest "https://repo.msys2.org/distrib/x86_64/$baseArchive" -OutFile $archive -UseBasicParsing
+        if ((Get-FileHash -LiteralPath $archive).Hash -ne $baseHash) { throw "$baseArchive does not have the pinned hash." }
+    }
+    $staging = Join-Path $installDir 'msys2-new'
+    if (Test-Path -LiteralPath $staging) { Remove-Item -LiteralPath $staging -Recurse -Force }
+    Start-Process -FilePath $archive -ArgumentList '-y', "-o$staging" -Wait -NoNewWindow
+    # The first login sets the installation up. The first update ends the shell when it replaces the runtime, so it
+    # runs twice, and whether the packages are there in the end is what decides.
+    foreach ($command in 'exit', 'pacman -Syuu --noconfirm', 'pacman -Syuu --noconfirm', "pacman -S --needed --noconfirm $packages") {
+        & "$staging\msys64\usr\bin\bash.exe" -lc $command
+    }
+    & "$staging\msys64\usr\bin\bash.exe" -lc "pacman -Q $packages" | Out-Null
+    if ($LASTEXITCODE) { throw 'The packages did not install into the pristine installation.' }
+    # the agents the setup started hold the installation open
+    Get-CimInstance Win32_Process | Where-Object { $_.ExecutablePath -like "$staging\*" -or $_.ExecutablePath -like "$pristine\*" } |
+        ForEach-Object { Stop-Process -Id $_.ProcessId -Force }
+    if (Test-Path -LiteralPath $pristine) { Remove-Item -LiteralPath $pristine -Recurse -Force }
+    Move-Item -LiteralPath "$staging\msys64" -Destination $pristine
+    Remove-Item -LiteralPath $staging
+    Set-Content -LiteralPath $built -Value "$baseHash $packages"
+}
 
 # The trusted login programs: the sshd, its helpers, the login shell and cat, with every DLL they load. The restore
 # task puts these into opt/kik, which the user cannot write.
@@ -480,6 +536,33 @@ foreach ($b in $loginBinaries) {
 }
 foreach ($f in $files.Keys) { Copy-Item -LiteralPath $f -Destination $binDir -Force }
 
+# The account database: the website's root user logs in to the protected zsh, the deployers as Administrator to
+# PowerShell, which nothing in the installation can replace. The ids are the ones MSYS2 derives from the SIDs of the
+# local accounts, in the group None that MSYS2 gives them. The group database is empty and the folder of per-user
+# mount tables too, so that the user cannot put their own in place.
+$accounts = @(
+    @{ Name = 'root'; Home = '/home/root'; Shell = '/opt/kik/zsh.exe' },
+    @{ Name = 'Administrator'; Home = '/c/Users/Administrator'; Shell = '/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe' }
+)
+$passwd = foreach ($account in $accounts) {
+    $user = Get-LocalUser -Name $account.Name
+    $rid = [int]($user.SID.Value -split '-')[-1]
+    '{0}:*:{1}:197121:U-{2}\{0},{3}:{4}:{5}' -f $account.Name, (0x30000 + $rid), $env:COMPUTERNAME, $user.SID.Value, $account.Home, $account.Shell
+}
+[IO.File]::WriteAllText("$pristine\etc\passwd", ($passwd -join "`n") + "`n")
+[IO.File]::WriteAllText("$pristine\etc\group", '')
+New-Item -ItemType Directory -Force -Path "$pristine\etc\fstab.d", "$pristine\etc\ssh" | Out-Null
+
+# The host keys are the server's identity, which the website pins: they are carried over from the installation, where
+# the user cannot change them, and are only made anew when there are none at all.
+if (Test-Path "$msys\etc\ssh\ssh_host_ed25519_key") { Copy-Item "$msys\etc\ssh\ssh_host_*" "$pristine\etc\ssh" -Force }
+if (-not (Test-Path "$pristine\etc\ssh\ssh_host_ed25519_key")) { & "$pristine\usr\bin\bash.exe" -lc 'ssh-keygen -A' }
+
+# The scripts of the tasks come last: a task that runs during the installation must not find a new script without the
+# pristine installation and the protected files it works from.
+[IO.File]::WriteAllText((Join-Path $installDir 'restore.ps1'), $restoreScript)
+[IO.File]::WriteAllText((Join-Path $installDir 'watch.ps1'), $watchScript)
+
 # Remote Desktop is the way in for the administrator, and the port is open to the internet. It needs network level
 # authentication, so that a scanner cannot get a session before it has proven that it knows the password.
 $rdpTcp = 'HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp'
@@ -492,19 +575,17 @@ if ($nla -ne 1) {
 
 # Only the administrator may log in over Remote Desktop. The user of the website is in no group that may log in
 # over it, and it must never be an administrator - that is taken away if it somehow got there.
-$rootAccount = Get-LocalUser -Name 'root' -ErrorAction SilentlyContinue
+foreach ($group in 'Administrators', 'Remote Desktop Users') {
-if ($rootAccount) {
+    $member = Get-LocalGroupMember $group -ErrorAction SilentlyContinue | Where-Object { $_.SID -eq $rootAccount.SID }
-    foreach ($group in 'Administrators', 'Remote Desktop Users') {
+    if ($member) {
-        $member = Get-LocalGroupMember $group -ErrorAction SilentlyContinue | Where-Object { $_.SID -eq $rootAccount.SID }
+        Remove-LocalGroupMember $group $member -Confirm:$false
-        if ($member) {
+        Write-Host "The root user was in '$group' and has been removed from it."
-            Remove-LocalGroupMember $group $member -Confirm:$false
-            Write-Host "The root user was in '$group' and has been removed from it."
-        }
     }
 }
 
-# The sshd reads its configuration and finds its helpers in opt/kik; nothing here depends on the user's /usr.
+# The sshd reads its configuration and finds its helpers in opt/kik; nothing here depends on the user's /usr. It is
-$sshdConfig = "$msys\etc\ssh\sshd_config"
+# written to the pristine installation, and the restore task puts it in place.
+$sshdConfig = "$pristine\etc\ssh\sshd_config"
 Set-Content -LiteralPath $sshdConfig -Encoding ascii -Value @(
     '# The website logs in as root, the deployers as Administrator. Both come in over the websocket bridge.',
     'Port 22',
@@ -517,7 +598,11 @@ Set-Content -LiteralPath $sshdConfig -Encoding ascii -Value @(
     'Subsystem sftp internal-sftp',
     '# The server and its per-connection helpers are the protected copies, not the ones the user can replace.',
     'SshdSessionPath /opt/kik/sshd-session.exe',
-    'SshdAuthPath /opt/kik/sshd-auth.exe'
+    'SshdAuthPath /opt/kik/sshd-auth.exe',
+    '# The sessions inherit the path of the server, which leads to the runtime in /usr/bin - where the user of the',
+    '# website can put programs. The deployers'' PowerShell must only find the programs of Windows.',
+    'Match User Administrator',
+    '    SetEnv PATH=/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0'
 )
 
 # The sshd runs as SYSTEM from the protected copy, with opt/kik first on the path so its own DLLs and the session's
@@ -582,10 +667,10 @@ function Start-Sshd {
 }
 
 # The restore task builds opt/kik and sets every permission; run it before starting the sshd that depends on it.
+# It can take long: after a wipe it copies the whole installation back.
 Start-ScheduledTask -TaskName $restoreTask
 Start-Sleep -Seconds 6
-$waited = 0
+while ((Get-ScheduledTask -TaskName $restoreTask).State -eq 'Running') { Start-Sleep 5 }
-while ((Get-ScheduledTask -TaskName $restoreTask).State -eq 'Running' -and $waited -lt 60) { Start-Sleep 2; $waited += 2 }
 
 # The server is not restarted here. It cannot be: the deployment runs in a session, and the port is still in use by
 # that session, so a server that starts now cannot bind it. The cleanup job restarts the server when nobody is

server/scripts/rdp.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Opens a remote desktop session to the shell server as Administrator, for when SSH is down. The session runs
+# headless on a virtual display, so it can be driven from the command line:
+#
+#   server/scripts/rdp.sh         opens the session (and closes one that is still open)
+#   server/scripts/rdp.sh stop    closes it
+#
+# - look:   DISPLAY=:73 import -window root shot.png
+# - type:   printf 'text' > $XDG_RUNTIME_DIR/kik-rdp/keys, and special keys with DISPLAY=:73 xdotool key Return
+# - files:  $XDG_RUNTIME_DIR/kik-rdp/share is \\tsclient\kik in the session
+#
+# The password is read from .password.txt in the root of the repository and handed to the client on stdin, so it
+# shows up neither on a command line nor in a log.
+set -euo pipefail
+
+repo=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+dir=${XDG_RUNTIME_DIR:-/tmp}/kik-rdp
+display=:73
+
+for pid in "$dir"/*.pid; do
+    [ -e "$pid" ] || continue
+    kill "$(cat "$pid")" 2>/dev/null || true
+    rm -f "$pid"
+done
+if [ "${1:-}" = stop ]; then exit; fi
+
+bin() { echo "$(nix build --no-link --print-out-paths "nixpkgs#$1")/bin/$2"; }
+mkdir -p "$dir/share"
+# the client creates the pipe itself and refuses one that is left over
+rm -f "$dir/keys"
+
+setsid "$(bin xorg.xvfb Xvfb)" "$display" -screen 0 1280x800x24 -nolisten tcp > "$dir/xvfb.log" 2>&1 < /dev/null &
+echo $! > "$dir/xvfb.pid"
+sleep 2
+
+{
+    printf '%s\n' /v:nowhere.wirs.ing /u:Administrator /cert:tofu /size:1280x800 +auto-reconnect \
+        "/drive:kik,$dir/share" "/kbd:unicode:on,pipe:$dir/keys" /log-level:WARN
+    printf '/p:%s\n' "$(cat "$repo/.password.txt")"
+} | DISPLAY=$display setsid "$(bin freerdp xfreerdp)" /args-from:stdin > "$dir/xfreerdp.log" 2>&1 &
+echo $! > "$dir/xfreerdp.pid"
+
+sleep 15
+if kill -0 "$(cat "$dir/xfreerdp.pid")" 2>/dev/null; then
+    printf 'The session is open on display %s; %s/share is \\\\tsclient\\kik in it.\n' "$display" "$dir"
+else
+    echo 'The client ended:' >&2
+    tail -5 "$dir/xfreerdp.log" >&2
+    exit 1
+fi