Keep the site up when the cleanup job is killed
The cleanup job stops the server, puts the files back and starts it again. On a slow disk that took minutes, and on 2026-10-28 the job was killed while the server was down: the task of the server only starts it again when it failed, never when it was stopped, so the site was down until the box was power-cycled. The job now puts the home directory, the four texts and the permissions of the installation back while the server runs, and only stops it for the protected copy, which cannot be replaced while the server holds it open. A new task, "KiK watch the server", starts the server every two minutes when nothing listens on port 22 and the cleanup job is not running. Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
Zebreus
parent
f4113033ac
commit
5a885e3781
2 changed files with 99 additions and 56 deletions
|
|
@ -58,19 +58,27 @@ user as well; everything else in the installation is theirs.
|
|||
|
||||
## The cleanup tasks
|
||||
|
||||
Two scheduled tasks run as SYSTEM, both at startup, and the user cannot stop either:
|
||||
Three scheduled tasks run as SYSTEM, all at startup, and the user cannot stop any of them:
|
||||
|
||||
- `KiK sshd` runs the sshd from `C:\msys2\opt\kik`. It replaces the installation's own `msys2-sshd` service, which
|
||||
is disabled.
|
||||
- `KiK restore shell files` runs every 30 minutes and rebuilds the protected copy from its trusted originals,
|
||||
re-applies the permissions, and puts the home directory, the shell configuration, the website's key and the four
|
||||
texts back. Its second job is to best-effort revert other mockery and bring a fresh server to the expected state.
|
||||
- `KiK watch the server` runs every two minutes and starts the server when nothing listens on port 22. The task of
|
||||
the server only starts it again when it failed, never when it was stopped, so a cleanup job that is killed while
|
||||
it has the server stopped would otherwise take the site down with it.
|
||||
|
||||
The cleanup job also restarts the server when nobody is connected, and when the server does not listen at all:
|
||||
it comes back by itself if it died, it takes over a server that was left behind, and it picks up the programs that
|
||||
were just restored. A session that is open keeps the port, so a server that is started now could not bind it, and
|
||||
the job never disturbs a session.
|
||||
|
||||
The server is only down for the short part of the job that needs it down: the protected copy cannot be replaced
|
||||
while the server holds it open. Everything else - the home directory, the four texts, the permissions of the
|
||||
installation - is put back while the server runs, because on a slow disk that takes minutes and the site should
|
||||
have a shell the whole time.
|
||||
|
||||
The task that runs the server ignores a start while it thinks it is still running, so a start that is dropped
|
||||
leaves the site without a shell. The cleanup job therefore tries three times, killing the processes of the task
|
||||
in between, before it gives up.
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ $binDir = Join-Path $installDir 'bin'
|
|||
$sshdBat = Join-Path $installDir 'run-sshd.bat'
|
||||
$restoreTask = 'KiK restore shell files'
|
||||
$sshdTask = 'KiK sshd'
|
||||
$watchTask = 'KiK watch the server'
|
||||
$service = 'msys2-sshd'
|
||||
|
||||
# The programs the journey needs, kept out of the user's reach. sshd-session and sshd-auth are the per-connection
|
||||
|
|
@ -215,60 +216,8 @@ if ($restart) {
|
|||
$runtimeBroken = (Invoke-Bounded "$msys\usr\bin\pacman.exe" @('-Qkk', 'msys2-runtime') 120) -ne 0
|
||||
}
|
||||
|
||||
if ($restart) {
|
||||
Stop-Sshd
|
||||
Remove-Item "$kik\msys-2.0.dll" -Force -ErrorAction SilentlyContinue
|
||||
if ($runtimeBroken) {
|
||||
# the package is put back while the server is stopped: the file cannot be replaced while it is mapped.
|
||||
# The pattern is quoted, or the runtime expands it to the files of the working directory.
|
||||
Invoke-Bounded "$msys\usr\bin\pacman.exe" @('-S', '--noconfirm', '--overwrite', "'*'", 'msys2-runtime') 300 | Out-Null
|
||||
Write-Log 'restored the runtime of the installation'
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
# The protected login path: the copy the sshd and the login shell run from. The user cannot write it, so even
|
||||
# after they replace or remove their own copies in /usr/bin the server and the journey keep working. The trusted
|
||||
# originals are in bin\; anything missing from opt/kik is put back.
|
||||
if (Test-Path -LiteralPath $binDir) {
|
||||
if (-not (Test-Path -LiteralPath $kik)) { New-Item -ItemType Directory -Path $kik | Out-Null }
|
||||
foreach ($file in Get-ChildItem -LiteralPath $binDir -File -Force | Where-Object { $_.Name -ne 'msys-2.0.dll' }) {
|
||||
$target = Join-Path $kik $file.Name
|
||||
try {
|
||||
if (-not (Test-Path -LiteralPath $target) -or (Get-Item -LiteralPath $target -Force).Length -ne $file.Length) {
|
||||
Copy-Item -LiteralPath $file.FullName -Destination $target -Force
|
||||
Write-Log "restored opt/kik/$($file.Name)"
|
||||
}
|
||||
} catch {
|
||||
# a running server holds its programs open; they are replaced in one of the next runs
|
||||
Write-Log "failed opt/kik/$($file.Name) : $_"
|
||||
}
|
||||
}
|
||||
New-Item -ItemType Directory -Force -Path "$kik\dev\shm", "$kik\dev\mqueue" | Out-Null
|
||||
Set-LockedAcl $kik $true 'ReadAndExecute'
|
||||
# /dev is scratch: the user may write it so POSIX shared memory works, the rest of opt/kik stays read-only
|
||||
if ($sid) { icacls "$kik\dev" /grant "*${sid}:(OI)(CI)M" | Out-Null }
|
||||
}
|
||||
|
||||
# The account database and the ssh keys and config the SYSTEM server reads: the user may read passwd (programs
|
||||
# map ids through it) but change none of it, and has no business in /etc/ssh at all.
|
||||
$rootShell = '/opt/kik/zsh.exe'
|
||||
$pw = "$msys\etc\passwd"
|
||||
$lines = Get-Content $pw
|
||||
if ($lines | Where-Object { $_ -match '^root:' -and $_ -notmatch ':/opt/kik/zsh\.exe$' }) {
|
||||
($lines | ForEach-Object { if ($_ -match '^root:') { $_ -replace ':[^:]*$', ":$rootShell" } else { $_ } }) | Set-Content $pw -Encoding ascii
|
||||
Write-Log 'set the root login shell to the protected zsh'
|
||||
}
|
||||
Set-LockedAcl $pw $false 'Read'
|
||||
if (Test-Path "$msys\etc\ssh") { Set-LockedAcl "$msys\etc\ssh" $true 'None' }
|
||||
|
||||
# The whole installation otherwise belongs to the user, so pacman can write it. The home directory keeps its own
|
||||
# protected permissions because they are set without inheritance.
|
||||
if ($sid) { icacls $msys /grant "*${sid}:(OI)(CI)F" | Out-Null }
|
||||
} catch {
|
||||
Write-Log "the protected login path is not fully in place: $_"
|
||||
}
|
||||
|
||||
# The home directory and the four texts: the server does not hold them open, so they are put back while it runs.
|
||||
# On a slow disk this takes minutes, and the site has a shell the whole time.
|
||||
if (-not (Test-Path -LiteralPath $HomeDir)) {
|
||||
New-Home $HomeDir
|
||||
Write-Log "created $HomeDir"
|
||||
|
|
@ -313,11 +262,86 @@ foreach ($copy in Get-ChildItem -LiteralPath $copies -Recurse -File -Force) {
|
|||
}
|
||||
}
|
||||
|
||||
# The whole installation belongs to the user, so pacman can write it. The home directory keeps its own protected
|
||||
# permissions because they are set without inheritance.
|
||||
if ($sid) { icacls $msys /grant "*${sid}:(OI)(CI)F" | Out-Null }
|
||||
|
||||
if ($restart) {
|
||||
Stop-Sshd
|
||||
Remove-Item "$kik\msys-2.0.dll" -Force -ErrorAction SilentlyContinue
|
||||
if ($runtimeBroken) {
|
||||
# the package is put back while the server is stopped: the file cannot be replaced while it is mapped.
|
||||
# The pattern is quoted, or the runtime expands it to the files of the working directory.
|
||||
Invoke-Bounded "$msys\usr\bin\pacman.exe" @('-S', '--noconfirm', '--overwrite', "'*'", 'msys2-runtime') 300 | Out-Null
|
||||
Write-Log 'restored the runtime of the installation'
|
||||
}
|
||||
|
||||
try {
|
||||
# The protected login path: the copy the sshd and the login shell run from. The user cannot write it, so even
|
||||
# after they replace or remove their own copies in /usr/bin the server and the journey keep working. The trusted
|
||||
# originals are in bin\; anything missing from opt/kik is put back. This is the only work that needs the server
|
||||
# to be stopped, because a running server holds its programs open.
|
||||
if (Test-Path -LiteralPath $binDir) {
|
||||
if (-not (Test-Path -LiteralPath $kik)) { New-Item -ItemType Directory -Path $kik | Out-Null }
|
||||
foreach ($file in Get-ChildItem -LiteralPath $binDir -File -Force | Where-Object { $_.Name -ne 'msys-2.0.dll' }) {
|
||||
$target = Join-Path $kik $file.Name
|
||||
try {
|
||||
if (-not (Test-Path -LiteralPath $target) -or (Get-Item -LiteralPath $target -Force).Length -ne $file.Length) {
|
||||
Copy-Item -LiteralPath $file.FullName -Destination $target -Force
|
||||
Write-Log "restored opt/kik/$($file.Name)"
|
||||
}
|
||||
} catch {
|
||||
# a running server holds its programs open; they are replaced in one of the next runs
|
||||
Write-Log "failed opt/kik/$($file.Name) : $_"
|
||||
}
|
||||
}
|
||||
New-Item -ItemType Directory -Force -Path "$kik\dev\shm", "$kik\dev\mqueue" | Out-Null
|
||||
Set-LockedAcl $kik $true 'ReadAndExecute'
|
||||
# /dev is scratch: the user may write it so POSIX shared memory works, the rest of opt/kik stays read-only
|
||||
if ($sid) { icacls "$kik\dev" /grant "*${sid}:(OI)(CI)M" | Out-Null }
|
||||
}
|
||||
|
||||
# The account database and the ssh keys and config the SYSTEM server reads: the user may read passwd (programs
|
||||
# map ids through it) but change none of it, and has no business in /etc/ssh at all.
|
||||
$rootShell = '/opt/kik/zsh.exe'
|
||||
$pw = "$msys\etc\passwd"
|
||||
$lines = Get-Content $pw
|
||||
if ($lines | Where-Object { $_ -match '^root:' -and $_ -notmatch ':/opt/kik/zsh\.exe$' }) {
|
||||
($lines | ForEach-Object { if ($_ -match '^root:') { $_ -replace ':[^:]*$', ":$rootShell" } else { $_ } }) | Set-Content $pw -Encoding ascii
|
||||
Write-Log 'set the root login shell to the protected zsh'
|
||||
}
|
||||
Set-LockedAcl $pw $false 'Read'
|
||||
if (Test-Path "$msys\etc\ssh") { Set-LockedAcl "$msys\etc\ssh" $true 'None' }
|
||||
} catch {
|
||||
Write-Log "the protected login path is not fully in place: $_"
|
||||
}
|
||||
|
||||
if (Start-Sshd) { Write-Log 'restarted the sshd' } else { Write-Log 'the sshd does not listen on port 22' }
|
||||
}
|
||||
'@
|
||||
|
||||
# The server is watched, because the cleanup job may be killed while it has the server stopped: the task of the
|
||||
# server only starts it again when it failed, never when it was stopped. The watcher is the only thing that brings
|
||||
# the site back when that happens.
|
||||
$watchScript = @'
|
||||
# Starts the server when it is not running. The cleanup job stops it while it puts the protected files back, and it
|
||||
# can be killed while the server is down - the task of the server does not start it again then.
|
||||
$log = Join-Path $PSScriptRoot 'watch.log'
|
||||
if (Get-NetTCPConnection -LocalPort 22 -State Listen -ErrorAction SilentlyContinue) { return }
|
||||
if ((Get-ScheduledTask -TaskName 'KiK restore shell files').State -eq 'Running') {
|
||||
# the cleanup job starts the server itself when it is done, but it must not hold it down for ever
|
||||
$running = (Get-Date) - (Get-ScheduledTaskInfo -TaskName 'KiK restore shell files').LastRunTime
|
||||
if ($running.TotalMinutes -lt 3) { return }
|
||||
}
|
||||
if ((Get-ScheduledTask -TaskName 'KiK sshd').State -eq 'Running') {
|
||||
# the task is running but nothing listens, so it is stuck and has to be started anew
|
||||
Stop-ScheduledTask -TaskName 'KiK sshd'
|
||||
Start-Sleep 5
|
||||
}
|
||||
Start-ScheduledTask -TaskName 'KiK sshd'
|
||||
"$(Get-Date -Format s) the server was not running and was started" | Out-File -FilePath $log -Append
|
||||
'@
|
||||
|
||||
if (-not ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
|
||||
throw 'Run this in an elevated PowerShell.'
|
||||
}
|
||||
|
|
@ -365,7 +389,7 @@ function Get-Libraries($program) {
|
|||
}
|
||||
|
||||
if ($Uninstall) {
|
||||
foreach ($t in $restoreTask, $sshdTask) { Unregister-ScheduledTask -TaskName $t -Confirm:$false -ErrorAction SilentlyContinue }
|
||||
foreach ($t in $restoreTask, $sshdTask, $watchTask) { Unregister-ScheduledTask -TaskName $t -Confirm:$false -ErrorAction SilentlyContinue }
|
||||
if (Test-Path -LiteralPath $installDir) { Remove-Item -LiteralPath $installDir -Recurse -Force }
|
||||
if (Test-Path -LiteralPath $kik) {
|
||||
icacls $kik /reset /t /q 2>$null | Out-Null
|
||||
|
|
@ -417,6 +441,7 @@ if (-not ($keys | Where-Object { ($_ -split '\s+')[1] -eq ($required[0] -split '
|
|||
[IO.File]::WriteAllText($adminKeys, (($keys + $required) -join "`n") + "`n")
|
||||
}
|
||||
[IO.File]::WriteAllText((Join-Path $installDir 'restore.ps1'), $restoreScript)
|
||||
[IO.File]::WriteAllText((Join-Path $installDir 'watch.ps1'), $watchScript)
|
||||
|
||||
# The trusted login programs: the sshd, its helpers, the login shell and cat, with every DLL they load. The restore
|
||||
# task puts these into opt/kik, which the user cannot write.
|
||||
|
|
@ -499,6 +524,16 @@ $restoreTriggers = @(
|
|||
$restoreSettings = New-ScheduledTaskSettingsSet -StartWhenAvailable -MultipleInstances IgnoreNew -ExecutionTimeLimit (New-TimeSpan -Minutes 30) -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 5) -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries
|
||||
Register-ScheduledTask -TaskName $restoreTask -Action $restoreAction -Trigger $restoreTriggers -Principal $atSystem -Settings $restoreSettings -Force | Out-Null
|
||||
|
||||
# The watcher looks after the server every two minutes, so that a cleanup job that is killed while it has the server
|
||||
# stopped does not take the site down with it.
|
||||
$watchAction = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -WindowStyle Hidden -File `"$installDir\watch.ps1`""
|
||||
$watchTriggers = @(
|
||||
New-ScheduledTaskTrigger -Once -At (Get-Date) -RepetitionInterval (New-TimeSpan -Minutes 2) -RepetitionDuration (New-TimeSpan -Days 3650)
|
||||
New-ScheduledTaskTrigger -AtStartup
|
||||
)
|
||||
$watchSettings = New-ScheduledTaskSettingsSet -StartWhenAvailable -MultipleInstances IgnoreNew -ExecutionTimeLimit (New-TimeSpan -Minutes 1) -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries
|
||||
Register-ScheduledTask -TaskName $watchTask -Action $watchAction -Trigger $watchTriggers -Principal $atSystem -Settings $watchSettings -Force | Out-Null
|
||||
|
||||
# The server is started and waited for: it must listen on port 22, or the site has no shell.
|
||||
function Start-Sshd {
|
||||
Start-ScheduledTask -TaskName $sshdTask
|
||||
|
|
|
|||
Loading…
Reference in a new issue