wip(#3001): convert remaining unblocked call sites; sweep docs
21 of 28 non-test call sites now insert directly. power.rs compiles.
The only remaining errors are server.rs's 5, which are blocked: those
sites feed the returned id into HostResponse::queued -> `queued_dags`,
a wire field hivectl polls via QueueDag. Removing the container without
answering that breaks hivectl's wait/progress loop; asked on the issue.
Also swept the deleted symbol out of prose, not just code:
- docs/coordinator.md: "the submit layer (job_queue/submit.rs)" ->
the power layer (job_queue/power.rs), and "submits" -> "inserts".
- templates.rs module doc: points at super::power for the power ops.
- lifecycle_ops.rs module doc: says which path each op takes now.
- mod.rs's insert_group comment restated the open issue verbatim
("a DAG is addressed by its container node, which submit inserts
itself"). Replaced with what is actually true for that path.
Dashboard behaviour deltas worth review: insert failures are now
logged per agent instead of swallowed, and UPDATE-ALL emits one queue
snapshot after the loop rather than one per agent.
This commit is contained in:
parent
7c0d9d2379
commit
f04a0cee92
6 changed files with 54 additions and 65 deletions
|
|
@ -106,7 +106,7 @@ subgraph each (independent roots, run concurrently on their own leases), not
|
||||||
N separate DAGs.
|
N separate DAGs.
|
||||||
|
|
||||||
**These are built dynamically from each agent's live running state** (an
|
**These are built dynamically from each agent's live running state** (an
|
||||||
async `lifecycle::is_running` read), so they live in `job_queue/submit.rs`,
|
async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
|
||||||
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
|
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
|
||||||
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
|
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
|
||||||
(convergence guarantee — cheap, noops when already converged) is ALWAYS
|
(convergence guarantee — cheap, noops when already converged) is ALWAYS
|
||||||
|
|
@ -161,12 +161,12 @@ Notable collapses:
|
||||||
Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
|
Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
|
||||||
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
|
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
|
||||||
`container_view` remains the observed *status*; `Reconcile` nodes converge the
|
`container_view` remains the observed *status*; `Reconcile` nodes converge the
|
||||||
two. Setting `wanted` is never a queued node: the submit layer
|
two. Setting `wanted` is never a queued node: the power layer
|
||||||
(`job_queue/submit.rs`) writes the row synchronously, then submits the DAG
|
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
|
||||||
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
||||||
Power toggles never commit to the meta repo. Every operator power surface —
|
Power toggles never commit to the meta repo. Every operator power surface —
|
||||||
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
|
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
|
||||||
rides the queue through that submit layer, so intent, lease serialization,
|
rides the queue through that power layer, so intent, lease serialization,
|
||||||
and crash-watch suppression can't drift per surface; the only direct starts
|
and crash-watch suppression can't drift per surface; the only direct starts
|
||||||
left are the root-agent bootstrap and infra containers (no lease, no
|
left are the root-agent bootstrap and infra containers (no lease, no
|
||||||
harness). Cancelling a still-queued power DAG reverts `wanted` to the
|
harness). Cancelling a still-queued power DAG reverts `wanted` to the
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,12 @@
|
||||||
//! Container lifecycle endpoints for the dashboard.
|
//! Container lifecycle endpoints for the dashboard.
|
||||||
//!
|
//!
|
||||||
//! Rebuild / restart / start / stop (hard + graceful) / update-all all
|
//! Rebuild / restart / start / stop (hard + graceful) / update-all all
|
||||||
//! submit DAGs to the job queue (`job_queue::submit`), so each shows a
|
//! insert DAGs into the job queue — the power ops via
|
||||||
//! visible queued→running transient on the dashboard — a direct
|
//! [`crate::job_queue::power`], the static shapes straight through
|
||||||
//! sub-second start/stop only flashed the badge. Start/stop also
|
//! `JobQueue::insert` — so each shows a visible queued→running transient on
|
||||||
//! persist the agent's `wanted` power intent before submitting; the
|
//! the dashboard; a direct sub-second start/stop only flashed the badge.
|
||||||
//! DAG's `Reconcile` converges to it. Destroy delegates to
|
//! Start/stop also persist the agent's `wanted` power intent before
|
||||||
|
//! inserting; the DAG's `Reconcile` converges to it. Destroy delegates to
|
||||||
//! `actions::destroy` (optionally purging).
|
//! `actions::destroy` (optionally purging).
|
||||||
|
|
||||||
use axum::{
|
use axum::{
|
||||||
|
|
@ -27,7 +28,6 @@ pub(super) struct GracefulParams {
|
||||||
}
|
}
|
||||||
|
|
||||||
use super::{AppState, Ident, error_response, guard_agent_name, strip_container_prefix};
|
use super::{AppState, Ident, error_response, guard_agent_name, strip_container_prefix};
|
||||||
use crate::job_queue::{Source, submit};
|
|
||||||
use crate::{actions, lifecycle};
|
use crate::{actions, lifecycle};
|
||||||
|
|
||||||
/// Queue a rebuild DAG for `name`.
|
/// Queue a rebuild DAG for `name`.
|
||||||
|
|
@ -50,12 +50,13 @@ pub(super) async fn post_rebuild(
|
||||||
if let Some(reject) = guard_agent_name(&state, &logical).await {
|
if let Some(reject) = guard_agent_name(&state, &logical).await {
|
||||||
return reject;
|
return reject;
|
||||||
}
|
}
|
||||||
submit::rebuild(
|
if let Err(e) = state.coord.job_queue.insert(|b| {
|
||||||
&state.coord,
|
crate::job_queue::templates::rebuild(b, &logical, true);
|
||||||
&logical,
|
Vec::new()
|
||||||
Source::Manual,
|
}) {
|
||||||
"manual via dashboard ↻ R3BU1LD button".to_owned(),
|
tracing::error!(agent = %logical, error = ?e, "rebuild: insert failed");
|
||||||
);
|
}
|
||||||
|
state.coord.emit_rebuild_queue_snapshot();
|
||||||
(StatusCode::OK, "ok").into_response()
|
(StatusCode::OK, "ok").into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -92,13 +93,11 @@ pub(super) async fn post_kill(
|
||||||
// timeout fallback to a hard stop). The agent's lifecycle
|
// timeout fallback to a hard stop). The agent's lifecycle
|
||||||
// lease keeps it from racing an in-flight rebuild for the same
|
// lease keeps it from racing an in-flight rebuild for the same
|
||||||
// agent, and per-node progress surfaces on the queue snapshot.
|
// agent, and per-node progress surfaces on the queue snapshot.
|
||||||
submit::graceful_stop(
|
if let Err(e) =
|
||||||
&state.coord,
|
crate::job_queue::power::stop_many(&state.coord, &[logical.clone()], true).await
|
||||||
&logical,
|
{
|
||||||
Source::Manual,
|
tracing::error!(agent = %logical, error = ?e, "graceful stop: insert failed");
|
||||||
"manual via dashboard graceful stop".to_owned(),
|
}
|
||||||
)
|
|
||||||
.await;
|
|
||||||
return (StatusCode::OK, "ok").into_response();
|
return (StatusCode::OK, "ok").into_response();
|
||||||
}
|
}
|
||||||
// Manager is stoppable from the dashboard like any other
|
// Manager is stoppable from the dashboard like any other
|
||||||
|
|
@ -111,13 +110,10 @@ pub(super) async fn post_kill(
|
||||||
// `socket_server.rs::Request::Kill` stays in place: a
|
// `socket_server.rs::Request::Kill` stays in place: a
|
||||||
// manager calling Kill on its own container is self-suicide
|
// manager calling Kill on its own container is self-suicide
|
||||||
// mid-call, not a legitimate operator action.
|
// mid-call, not a legitimate operator action.
|
||||||
submit::stop(
|
if let Err(e) = crate::job_queue::power::stop_many(&state.coord, &[logical.clone()], false).await
|
||||||
&state.coord,
|
{
|
||||||
&logical,
|
tracing::error!(agent = %logical, error = ?e, "stop: insert failed");
|
||||||
Source::Manual,
|
}
|
||||||
"manual via dashboard stop".to_owned(),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
(StatusCode::OK, "ok").into_response()
|
(StatusCode::OK, "ok").into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -149,22 +145,18 @@ pub(super) async fn post_restart(
|
||||||
return reject;
|
return reject;
|
||||||
}
|
}
|
||||||
if params.graceful {
|
if params.graceful {
|
||||||
submit::graceful_restart(
|
if let Err(e) =
|
||||||
&state.coord,
|
crate::job_queue::power::restart_many(&state.coord, &[logical.clone()], true).await
|
||||||
&logical,
|
{
|
||||||
Source::Manual,
|
tracing::error!(agent = %logical, error = ?e, "graceful restart: insert failed");
|
||||||
"manual via dashboard graceful restart".to_owned(),
|
}
|
||||||
)
|
|
||||||
.await;
|
|
||||||
return (StatusCode::OK, "ok").into_response();
|
return (StatusCode::OK, "ok").into_response();
|
||||||
}
|
}
|
||||||
submit::restart(
|
if let Err(e) =
|
||||||
&state.coord,
|
crate::job_queue::power::restart_many(&state.coord, &[logical.clone()], false).await
|
||||||
&logical,
|
{
|
||||||
Source::Manual,
|
tracing::error!(agent = %logical, error = ?e, "restart: insert failed");
|
||||||
"manual via dashboard ↺ R3START button".to_owned(),
|
}
|
||||||
)
|
|
||||||
.await;
|
|
||||||
(StatusCode::OK, "ok").into_response()
|
(StatusCode::OK, "ok").into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -226,13 +218,9 @@ pub(super) async fn post_start(
|
||||||
return (StatusCode::OK, "ok").into_response();
|
return (StatusCode::OK, "ok").into_response();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
submit::start(
|
if let Err(e) = crate::job_queue::power::start_many(&state.coord, &[logical.clone()]).await {
|
||||||
&state.coord,
|
tracing::error!(agent = %logical, error = ?e, "start: insert failed");
|
||||||
&logical,
|
}
|
||||||
Source::Manual,
|
|
||||||
"manual via dashboard start".to_owned(),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
(StatusCode::OK, "ok").into_response()
|
(StatusCode::OK, "ok").into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -411,13 +399,14 @@ pub(super) async fn post_update_all(State(state): State<AppState>) -> Response {
|
||||||
else {
|
else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
submit::rebuild(
|
if let Err(e) = state.coord.job_queue.insert(|b| {
|
||||||
&state.coord,
|
crate::job_queue::templates::rebuild(b, &logical, true);
|
||||||
&logical,
|
Vec::new()
|
||||||
Source::Manual,
|
}) {
|
||||||
"manual via dashboard 🌀 UPDATE ALL".to_owned(),
|
tracing::error!(agent = %logical, error = ?e, "update-all: insert failed");
|
||||||
);
|
}
|
||||||
}
|
}
|
||||||
|
state.coord.emit_rebuild_queue_snapshot();
|
||||||
(StatusCode::OK, "ok").into_response()
|
(StatusCode::OK, "ok").into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,7 +21,6 @@ use utoipa::ToSchema;
|
||||||
use problem_details::ProblemDetails;
|
use problem_details::ProblemDetails;
|
||||||
|
|
||||||
use super::{AppState, error_problem};
|
use super::{AppState, error_problem};
|
||||||
use crate::job_queue::{Source, submit};
|
|
||||||
|
|
||||||
/// `POST /api/topology/set-parent` body. `child` is required.
|
/// `POST /api/topology/set-parent` body. `child` is required.
|
||||||
/// `new_parent` may be:
|
/// `new_parent` may be:
|
||||||
|
|
|
||||||
|
|
@ -161,9 +161,9 @@ fn insert_group(
|
||||||
inner
|
inner
|
||||||
.insert_job(group_parent, |b| {
|
.insert_job(group_parent, |b| {
|
||||||
declare(b);
|
declare(b);
|
||||||
// c0re names no handles: a DAG is addressed by its container node,
|
// A runtime-appended subgraph is addressed by the node that emitted
|
||||||
// which `submit` inserts itself, and nothing downstream looks an
|
// it (`group_parent`), so this path names nothing. Callers that DO
|
||||||
// individual step up by id.
|
// want a handle use `JobQueue::insert` and name the node there.
|
||||||
Vec::new()
|
Vec::new()
|
||||||
})
|
})
|
||||||
.map_err(|e| anyhow::anyhow!("job_queue: graph insert failed: {e}"))?;
|
.map_err(|e| anyhow::anyhow!("job_queue: graph insert failed: {e}"))?;
|
||||||
|
|
|
||||||
|
|
@ -16,8 +16,9 @@ use std::sync::Arc;
|
||||||
|
|
||||||
use hive_jobq::NodeId;
|
use hive_jobq::NodeId;
|
||||||
|
|
||||||
use super::JobBuilder;
|
use super::resource::Resource;
|
||||||
use super::templates;
|
use super::templates::rebuild_nodes;
|
||||||
|
use super::{JobBuilder, NodeKind};
|
||||||
use crate::coordinator::Coordinator;
|
use crate::coordinator::Coordinator;
|
||||||
use crate::lifecycle;
|
use crate::lifecycle;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -17,8 +17,8 @@
|
||||||
//!
|
//!
|
||||||
//! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here:
|
//! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here:
|
||||||
//! their per-agent shape depends on live running state (an async
|
//! their per-agent shape depends on live running state (an async
|
||||||
//! `lifecycle::is_running` read), so `submit.rs` assembles them out of the
|
//! `lifecycle::is_running` read), so [`super::power`] assembles them out of
|
||||||
//! primitives this module exports ([`rebuild_nodes`]).
|
//! the primitives this module exports ([`rebuild_nodes`]).
|
||||||
|
|
||||||
use hive_jobq::TerminalState;
|
use hive_jobq::TerminalState;
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue