d4551124f7
The worked example keyed a stream selector on `_HOSTNAME`, which the collector config warns against: a hostname is a config value two machines can share, so keying on it merges their streams. Use `_MACHINE_ID`, which is the field that actually distinguishes them. Refs #4460
89 lines
4.5 KiB
Markdown
89 lines
4.5 KiB
Markdown
---
|
|
name: swarm-logs
|
|
description: Build a bounded `swarm-logs query` LogsQL call when investigating an error, confirming a deploy or restart actually happened, checking whether a service emitted something, or chasing any "why did/didn't X happen" question that logs would answer. `swarm-logs --help` documents no time predicate and no per-hive scoping, so an agent copying only what `--help` shows ends up either scanning the whole swarm's history or trusting an empty result it never confirmed. Use this before running `swarm-logs query` for the first time in a session, and any time you're about to call an empty result "nothing happened."
|
|
---
|
|
|
|
# swarm-logs
|
|
|
|
`swarm-logs query '<LogsQL>'` is the one verb (plus `--limit` and
|
|
`--format`). Everything below fills in what `--help` doesn't say, from
|
|
actually using the tool.
|
|
|
|
## Bound the query with `_time:`
|
|
|
|
`--help` shows no time predicate at all — there's no flag for it, because
|
|
it's part of the LogsQL string itself, not a CLI option. Without one, a
|
|
query runs over the store's whole retention window.
|
|
|
|
LogsQL's `_time:` filter (from the store's own query language, VictoriaLogs'
|
|
LogsQL — this isn't `swarm-logs`-specific syntax):
|
|
|
|
- **Relative**: `_time:5m` (last 5 minutes), `_time:1h`, `_time:2d`.
|
|
- **Absolute range**: `_time:[2026-09-16T00:00:00Z, 2026-09-16T12:00:00Z]`.
|
|
- **Combine with a search term** — juxtaposition is an implicit `AND`:
|
|
|
|
```console
|
|
$ swarm-logs query '_time:1h "connection refused"' --limit 50
|
|
$ swarm-logs query '_time:[2026-09-16Z, 2026-09-17Z] deploy AND atlas'
|
|
```
|
|
|
|
Reach for `_time:` on every query you write by hand, not just the slow
|
|
ones — it's the only thing in this tool that keeps a broad question from
|
|
becoming a full-retention scan.
|
|
|
|
## A bare word searches every hive, not just yours
|
|
|
|
`--help` says it plainly: "nothing between here and the store narrows
|
|
one, so a bare word matches across every hive in the swarm." There's no
|
|
per-hive flag to fix this from the CLI side — scope it in the query
|
|
string instead. The collector tags every log line with `_HOSTNAME`,
|
|
`_MACHINE_ID`, and `_SYSTEMD_UNIT` as stream fields, so add one of those
|
|
to a query you want confined to your own machine or service:
|
|
|
|
Reach for `_MACHINE_ID` rather than `_HOSTNAME` when you mean one
|
|
machine. A hostname is a configuration value, so two machines can carry
|
|
the same one, and a query keyed on it silently merges their streams —
|
|
the collector's own config warns about this case for exactly that reason.
|
|
|
|
```console
|
|
$ swarm-logs query '_time:1h _stream:{_MACHINE_ID="<id>"} "connection refused"'
|
|
$ swarm-logs query '_time:1h _SYSTEMD_UNIT:="hive-agent.service"' --format json
|
|
```
|
|
|
|
If you don't know your own hostname or unit name offhand, a first query
|
|
scoped only by `_time:` and a distinctive search term is fine — just
|
|
don't leave a bare word unscoped and assume the result is about you.
|
|
|
|
## `--format message` vs `--format json`
|
|
|
|
`message` (the default) prints exactly `_msg` — nothing else, deliberately:
|
|
the ask this tool answers is "pipe and grep like any other command," and a
|
|
grep pattern targets the message text alone. If you're missing
|
|
`_time`, `_stream`, or another field, that's the format choice, not a
|
|
missing capability — pass `--format json` and read the store's response
|
|
body straight, one JSON object per line, for `jq` or for the timestamp.
|
|
|
|
## Telling "nothing matched" from "the store refused you"
|
|
|
|
These look nothing alike once you know the shape, and identical if you
|
|
don't:
|
|
|
|
- **No matching records**: exit 0, empty stdout.
|
|
- **The store refused the request** (bad auth, malformed query, etc.):
|
|
non-zero exit, and the error text on stderr includes the HTTP status
|
|
and the response body. A bare `401` with no further explanation is a
|
|
real, distinct failure mode here — treat any non-zero exit as "this
|
|
query didn't run," not as "this query found nothing."
|
|
|
|
So: empty output with exit 0 is a real empty result. Anything else is a
|
|
tooling or auth problem to fix, not a log finding to report.
|
|
|
|
## Before you believe an empty result, run a control
|
|
|
|
An empty result *looks* like proof of absence, but it's just as
|
|
consistent with a `_time:` window that missed the event, a scoping field
|
|
that doesn't match what you expected, or a typo in the search term. Before
|
|
reporting "no matching logs," run a second query you know must return
|
|
rows — widen the `_time:` window, or search a bare word you just logged
|
|
yourself — and confirm it actually returns something. If the control also
|
|
comes back empty, the query is wrong, not the log store.
|