Two deployments, one codebase

This repository ran two deployments that share every line of service code and agree on almost nothing else. They are the clearest answer to what hookstack actually is: the graph is config, and two useful graphs look nothing alike.

Status, 2026-09-23. The alert deployment (deploy/shadow.yaml) was retired at the operator’s decision. Its host now runs only the alerting platform it sat behind, nothing of hookstack is left there, and its data is archived off the host. Its configuration stays in this repository as the worked example of the alert shape, and everything below about it is a record of how it ran. The work deployment (deploy/work.yaml) runs on the operator’s own machine and is the only one running.

  deploy/shadow.yaml deploy/work.yaml
Carries alerts from a monitoring platform work signals from chat and a ticket tracker
Front door one signed door the platform posts to a timer, and the watcher’s own findings
Brains one judge (two comparison arms ran until 2026-09-09) none — the watcher already judged
Investigator one, escalated by level one, escalated by level AND kind
Destinations one chat, cards with buttons two chats, no buttons
Services 3 containers + a bridge 4 containers

Neither needed a line of Python that the other did not.

The alert shape

platform ──► /hook/ww ──► to-judge ──► /hook/judge-notify ──┐
                                                            │
                                          ├──► to-me      ── card, with buttons
                                          └──► to-probe   ──► investigator
                                                    └──► /hook/probe-notify ──► to-me

Two comparison arms ran here until 2026-09-09 and the fan-out that fed them is still the reason the route is shaped this way, so it is worth saying what they were for and what ended them. They judged the same traffic and disagreed in a ledger, which is labelled data at production rates. They had no return door, deliberately — an arm that answered back would be a second copy of the chain rather than a second opinion.

What ended them was reading their own numbers. hookjudge-b ran the same model as the live judge and differed only in its price constants, so it was still comparing a model with itself. hookjudge-c was the genuine cross-vendor arm and did think differently — it upgraded severity where the live judge downgrades — but it cost 25.7x the live judge for the same calls at 2.7x the latency, and nothing read its verdicts. A second opinion nobody reads is a subscription, not an instrument. Their ledgers are kept; restoring an arm means a service block, a channel, and one name in the fan-out.

The quiet-wake-no filter stage drops a card when the judge said nobody needs to act now. Measured the week it was added: 440 interruptions, 95% repeats.

The fold-repeats stage after it sends a card the judge did mark for a person once per condition per window: an hour, widening to four for a condition that keeps coming back. A recovery the person is waiting for always goes, and a firing that comes back after a recovery card goes an hour later unless it ends again. Replayed on the retired deployment’s ledger it sends 55% fewer cards. It has not run live, because this shape runs nowhere today.

The work shape

timer ──► scan (plain code) ──► nothing new? the round ends here. No event,
                             │  no model, no bill — which is most rounds.
                             ▼
              /hook/watch-due ──► watcher   reads findings, decides which are
                                     │      worth interrupting a person for
                                     ▼
                          /hook/watch ──┬──► chat A  (the signal)
                                        │
                        (kind=task) ────┴──► planner ──► /hook/plan-notify ──► chat B

The scan is not the watcher. Reading a window of chat and tickets, comparing timestamps against a cursor and dropping fragments is procedure, and procedure written into a prompt is procedure a model can decline to follow — it declined once, posting a signal and advancing neither cursor. It is a script now, and the watcher is handed findings rather than instructions for finding them. The saving is not tokens: an empty round used to cost as much as a busy one, because the agent had to complete the whole scan to learn there was nothing in it.

The timer’s side of that is PATROL_PRESCAN in patrol-timer.sh, which is generic — a prescan that exits quiet skips the round, and one that FAILS fires anyway, carrying the failure. A prescan that broke and a prescan that found nothing are the same silence from outside, and only one of them is a quiet day.

The cursor moves when the round is known to have landed (2026-10-19). The scan used to write its new cursors before the model ran, so a round whose run failed was never looked at again; a second deployment running a copy of the scanner lost two rounds that way in one day. Now a round that fires stages its cursors (and Jira’s state) and the next tick settles them against the node’s own run record for the event the pipe numbered, which the timer writes down after the POST: completed commits, failed, never-posted, never-routed or timed out puts the cursors back and re-scans with a line saying so, a run still going holds the tick rather than offering the same messages twice. After two replays and an hour the first lost round’s content is skipped, named in the digest until a round lands. Jira is not skipped: its lookback re-reports the span, and a repeat is a nuisance where a miss is a failure.

Two chats rather than one, which is the whole reason this is not the alert shape: a signal (“someone assigned you this”) and a plan (“here is how it would be done”) are read at different moments. One bot would interleave a five-second notice with a five-minute investigation of the notice before it.

No judge. The watcher already decided what deserves attention, and a judge calibrated on alerts — severity keywords, recovery semantics, flap suppression — has no vocabulary for a colleague’s question.

The first real write

On 2026-09-30 the work shape made its first change with a write credential, end to end:

  1. A request came in through the watch door: a daily cleanup of application logs older than three days on the three app nodes of a test cluster.
  2. The planner, read-only, answered with a plan, the permission it lacked, and one question: when to run it. The operator answered in the card’s thread, 03:00 Asia/Shanghai, and pressed Act on this plan.
  3. The work node held a write credential of its own for the first time. It installed a cleanup script, a systemd service and a timer on each node through AWS Systems Manager, ran the cleanup once, and read every node back. About 74 GB came free across the three. Each timer was enabled with its next run at 03:00, and the files on the nodes matched its commits by hash.
  4. It returned the change as commits in its scratch clone, a 79-line diff, in four minutes for $0.05. The pipe’s board shows the request, the plan, the press and the work as one row.

The report also said what it could not do. There was no toolchain for a static check of the units, and its first command came back failed after the work was done, from a bug in its own check; a second command confirmed the result. It named the catch for a person: the nodes belong to a managed node group, so a replaced node comes up without the timer until the launch template carries it.

An earlier handoff the same day ran on the planner’s read-only credential. It could not run a command on its nodes, changed nothing, and said so. That is why the board grades a work report by what it changed rather than by the fact that it came back (the decision).

The credential was the operator’s own administrator profile, far wider than the job needed. It was mounted for the run as a directory and two .env lines (PROBE_WORK_AWS_DIR, PROBE_WORK_AWS_PROFILE) and taken off the node once the result had been read back; the node’s posture check measured no write actions afterwards. The next write credential is to be scoped to its job — and the run above is the handoff path, not the allowlisted one: a plan a person acted on, not a procedure the executor ran.

The executor’s gate files, since 2026-10-06. The write node also carries the remediation executor’s two gate knobs (HOOKPROBE_REMEDIATION_ALLOWLIST, HOOKPROBE_REMEDIATION_HIGH_RISK_ALLOWLIST), empty by default: wiring them arms nothing, they make the operator’s files nameable at all. The files live in hookprobe/deploy/operator/ (git-ignored; remediation-allowlist.example beside them says how to write one) and are mounted read-only at /etc/hookprobe/operator/ on probe-work alone — the write node, the one place an approved procedure can change anything. A procedure parked there is approved on the runner’s own card since 2026-10-08 — the pipe routes approve by the door the card came in through, so the press goes back to probe-work — or in its console (127.0.0.1:8090); the card offers the button only once the runner’s allowlist has a rule. A scoped credential and one allowlisted line complete the first real remediation; until both exist the knobs stay empty and nothing runs.

Four decisions that differ, and why

dedup: off for alerts, on for work signals. In the alert shape the judge does the noise accounting, and two counters mean two ledgers and neither one true. In the work shape nothing else counts: the watcher dedups by TOPIC in its own state, which is a different question from “is this the identical signal again” — a crashed watcher re-running replays what it already sent.

Return doors: signed on both, since 2026-09-07. They used to differ — unsigned on the server as “an in-network hop between two containers of one deployment”, signed on the laptop where every process on this machine could otherwise post a fabricated card into somebody’s chat. The server’s premise stopped holding the day the compose network started carrying the platform’s observability containers (Prometheus scraping, Alloy collecting): “in-network” no longer meant “ours”. Both return doors now verify the same secret their brain signs with, and scripts/deploy_preflight.py refuses a deploy whose .env would leave any door unsigned — the baseline the compose comments described is enforced, not remembered.

Escalation gating: level alone, or level and kind. The investigator gates on level and uses fields.kind only to pick which prompt it runs. So a work deployment whose contract is “task AND high buys a plan” has to express the kind half in its ROUTE TABLE — left as one fan-out, a note at high quietly funds a plan nobody asked for.

Posture: read-only on both, plus one danger-only node on the work side. Every investigator that faces an event door runs HOOKPROBE_BASH_GUARD=readonly on both deployments, and since 2026-09-07 each one measures its mounted credentials against that declaration at startup and refuses to start when they are wider. The work deployment adds a node the alert deployment has no use for: probe-work, running the same image under danger-only, reachable only through the handoff a person clicks on a plan card. Its posture check records the credentials’ blast radius instead of confirming a boundary. By default those are the planner’s read-only credentials and the record is a list of zeros; since 2026-09-30 the node can hold a write credential of its own (PROBE_WORK_AWS_DIR, PROBE_WORK_AWS_PROFILE), and that list is what an operator reads before approving a handoff. The alert deployment’s investigator has no credentials mounted at all and says so (no-credentials). Since 2026-09-22 the handoff is also the first real use of the write posture: the plan-approved door speaks kind: brief, its body is the instruction to carry the plan out and commit locally, and probe-work mounts the plan’s tree writable at /data/code (PROBE_WORK_CODE) — the one writable mount on any node. The posture check measures credentials, not mounts, so what that variable points at is a grant the operator makes by hand: a scratch copy first, the real tree only after a run’s commits have been read back. A handoff quotes nothing; it carries the plan’s work item, the hr-<id> the pipe minted for the request, and the pipe’s journey and board gather the work run under that request by it. The work run’s report says how many files its commit-backed diff changed and how many gaps it named, so the board’s fix stage reads carried out, blocked or failed rather than “a report came back”.

The chat’s tools are behind a gate, since 2026-09-30. The investigator’s MCP allowlist is enforced where the agent asks for a tool and says nothing about a socket. Measured that day: one plain HTTP request from inside probe-watch reached the chat client’s own MCP port, with no credential, listed 34 tools — 20 of which write, including sending a message as the operator — and called one the allowlist does not name. The watcher reads colleagues’ messages for a living, so that is attacker-influenced text one injection away from posting as the operator.

So the list became a place. The three probes now sit on a docker network with internal: true: no route to the host, no route to the internet, no DNS for either. Their way out is the egress proxy on its allowlist, as before, and their way to the chat is mcp-gate (deploy/sidecars/gate.py), which holds the only reach to the chat client and forwards a tools/call only when the tool’s name is on the calling node’s list. The watcher now sees eight read tools, the planner eleven, and every refusal is recorded in work-data/mcp-gate/calls.jsonl. An internal network answers no published port, so the three consoles moved to probe-ingress, a byte pump on both networks, at the same addresses as before. Since 2026-10-06 the proxy, the gate and the ingress run in one container, doors (deploy/sidecars/doors.py), that keeps the three names as aliases.

The watcher’s signature moved out of its reach, the same day. post_watch_signal.py signs from a file, which is right on the operator’s laptop and upside down inside probe-watch: the process deciding what to post is an agent whose input is colleagues’ messages, and it could read the secret. The door’s only check — “signed by something holding the secret” — then answered yes to whatever an injected round chose to say, including a fabricated request at high as a task, which buys a paid planner run and a card that reads as real. the watch signer (deploy/sidecars/signer.py) holds the secret now. The watcher posts unsigned to it; it signs a signal only when the signal names a conversation the scanner offered that round, with a level and kind from a closed set, text cut to a length and a ceiling per round. The refusals are in work-data/watch-signer/signals.jsonl.

The pipe’s ledger is a bind mount, and only the container opens it. work-data/hookrelay/hookrelay.db runs in WAL (hookrelay/store.py) on a directory the desktop container runtime shares from the host, and WAL’s shared-memory file is not coherent across that boundary: a sibling deployment on this machine records it as a silent-corruption hazard and runs its databases in rollback mode. Nothing on the host opens this one while the pipe runs, and that is the rule rather than a habit. Read it with the container stopped and the file opened immutably (?immutable=1); the integrity check of 2026-10-19 was clean at 446 events.

What a review of those two days found, fixed on 2026-10-05. The timer’s contract-violation signal had signed from the file the watcher’s copy was deleted from, so every violation since the redeploy failed to post with one log line nobody read; the timer now signs from work-data/watch-timer-secrets/watch.env, a directory only it mounts. The signer refused the scanner’s own fault notes (a source that cannot be read was becoming a quiet round); the brief now posts them as scanner / scanner-notes, which the signer admits without an offer and forces to low/note. A deployment with no prescan hit a lifetime ceiling of twenty signals; the ceiling now counts per twenty-minute window when the scan states no round. The plan-approved door’s secret was never on the list of secrets withheld from the agent, so the planner’s own Bash step inherited the one credential that starts a work run with the write credential — it is withheld now, which is what handoff.py had claimed all along. The gate and the signer refuse a bearer with a non-ASCII byte instead of dropping the connection, bound and validate the request length before reading it, report an upstream that dies mid-answer as a 502 rather than a closed socket, pass the client’s JSON-RPC responses through, and write forwarded only after the chat server answered. probe_net sets the isolated gateway mode, so “no route to the host” holds on a Linux engine and not only on this laptop; the probes’ MCP config, which is not tracked, has a tracked shape in deploy/sidecars/mcp.example.json. The same day, second pass: the scanner keeps the round before alongside the current offer, and the signer and the watch wrapper admit a conversation from either, counted against its own round, so a run that outlasts a twenty-minute tick is no longer checked against the offer that replaced its own; and the producer half of an origin is a closed set (the chat tool, Jira, the scanner), so a round cannot label a chat finding as a Jira one. Third pass: the signer’s ledger is the record of what was reported. Every signed signal carries the conversation and the cursor the scan offered it at, written by a process the agent cannot reach, and the watch wrapper writes no state at all. The next day the before/after contract check retired with it: once the signer held the record, two of its three promises held by construction, and the third is the one check the timer still runs each tick (scripts/assert_watch_signed.py): a watch signal in the pipe with no signer row went around the boundary.

The boards on a phone, since 2026-10-05. All three boards are web apps (the judge’s and the investigator’s since 2026-10-06): a manifest, icons and a service worker, linked with the page’s own base so a board behind a path prefix installs under it; an investigator’s manifest carries its node’s name, so the planner, the watcher and the work runner are three icons, not one. A browser installs an app only over https, so a board has to be reachable from the phone over https first — on a laptop, a tailnet (tailscale serve https / http://127.0.0.1:8100) or a tunnel; the work stack’s board is on 127.0.0.1:8100, its three consoles on 127.0.0.1:8088–8090, and nothing here opens them to the network. Then: open the address, paste the read token in Settings (the app keeps it in that browser and nothing else), and Add to Home Screen. The worker keeps the shell and never the data, so offline the app shows the page and says its service is out of reach. The operator runs the pipe’s board and the watcher’s console this way, through a tunnel of their own. iOS keeps the status-bar style an icon was added with, so after an update that changes it the icon has to be added again; each board’s settings drawer says so when it finds itself under the old, opaque bar.

The chat is behind a protocol, not in the pipe. Both deployments reach Feishu through a bridge channel and the lark-bridge sidecar: the pipe sends a card model — title, tone, summary, links, actions as plain facts — and the sidecar renders, posts as the app, and carries presses and replies back (docs/bridge-protocol.md). Nothing in the pipe, the judge or the investigator names the platform; another IM is another sidecar, tested against the same fixture. The operator’s personal watch bot is a custom-bot webhook, so it gets a second bridge in webhook mode (lark-bridge-watch): same image, no app, the rendered card posted to the bot’s URL. The pipe has no Feishu channel type any more; DingTalk and WeCom markdown are a shipped plugin.

Follow-ups from the chat thread: alert deployment only, since 2026-09-08. A person who replies under a verdict or report card in the group continues that alert’s investigation: the bridge forwards the reply, signed, to the pipe’s lark-thread door; the thread_lookup stage resolves the card’s platform id against the ledger to the chain and its investigation session (a reply under a card the pipe never sent is skipped as unknown_thread); the investigator continues its own engine session under the same read-only posture and answers back through probe-notify with thread_root, which the to-me channel turns into an in-thread reply. Three things have to be true on the host for this to work, none of them in this repository (deploy/lark-bridge/README.md has the full list). The bridge needs a Lark app of its own: Lark allows one event long-connection per app, so an app shared with any other consumer means one of them never receives anything (2026-09-08: the bridge moved to a dedicated app for exactly this reason). That app must be allowed to receive the group messages that mention it (im:message.group_at_msg:readonly; im:message.p2p_msg:readonly is what the CLI lists for the event) and to send as a bot; a reply therefore has to @-mention the bot to reach the bridge, unless the operator also grants im:message.group_msg:readonly, which reads every group message. The im.message.receive_v1 event must be enabled in the developer console, and the bot must be a member of the group — added by a person; no bot can add itself and the old app lacked the scope to add another. Then .env: LARK_THREAD_SECRET (the door and the bridge share it) and HOOKPROBE_FOLLOW_UP_SENDERS (who may spend a turn from chat; empty refuses everyone). Switching the bridge’s app means changing LARK_APP_ID / LARK_APP_SECRET and recreating the container: the bridge keeps no state, so a fresh one re-derives the CLI config from the environment. A bare docker restart reuses the writable layer, where the old config still is, and that is the case needing lark-cli config remove first. Check the new app against the chat BEFORE cutting over: an app and a chat in different tenants are refused outright (232010) and no membership change fixes it, so a new app can force a new chat id as well. The work deployment gets the same path through its bridge profile: a lark-bridge of its own (a third Lark app — one connection per app, and the alert deployment’s bridge holds the second), serving both of its chats (LARK_WATCH_CHAT_ID, LARK_PLAN_CHAT_ID; the pipe names the chat per channel with options.chat_id). Its two chat channels are type: bridge and post through that bridge (LARK_WATCH_URL / LARK_PLAN_URL, channel secrets equal to LARK_BRIDGE_SECRET); a reply is routed by return_source to the node that holds the session — plan-notify to the planner, work-notify to the work node. And a person can open a topic instead of waiting for a card: a top-level @-mention in the group starts a work item on this deployment (on_new_topic: {kind: task}) and an investigation on the alert deployment ({kind: brief}: the message is the question); the first card lands inside the topic and the conversation continues there.

Timer: host crontab, or a container. The work deployment ships its own, after the host version could not read its own brief: macOS keeps ~/Documents behind TCC and cron is not allowed through, so every fire logged Operation not permitted while the pipe had no delivery to fail, the investigator sat healthy and idle, and the chat stayed quiet. A watcher that was never triggered and a quiet afternoon look identical from outside. patrol-timer.sh is the general form.

Three kinds of event, one door

POST /hooks/event dispatches on fields.kind:

kind The question Body cap
absent, or anything else what broke — root cause, impact, remediation 4 KB
task how would this be done — what exists, what is missing, the steps 4 KB
brief the body IS the instruction: read it and do what it says 16 KB

brief exists because the other two presume a SUBJECT to analyse and wrap the body in that framing. A scheduled procedure has neither shape, and the framing fights it: given a brief and the alert wrapper, one run did the brief’s first step, then the wrapper’s (“open the case files first”), called no MCP tool at all, and answered with the brief’s silence token. A blend of two instructions.

Its larger cap is not generosity. Truncating an alert loses detail about one incident; truncating a procedure deletes STEPS, and the run then does most of a job and reports success.