Agentic and local, anywhere, at all times

herdr server

[Unit]
Description=Herdr agent fleet (headless)
After=network-online.target
Wants=network-online.target

[Service]
Environment=HERDR_SESSION=main
ExecStart=%h/.local/bin/herdr server
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now herdr-fleet.service
sudo loginctl enable-linger "$USER"
systemctl --user status herdr-fleet.service --no-pager

herdr --remote devbox --session main

You are the <Seat> for <product>. Your north star is <one sentence>.

You own:
- <the surface you are accountable for>
- <the decisions you make without asking>
- <the recurring cadence you run>

Facts you never re-derive (read the file, don't guess):
- <the seams, where this seat meets the others>

You never:
- <the hard boundaries; what is out of scope>

Good output here looks like:
- <the concrete shape of a finished piece of work>

You are the Chief of Staff for <product>. Your north star is that the
owner never has to hold more than one decision at a time.

You own:
- the daily/weekly briefing: what moved, what's blocked, what needs a call
- sequencing: which seat works next, and why
- the meeting record and the follow-up list, closed to zero

Facts you never re-derive (read the file, don't guess):
- the seat roster and each seat's scope (docs/agents/)
- the open follow-ups and their owners (docs/ops/)

You never:
- build product, write copy for publication, or move money
- make a seat's call for it; you surface the decision, the seat owns it

Good output here looks like:
- a briefing in three sections (moved / blocked / needs-a-call), each item
  one line, each blocker naming the seat that can clear it

# on the box: a server key, region pinned so the address never rotates
tailcat genkey --key=default --fixed-region
# -> prints the address blob; keep it, you'll resolve it below

# on your laptop: a client key
tailcat genkey --client --key=client-default
# -> prints your client's public key; hand it to the box's owner

[Unit]
Description=tailcat relay for SSH access
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=%h/.local/bin/tailcat serve --key=default --allow=nodekey:<client-pubkey> 22
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now tailcat-ssh.service
tailcat resolve <address-blob>   # a longer, self-contained token; no DERP-map fetch needed to connect

echo 'KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512' \
  | sudo tee /etc/ssh/sshd_config.d/99-pq-kex.conf
sudo sshd -t                        # validate before reloading
sudo systemctl reload sshd          # SIGHUP: re-reads config, keeps active sessions

Host devbox
    ProxyCommand tailcat <resolved-address-blob> 22
    User dev
    DynamicForward 1080
    ServerAliveInterval 30
    ServerAliveCountMax 3
    ExitOnForwardFailure yes

# the box's owner mints one per consumer that's going to stay open,
# plus one more for everything ad-hoc: interactive sessions, one-off commands
tailcat genkey --client --key=client-tunnel      # e.g. the SOCKS service from part one
tailcat genkey --client --key=client-inference   # e.g. a second, unrelated forwarded port
tailcat genkey --client --key=client-default     # ad-hoc only -- never a persistent service

# server side: allow every one of them, not just the first
tailcat serve --key=default \
  --allow=nodekey:<tunnel-pubkey>,nodekey:<inference-pubkey>,nodekey:<default-pubkey> 22

#!/usr/bin/env bash
HOST=devbox              # your ProxyCommand host, pinned to one key (see above)
LOCAL_PORT=8080           # whatever this tunnel forwards
HEALTH_URL="http://127.0.0.1:${LOCAL_PORT}/health"   # anything the far end actually answers

start_ssh() {
    ssh -N -o ExitOnForwardFailure=yes \
        -o ServerAliveInterval=10 -o ServerAliveCountMax=3 \
        -L "${LOCAL_PORT}:127.0.0.1:${LOCAL_PORT}" "$HOST" &
    SSH_PID=$!
}

start_ssh
fails=0
while true; do
    sleep 10
    if ! kill -0 "$SSH_PID" 2>/dev/null; then start_ssh; fails=0; continue; fi
    if curl -s -o /dev/null --max-time 8 "$HEALTH_URL"; then
        fails=0
    else
        fails=$((fails + 1))
        [ "$fails" -ge 3 ] && { kill -9 "$SSH_PID"; start_ssh; fails=0; }
    fi
done

pick() {
    # Bounded-timeout probe against the real endpoint. Reading this
    # machine's own interface prefix and comparing it to the LAN's is
    # not a location check: 192.168.1.0/24 is also the out-of-the-box
    # default on plenty of routers that are not this one, so the
    # prefix can match on a network you're only visiting -- and then
    # the picker dials a LAN IP that isn't there and just hangs.
    if nc -z -G 2 "$LAN_IP" 22 2>/dev/null; then
        echo devbox-lan
    else
        echo devbox-relay
    fi
}

while true; do
    host=$(pick)
    ssh -N -D 1080 -o BatchMode=yes -o ExitOnForwardFailure=yes "$host"
    sleep 10
done

hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --local-dir ~/models

llama-server \
  -m ~/models/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -ngl 999 -fa \
  -c 32768 --cache-type-k q8_0 --cache-type-v q8_0 \
  --host 127.0.0.1 --port 8080

Set up a persistent, remotely-attachable agent fleet on a Linux dev box, plus
a relayed SSH transport with post-quantum key exchange so it is reachable
from any network. I'll fill the placeholders.

MY SETUP
- Box: a Linux machine I can SSH into as user <dev>. It stays on; it is a
  server, not a laptop.
- Existing SSH entry in ~/.ssh/config:
    Host devbox
        HostName <lan-ip>
        User <dev>
        DynamicForward 1080
        ServerAliveInterval 30
        ServerAliveCountMax 3
        ExitOnForwardFailure yes
  (drop any of the last three lines that aren't actually there; don't add
  ones I don't have)
- A persistent SOCKS tunnel service already dials this host in the
  background (autossh under a user service, or the launchd/Task
  equivalent). Name: <service name>.
- herdr is installed on the box (verify with: herdr --version). If it is not
  installed, stop and ask me.
- tailcat is installed on the box and on my laptop (verify with:
  tailcat version). If it is not installed, stop and ask me.
- GPU on the box (optional, only needed for step 4): <card and VRAM, e.g.
  "24GB">. Leave this blank and skip step 4 entirely if I don't fill it in.

DO THIS
1. FLEET. Create ~/.config/systemd/user/herdr-fleet.service:
     [Unit]
     Description=Herdr agent fleet (headless)
     After=network-online.target
     Wants=network-online.target

     [Service]
     Environment=HERDR_SESSION=main
     ExecStart=%h/.local/bin/herdr server
     Restart=always
     RestartSec=10

     [Install]
     WantedBy=default.target
   (Fix the ExecStart path if herdr lives elsewhere: `command -v herdr`.)
   Then run:
     systemctl --user daemon-reload
     systemctl --user enable --now herdr-fleet.service
     sudo loginctl enable-linger "$(whoami)"
   Confirm with `systemctl --user status herdr-fleet.service --no-pager`
   that it is active (running).

2. SEATS. Create docs/agents/ on the box with eight seat files, one per role,
   each following this five-part skeleton (keep the shape identical). Base
   the role-specific parts on my actual project, not placeholders: read my
   repo's README/docs and, if present, ~/.claude/CLAUDE.md and
   ~/.claude/projects/*/memory/ for what's already true about the product
   and how I work. Ask me only for what you can't find there.
     You are the <Seat> for <product>. Your north star is <one sentence>.

     You own:
     - <the surface you are accountable for>
     - <the decisions you make without asking>
     - <the recurring cadence you run>

     Facts you never re-derive (read the file, don't guess):
     - <the seams, where this seat meets the others>

     You never:
     - <the hard boundaries; what is out of scope>

     Good output here looks like:
     - <the concrete shape of a finished piece of work>
   The eight seats: chief-of-staff, research-lead, content-lead,
   engineering-lead, design-lead, platform-ops, finance, and one domain seat
   named for my product. Every seat prepares; none of them execute: no seat
   pushes, deploys, publishes, sends, or moves money on its own.

3. RELAY + PQ SSH. On the box:
     tailcat genkey --key=default --fixed-region
     # -> prints an address blob; note it
   On my laptop:
     tailcat genkey --client --key=client-default
     # -> prints a public key; tell it to me, don't guess it
   On the box, lock the relay to that one client key from the start:
     ~/.config/systemd/user/tailcat-ssh.service:
       [Unit]
       Description=tailcat relay for SSH access
       After=network-online.target
       Wants=network-online.target

       [Service]
       ExecStart=%h/.local/bin/tailcat serve --key=default --allow=nodekey:<client-pubkey> 22
       Restart=on-failure
       RestartSec=5

       [Install]
       WantedBy=default.target
     systemctl --user daemon-reload
     systemctl --user enable --now tailcat-ssh.service
     tailcat resolve <address-blob>   # a durable, self-contained token
   On the box (needs root; hand me the exact command instead of running it
   yourself if you don't have sudo):
     echo 'KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512' \
       | sudo tee /etc/ssh/sshd_config.d/99-pq-kex.conf
     sudo sshd -t
     sudo systemctl reload sshd
   Rewrite the devbox entry in ~/.ssh/config to dial through the relay.
   Replace only the HostName line with ProxyCommand; keep every other line
   (DynamicForward, and ServerAliveInterval/ServerAliveCountMax/
   ExitOnForwardFailure if I have them) exactly as it was:
     Host devbox
         ProxyCommand tailcat <resolved-address-blob> 22
         User <dev>
         DynamicForward 1080
         ServerAliveInterval 30
         ServerAliveCountMax 3
         ExitOnForwardFailure yes
   If my persistent SOCKS tunnel service is running, restart it now so it
   reconnects through the relay instead of the stale HostName, then tell me
   to rerun its own verification (curl --socks5-hostname) before I trust
   the browser side again.

4. LOCAL MODEL. Skip this step entirely if I left GPU blank above.
     Install llama.cpp (the llama-server binary) for the box's OS and GPU
     vendor via its own install docs; confirm after with:
       llama-server --version
     Pull the model. If I gave a GPU at or above 24GB, use UD-Q4_K_XL; below
     that, use UD-Q3_K_XL instead and tell me you made that substitution:
        hf download unsloth/Qwen3.8-27B-GGUF \
          Qwen3.8-27B-UD-Q4_K_XL.gguf --local-dir ~/models
     Create ~/.config/systemd/user/llama-server.service (adjust the quant
     filename to match whichever one you pulled):
       [Unit]
       Description=local model server
       After=network-online.target

       [Service]
       ExecStart=%h/.local/llama.cpp/llama-server -m %h/models/Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 999 -fa -c 32768 --cache-type-k q8_0 --cache-type-v q8_0 --host 127.0.0.1 --port 8080
       Restart=on-failure

       [Install]
       WantedBy=default.target
     systemctl --user daemon-reload
     systemctl --user enable --now llama-server.service
     Confirm with: curl -s http://127.0.0.1:8080/v1/models
     Point one seat's model config at http://127.0.0.1:8080/v1 so at least
     one seat runs fully local; leave the rest on whatever cloud agent they
     already use.

CONSTRAINTS
- Do not touch anything outside the fleet unit, docs/agents/, the devbox SSH
  entry, the tailcat relay unit, the sshd_config.d drop-in, ~/models and the
  llama-server unit (only if step 4 runs), and restarting (never editing or
  removing) my existing SOCKS tunnel service. No other services, no repo
  changes.
- Never read or print the contents of a tailcat private key file
  (~/.config/tailcat/keys/*.private.json), an SSH private key, or any other
  secret. Public address blobs and public keys are fine to print.
- If herdr or tailcat is missing, or the box is not the machine you think it
  is, STOP and report before doing anything destructive.
- You will not have sudo. For the sshd_config.d write, validate, and reload
  steps, hand me the exact commands to run myself rather than attempting them.

KNOWN TRAPS (check these, in order, if a step fails)
- `herdr --remote devbox --session main` can start a fresh, unmanaged session
  if herdr-fleet isn't running on the box, instead of failing: it looks fine
  until the laptop closes. Check the unit's status before trusting an attach.
- Leaving tailcat's --allow unset lets any client with the address blob in.
  Set it to the client key's public key from the first enable --now, not
  after. A server-key rotation with --fixed-region --force leaves --allow
  untouched: it's keyed on the client's public key, not the server's.
- tailcat's `no-auth-ssh` service never touches the box's real sshd, so it
  never negotiates the post-quantum KexAlgorithms. Serve plain port 22.
- The idle/working/blocked/done badge is herdr's heuristic read of the pane,
  not ground truth: verify a pane's real state (ps, ports) before reusing it.
- Managed panes set HERDR_ENV=1. If a tool misbehaves inside a pane but works
  in a bare shell, that variable is the likely cause.
- enable-linger + Restart=always means a reboot does not clear state; an
  old port or branch survives. Check before assuming a fresh box.
- `--handoff` is not for moving between devices: pass on it unless you
  specifically need to survive a remote herdr update mid-attach.
- Rewriting the devbox Host block from scratch can silently drop
  ServerAliveInterval, ServerAliveCountMax or ExitOnForwardFailure if they
  were already there. Change the connection line only. And my persistent
  SOCKS service won't notice the new ProxyCommand until it reconnects: a
  working `herdr` attach afterward does NOT prove that service is healthy,
  check it separately.
- If that persistent SOCKS service picks between a LAN route and this relay
  on its own, check what it bases that choice on before trusting it. Reading
  this machine's own interface prefix and comparing it to the LAN's is not a
  location check: plenty of routers hand out the same default prefix, so it
  can match on a network I'm only visiting and dial a LAN IP that isn't
  there, hanging with nothing logged more specific than a stalled connect.
  It should probe the LAN endpoint directly instead (e.g.
  `nc -z -G 2 <lan-ip> 22`, bounded to a couple seconds) and route on
  whether that succeeds, not on what my own address happens to look like.
- If step 4 runs: this quant is a 27B vision-capable model, not the smaller
  one it may replace on an existing box, and it eats VRAM fast. If
  llama-server fails to start or the card OOMs, drop to UD-Q3_K_XL before
  lowering -c: the smaller quant buys back more headroom per GB than
  trimming context does.

HOW TO REPORT
Show me, in order:
  - the output of `systemctl --user status herdr-fleet.service --no-pager`
  - `ls docs/agents/` (should list all eight seat files)
  - the output of `systemctl --user status tailcat-ssh.service --no-pager`
  - the resolved address blob and the client's public key (never the private
    key files themselves)
  - confirmation that `sudo sshd -t` passed and `KexAlgorithms` is active
  - the final devbox block of ~/.ssh/config, diffed against what it was
    before this prompt, so I can see nothing but HostName/ProxyCommand moved
  - confirmation the SOCKS tunnel service was restarted and its own
    verification still passes
  - if step 4 ran: the output of `curl -s http://127.0.0.1:8080/v1/models`,
    which quant it pulled, and which seat (if any) now points at it
  - the exact command to attach from the laptop:
        herdr --remote devbox --session main
And list every trap that fired and how you resolved it.

# stop and disable the fleet (this tears down the panes and their agents)
systemctl --user disable --now herdr-fleet.service
rm ~/.config/systemd/user/herdr-fleet.service
systemctl --user daemon-reload

# remove the seats, if you want the box clean
rm -rf docs/agents

# stop and disable the relay
systemctl --user disable --now tailcat-ssh.service
rm ~/.config/systemd/user/tailcat-ssh.service
systemctl --user daemon-reload
tailcat genkey --delete --key=default   # and, on the laptop: --key=client-default

# drop the PQ pin (needs root)
sudo rm /etc/ssh/sshd_config.d/99-pq-kex.conf
sudo systemctl reload sshd

# stop and disable the local model server, if you added one
systemctl --user disable --now llama-server.service
rm ~/.config/systemd/user/llama-server.service ~/models/Qwen3.8-27B-UD-*.gguf
systemctl --user daemon-reload

# swap ProxyCommand back for the LAN HostName in ~/.ssh/config, leaving
# DynamicForward and any ServerAlive*/ExitOnForwardFailure lines untouched
# (or delete the whole entry, if the box is going away entirely), then
# restart part one's SOCKS tunnel service so it reconnects over the LAN