Recover the home test bed

Scope and prerequisites

Targets: the build server sn1-1f0ccb and the edge appliance sn7d-jetnpd of the home test bed. SSH as pi to both (the appliance needs the Vault-signed certificate). Nothing here touches production, customer data or the product’s cloud side. Every change on either machine is tried by hand first and recorded with its revert in the appliance repository notes changelog; config files on the appliance are described for the owner to edit.

Symptoms and diagnostics

Symptom	Read-only check	What it means
Webhook actions answered 500; receiver lines missing	systemctl is-active webhook-receiver; ss -ltn | grep 8090 on the build server	The receiver is down. It ran by hand until 6 October and died with a reboot.
A simulator path has no reader; detection rows stop for a camera	curl -s http://192.168.1.12/api/paths (readers per path); docker ps on the build server; docker logs ingest --since 5m on the appliance	The simulator is down or the appliance dropped the stream.
ssh pi@sn1-1f0ccb.local fails from the Mac	ifconfig on the Mac: is it on 192.168.1.x? ssh pi@192.168.1.12 hostname	mDNS does not cross subnets; the address still routes. If the address itself changed, the appliance’s camera entries are wrong too.
The product cannot see the appliance: scan finds nothing, cameras cannot be added	docker logs tunnel --since 10m on the appliance for “certificate has expired”; openssl x509 -enddate -noout -in /data/home/pi/vault_secrets/cert.pem	The appliance certificate expired; vault-agent cannot log in and the tunnel cannot reach its origin.
Both machines show a fresh uptime	uptime -s on both; journalctl -b -1 | tail on the build server	A house power loss. Any run in progress is void; the receiver and the seven streams must be checked, not assumed.

Repair

• Receiver down: sudo systemctl restart webhook-receiver on the build server, then curl -s -o /dev/null -w '%{http_code}' -X POST -d '{}' http://192.168.1.12:8090/hook/test from the Mac; expect 200 and a new line at the end of /home/pi/webhooks/received.jsonl. No data is lost; the log is append-only. Exercised on 6 October when the unit was first started.
• Simulator path without a reader: if the container is gone, docker start camera-simulator; if it runs, docker restart camera-simulator brings every clip back to its first frame and drops all seven streams for about a minute, so never during a run. The appliance reconnects on its own; all seven paths were read again within 75 seconds on 6 October (two runs).
• Build server by name: use 192.168.1.12. If the address moved, check the router’s static DHCP page for the entry b0:6e:bf:1f:0c:cb to 192.168.1.12; the cameras in the product would need re-pointing, which has not been needed yet.
• Expired appliance certificate: owner’s go first. Back up /data/home/pi/vault_secrets to a dated copy, run utils-dev/reset_identity_credentials.sh -i sn7d-jetnpd from the appliance repository checkout on the Mac, then docker restart spotapp so it refetches the local SSL certificate. Exercised on 6 October 05:05 UTC: vault-agent logged in within a minute, the tunnel errors stopped, the product’s scan listed the build server. Effect on data: none; a new certificate is registered in Vault.
• After a power loss: check before trusting anything. On the build server: systemctl is-active webhook-receiver and docker ps show the receiver and the simulator up (Docker and the simulator came back by themselves on 6 October; the receiver unit has not been through a reboot yet, so its return is unverified). From the Mac: the POST test above returns 200. On the simulator: GET /api/paths shows one reader on each of the seven paths. On the appliance: docker ps lists the twenty containers. Only then discard the interrupted run and start a new one; if a check fails, use the matching repair above.

Verification

Receiver: a POST from the Mac and one from the appliance both return 200 and appear in the log. Streams: GET /api/paths shows one reader on each of the seven paths and detection rows resume on the appliance within two minutes. Cloud link: docker logs vault-agent --since 5m shows a successful login and the product lists the appliance’s cameras. Address: ssh pi@192.168.1.12 hostname prints sn1-1f0ccb.

Rollback or escalation

The receiver unit is removed with sudo systemctl disable --now webhook-receiver and deleting its unit file. The identity reset keeps the old files in the dated backup directory; they are expired, so only for forensics. If the appliance does not come back after a power loss, or the simulator does not start, stop and tell the owner; do not change boot configuration on either machine.