rocketgraph-systemd-agent

Fleet-wide discovery agent for unmanaged systemd services on bare metal — ships their journald logs into Rocketgraph. See PLAN.md for the full design writeup and open questions.

What's here

services/          10 fake bare-metal workloads (bash scripts, write to stdout -> journald)
  lib/common.sh     shared helpers (timestamp, log(), emit_stack(), on_shutdown())
units/              matching systemd unit files (rg-<name>.service / .timer)
otel/
  otelcol-rocketgraph.yaml    production collector config: journald -> OTLP -> Rocketgraph
  otelcol-local-debug.yaml    local-only variant: journald -> debug exporter (no network, no API key needed)
install-services.sh   installs + enables all units in services/ + units/ (generalizes over N, not hardcoded)
install-collector.sh  installs otelcol-contrib and points it at otel/otelcol-rocketgraph.yaml
Dockerfile.systemd-test, docker-compose.systemd-test.yml
                    privileged systemd-in-Docker container for local validation (macOS has no native systemd)

Try it locally

docker compose -f docker-compose.systemd-test.yml up -d --build
docker compose -f docker-compose.systemd-test.yml exec rg-systemd-test bash

# inside the container:
cd /opt/rocketgraph-systemd-agent
./install-services.sh
ROCKETGRAPH_API_KEY=rg_live_... ./install-collector.sh

# verify:
curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_(sent|send_failed)_log_records'
journalctl -u rg-api-server.service -f

Tear down: docker compose -f docker-compose.systemd-test.yml down

Validated (2026-07-08)

Ran all of the above in the Docker container (arm64 Mac — install-collector.sh auto-detects amd64/arm64):

• All 10 rg-* units + the rg-cron-batch-report.timer install and run cleanly (systemctl is-system-running → running).
• journalctl captures all 10 services correctly, including multi-line ERROR/stack-trace entries and the timer-driven oneshot's per-run PID.
• otelcol-local-debug.yaml confirmed the collector pipeline end-to-end with no network calls: multiline recombine correctly stitches stack traces and JSON-payload continuation lines into single records, and service.name / host.name / deployment.environment resource attributes are set correctly per record.
• otelcol-rocketgraph.yaml (the real production config) run against a real ROCKETGRAPH_API_KEY: otelcol_exporter_sent_log_records climbed continuously (2625+ sent, 0 failed) with no errors in journalctl -u otelcol-contrib. Logs are live in Rocketgraph.

Known gaps / not yet done

• Scaling story (10 → 1,000-2,000): this proves the pattern, not the scale. Fleet-wide rollout needs a real install mechanism across hosts (config management / golden image), not manual install-*.sh runs per box.
• Real bare-metal/EC2 parity: the Docker path above is the local validation harness. For the production path — 1,000+ services on real x86 Ubuntu EC2, no Docker, no push agent, journald auto-discovery, and an RG_SOURCE origin flag (e.g. ubuntu-ec2-x86) — see EC2-BAREMETAL.md.
• On-prem Rocketgraph deployments: confirmed there's no implemented on-prem stack in ../dashboard (marketing copy only) — out of scope here.
• The 10 services are invented analogs, not a literal customer-provided list — swap them out if real examples surface later.