Observability architecture
The fluxrig observability strategy follows a non-intrusive model: Native Telemetry Tapping. The Rack takes telemetry from the execution path itself, on the same code path the message travels. No separate process observes it from outside.
The zero-agent advantage
There is no sidecar and no collector to install: instrumentation compiles into the Rack and Mixer binaries.
- No second process: nothing else competes for CPU or memory on the node, and nothing else has to be deployed, upgraded or firewalled.
- Unified Transport: Telemetry, logs, and control signals share the existing secure tunnels. This simplifies firewall rules and reduces network overhead.
- W3C TraceContext: fluxrig natively implements the W3C TraceContext standard, allowing it to participate in distributed traces started by upstream load balancers or client applications.
Resource efficiency
The telemetry tap lives embedded in the single Rack binary. No separate sidecar/collector process runs. Fluxrig therefore avoids the memory, CPU, and operational overhead of a multi-process observability stack. The tap shares the Rack process. No second process runs.
| Dimension | Industry Standard (Sidecar/Collector) | fluxrig (Embedded Tap) |
|---|---|---|
| Footprint | Separate collector process(es) | In-binary, no extra process |
| Operational Surface | Multi-process / Sidecar | Single Binary (reduced attack surface) |
NOTE
Comparative resource figures will appear once a reproducible benchmark is available. Earlier hard numbers were illustrative and are removed.
Operational telemetry (OpenTelemetry)
fluxrig achieves broad visibility by generating three distinct telemetry types for every transaction, fully compliant with the OpenTelemetry (OTel) standard.
- Traces: Distributed spans following a request across the entire system.
- Metrics: Latency and throughput as histograms rather than averages (latency, throughput, error rates).
- Logs: Structured, context-rich events attached directly to the transaction trace span for surgical root-cause analysis.
Multi-dimensional correlation
To bridge the gap between business operations and technical troubleshooting, every event correlates across three axes:
flux_id: The Business Context (The Transaction ID).trace_id: The Operational Context (The OTel Trace ID).entity_id: The Source Context (The specific Rack/Gear origin, utilizing UUID v7).
Telemetry during an outage
The Rack writes logs to a local write-ahead log. It ships them to the Mixer when the bus is reachable. The Rack exports metrics and traces over the bus. It does not keep them while the bus is away.
Traffic prioritization and backpressure
The Transactional Hot-Path (Business Logic) always takes absolute precedence over the Telemetry Path (Observability).
Traffic prioritization
The Rack implements a multi-lane architecture to ensure telemetry never congests critical data processing.
| Lane | Content | Priority | Mechanism | Under Extreme Congestion |
|---|---|---|---|---|
| Mission-Critical Lane | Business Logic (fluxMsg) | P0 | NATS JetStream | Guaranteed. |
| Audit Lane | Transaction Logs | P1 | Local WAL | Delayed, Never Lost. |
| Metric Lane | Metrics & Debug Spans | P2 | Buffer Management | Dropped if capacity exceeded. |
The pressure chain (fail-to-local)
To protect the system during backend saturation or network isolation:
- Backend Saturation: If the central analytics sink becomes unreachable, the management layer signals backpressure.
- Network Congestion: The secure tunnel detects pressure and restricts ingestion.
- Local Diversion: The Rack automatically diverts telemetry from the network path to the Local CBOR WAL.
- Resumption: Once the pressure clears, the Rack trickles the archived WAL data back to the central sink using a rate-limited background worker.
Compliance and governance
Deterministic sanitization
Deterministic masking [Roadmap] will scrub sensitive information at the infrastructure boundary before data enters the persistent observability bus. Sensitive fields (like PANs) then never reach the centralized telemetry backend. This sharply reduces the audit scope of the central infrastructure. Until it ships, a logic gear (for example a bento mapping) placed before the telemetry path handles scrubbing.
CAUTION
Production Logging: Enabling DEBUG or TRACE log levels may output raw hex payloads to the log stream. In production environments, restrict these levels to verify compliance with institutional "No Storage" security requirements.