Telemetry & analytics

This section provides query patterns for analyzing fluxrig observability data across different storage tiers.

The Embedded Tier is the tier that the current release implements. In this tier, the system stores all telemetry (traces, logs, metrics) and business messages as Partitioned Parquet files. This "Cold Storage" strategy provides institutional audit readiness without the overhead of a centralized database, and it remains queryable via the Operational Ledger (DuckDB).

Warehouse structure

The system automatically exports data from the Rack's active buffers to the local Warehouse with an hourly partitioning scheme:

• Logs: data/telemetry/logs/YYYY/MM/DD/HH/logs_<timestamp>.parquet
• Metrics: data/telemetry/metrics/YYYY/MM/DD/HH/metrics_<timestamp>.parquet
• Messages: data/messages/<wire_id>/YYYY/MM/DD/HH/messages_<wid>_<timestamp>.parquet

Hybrid analysis

The platform uses DuckDB's read_parquet capabilities that the Registry exposes to join active state with the hourly archives.

-- Analyze errors by Gear across active memory AND cold storage
SELECT gear_id, count(*) as error_count
FROM (
  SELECT gear_id, level FROM active_logs
  UNION ALL
  SELECT gear_id, level FROM read_parquet('./data/telemetry/logs/**/*.parquet')
)
WHERE level = 'error' AND ts > NOW() - INTERVAL '4 hours'
GROUP BY 1 ORDER BY 2 DESC;

Business intelligence (fluxspec)

Analyze promoted fields from business messages.

-- Analyze transactions by BIN and calculate totals
SELECT bin, currency, SUM(amount) as total
FROM './data/messages/fluxSpec/visa-v1/**/*.parquet'
WHERE ts > NOW() - INTERVAL '24 hours'
GROUP BY 1, 2 ORDER BY total DESC;

---

Real-time metrics (Prometheus / OTel)

For DevOps and SREs requiring real-time dashboarding and alerting, fluxrig natively exposes metrics compatible with OpenTelemetry and Prometheus.

Prometheus scraping

If you use standard Prometheus scraping, you can extract core runtime telemetry (Goroutines, Memory, GC cycles) and Gear throughput.

Endpoint (Mixer & Rack):

GET /metrics

The port depends on the configured API port (default 8090 for Mixer).

Key metrics to monitor

• flux.gear.messages_in: Throughput capacity (entering gears).
• flux.gear.messages_out: Throughput capacity (leaving gears).
• flux.gear.processing_time_ms: Gear execution latency histogram.
• flux.port.bytes_in: Ingress data volume at the port level.
• flux.port.bytes_out: Egress data volume at the port level.
• flux.nats.publish_latency_ms: Message bus propagation health.
• flux.bus.publish_errors: System-level emission failures.

Bus counters, by subject:

• flux.nats.messages_published, flux.nats.bytes_published: what the Rack or Mixer put on the bus. The byte counter covers the raw publish path (where the size is in hand); the message counter covers every publish.
• flux.nats.messages_received, flux.nats.bytes_received: what a subscriber took off the bus, including raw subscriptions (the Coat Check daemon). The byte counter covers the raw path, where the payload carries its own length.

---

Specialized logging: the TRACE level

For deep protocol inspection and high-volume signal debugging, fluxrig implements a custom TRACE log level (slog -8).

• Role: Use it for full bit-perfect dumps of incoming/outgoing payloads and complex dialect parsing results.
• Usage: Activate via the --level trace flag in the Rack or via the Mixer API.

# Start a rack with per-message tracing
fluxrig rack --level trace

CAUTION

Performance Impact: Activating TRACE level on production high-frequency gears (e.g., 1000+ tps) can generate gigabytes of logs per minute. You should use it surgically for diagnostic sessions.

---

OpenSearch analytics (enterprise tier)

For high-volume Enterprise deployments, the system indexes logs in OpenSearch for full-text search and complex aggregations.

Search patterns

Use the OpenSearch DSL to find specific events.

{
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "timeout" } },
        { "range": { "ts": { "gte": "now-1h" } } }
      ]
    }
  }
}

---

Data schema reference

Field	Type	Description
ts	Timestamp	Event generation time
trace_id	String	W3C Correlation ID
flux_id	UUID	UUID v7 Business ID
entity_name	String	Name of the Rack or Mixer
gear_id	String	ID of the Gear that generated the signal
level	String	Log level (info, warn, error, debug)