Technical stack

Every dependency below is permissively licensed (Apache 2.0, MIT, BSD, ISC or MPL-2.0), which is a constraint this project holds rather than a preference: a copyleft dependency would reach the Apache 2.0 codebase.

Stand on the Shoulders of Giants: fluxrig is proudly built upon the open source ecosystem. The projects below do the work this one is built on, and the communities maintaining them deserve the credit for it.

Mixer & Rack (OSS)

• Language: Go (golang) 1.26+
  • Stability policy: We adhere to the level n version policy (bleeding edge). We track the latest stable Go release.
  • Rationale: To support modern integrations (e.g., bento, Wasm) and leverage cutting-edge ergonomics (iterators), we accept the trade-off of being on the latest release.
  • Deployment targets: the Rack is pure Go and builds fully static with CGO_ENABLED=0, so it ships as a native Linux executable or in a scratch container. The Mixer requires CGO (the embedded DuckDB store), so it cannot be built with CGO_ENABLED=0.

Rack build variants

The Rack ships in two variants. Binaries are stripped (-ldflags "-w -s"); sizes below are measured on the current release:

Variant	Build	Size	Contents
Full	make build	34 MB	All built-in gears, including bento
Lean	make build-lite (-tags nobento)	20 MB	Everything except the bento gear

The bento gear accounts for 14 MB (~43%) of the full Rack, because it links Bento's engine plus protobuf, cue, avro and gojq transitively. Deployments that do not use type: bento can drop all of it with the lean build.

The tag also decides the licences in the binary. Bento reaches hashicorp/golang-lru, the only Mozilla Public License code in either binary; a lean build links 61 modules and every one of them is permissive.

IMPORTANT

On a lean Rack, a scenario declaring type: bento fails at activation with unknown gear type: bento. This is deliberate and loud, rather than a silent no-op. Use make build-all-variants to produce both, and check which variant a binary is by the gear types it registers.

The Mixer is 105 MB, dominated by the statically linked DuckDB engine, and 91 MB with -tags nobento. It does link the gear factory, so the tag applies to it as well.

• Orchestration: Temporal.io (Go SDK) - [Roadmap], for durable long-running business workflows. Scenario orchestration today is NATS subject-push from the Mixer, with no Temporal dependency.

• Wasm sandbox: [Roadmap] Secure, polyglot execution of custom business logic in Rust, Go, or TypeScript. (The Wasm filter gear itself is already available, see the gear runtime below.)

• ISO8583 library: Moov-io/iso8583 (apache 2.0) - core parsing engine (used in both native core and Wasm gears).

• API architecture: cbor (data plane) + Rest (control plane).

  • Interface definition: Go structs (serialized via cbor).
  • Control plane: Rest API (served by Mixer).
  • Internal transport: cbor messages over NATS (snake protocol).

• Messaging & streaming:

  • Transport: NATS JetStream (Synadia) - chosen for high-performance, distributed persistence (snake protocol).
    • Streams: See wire protocols for the detailed subject topology (flux.ctrl.>, flux.msg.>, flux.telemetry.>).
  • Library: Watermill (mit) - Go library for building event-driven applications.
    • Role: Standardizes the "publisher/subscriber" interface. Current: NATS (durable). Planned: gochannel (RAM) for "turbo wire".
    • Benefits: Middleware (OTel, pivot tracing), mockability, and router pattern.

• CLI framework: Cobra (apache 2.0) - standard library for building powerful CLI applications.

  • Key commands: Run, inspect config, inspect logs, logs, data query

• Configuration: Koanf (mit) - lightweight, extensible configuration management (toml/yaml/env) replacing viper.

• Logging:

  • Library: Log/slog (Go stdlib) - structured, high-performance logging.
  • Rotation: Lumberjack (mit).
  • Environment overrides:
    • FLUXRIG_TRACE=1 - Override all log levels to trace (useful for debugging).
    • FLUXRIG_DISABLE_TELEMETRY=1 - Disable telemetry shipping.

• Gear runtime: Wazero (apache 2.0) - zero-dependency WebAssembly runtime for Go, enabling secure, platform-independent gear execution. Available today for the Wasm filter gear; polyglot source gears are [Roadmap].

Integration & ecosystem

• Universal I/O engine: Bento (mit)
  • Fork: We explicitly use the WarpStream Labs fork (mit) to ensure permissive licensing, avoiding the Bento/Redpanda (BSL) restrictions.
  • Role: Provides the bloblang mapping language and connectors. The shipped binary registers the pure and local I/O sets (file, csv, socket, websocket, HTTP, subprocess); the ecosystem's institutional connectors (Kafka, S3, AMQP) need a custom build.
  • Integration: Wrapped as a native Gear (pkg/gears/native/bento).

Data & analytics

• Operational db:

  • Standard tier: DuckDB (embedded). high-performance OLAP + oltp for local state.
  • Enterprise: Distributed sql (external). supports ha, replication, and concurrency for Mixer state.

• Analytics dw: DuckDB (mit) + parquet (apache 2.0) - high-performance OLAP on local storage.

• Visualization:

  • Oss: Grafana - industry standard for observability dashboards.
  • Enterprise: Signoz (open source apm) - evaluation candidate (full-stack apm & tracing ui).

• Secure store:

  • OSS: AES-256-GCM (software encryption).
  • Enterprise: HashiCorp Vault or AWS KMS.

Scenario visualization & management UI

• Topology model & viewer: LikeC4 (mit) - architecture-as-code model with interactive drill-down diagrams. The CLI generates a model from any scenario (fluxrig scenario viz); viewing uses the likec4 toolchain at development time, never in the shipped binaries.
• Console framework [Roadmap]: React (mit) + Vite (mit) - the operation console builds on @likec4/diagram, which renders via React flow (mit), and ships embedded inside the Mixer binary (no external toolchain at runtime).
• Code editor [Roadmap]: Monaco editor (mit) - embedded for editing specs and scenarios with server-side validation.

AI & analytics (local) (future / research)

• Inference engine: ONNX Runtime / llama.cpp - for running AI models locally within the container.
• Models:
  • Scikit-Learn (BSD) (Isolation Forest) for Anomaly Detection.
  • Quantized LLMs (e.g., Llama-3) (Community License) for the Natural Language Assistant.

Observability, tracing & ID generation

fluxrig uses a simplified observability strategy based on OpenTelemetry and the DuckLake pattern:

Tier	Backend(s)	Scale	Phase
Embedded	DuckDB + Parquet (S3/Local)	< 100M events/day	Stable (current release)
Standard	Arc + OpenSearch	Team deployments	[Roadmap]
Enterprise	ClickHouse + SigNoz	Petabyte scale	[Roadmap]

Category	Tool/Library	Type	Purpose
Telemetry SDK	OpenTelemetry Go (Apache 2.0)	Library	Metrics, Logs, & Distributed Tracing.
Query Engine	DuckDB (MIT)	OLAP	Stateless SQL engine for Metrics & Traces (Embedded).
Query Lang	PRQL (Apache 2.0)	Language	Time-series optimized query language (compiles to SQL).
Logs	OpenSearch (Apache 2.0)	Search	Evaluation Candidate for Log Search.
Analytics	ClickHouse (Apache 2.0)	Database	Evaluation Candidate for Petabyte-scale OLAP (Enterprise).
ID Generation	Google/uuid (BSD)	Library	Distributed 128-bit UUID v7 (RFC 9562)
OTLP Export	OTLP	Standard	Vendor-neutral export to any OTel-compatible backend.

Ref: See Observability Architecture for detailed tier documentation.

Testing, security & compliance

Security First: fluxrig development follows strict security standards (PCI-DSS, OWASP, SSF) from the very beginning. All architectural decisions, dependency choices, and coding practices are audited to ensure compliance.

• Testing frameworks:

  • Robot framework (Apache 2.0): Keyword-driven testing for E2E and Acceptance testing.
  • Python (PSF): Scripting language for advanced test scenarios and the fluxrig SDK.
  • Testcontainers (MIT): Ephemeral Docker containers for integration testing.

• Static analysis (SAST):

  • Gosec (Apache 2.0): Go security checker (SQLi, credentials, etc.).
  • Govulncheck (BSD): Go vulnerability database checker.
  • Bandit (Apache 2.0): Python security linter.

• Container & secret security:

  • Trivy (Apache 2.0): Comprehensive container and filesystem vulnerability scanner.
  • Trufflehog (AGPL - External CLI): Scans git history for high-entropy secrets.

• Code coverage:

  • Go cover: Built-in Go test -coverprofile for tracking unit test coverage.
  • Standards:
    • Core Libraries (pkg/*): > 70% required.
    • Overall project: > 60% required.
  • Coverage.py: Code coverage measurement for Python scripts.

Utility libraries

fluxrig also utilizes these key utility libraries:

• Gin: High-performance http web framework.
• Google/uuid: Robust uuid generation.
• Gopkg.in/yaml.v3: Strict yaml marshaling/unmarshaling.

Documentation & build

The documentation site and PDF generation are powered by:

• Docusaurus: Modern static site generator (Web version).
• MkDocs: Static site generator (PDF Core version).
• Material for MkDocs: Theming and UX for the PDF pipeline.
• Mike: Versioning management for MkDocs (git-based).
• Docusaurus OpenAPI: Modern OpenAPI reference renderer (Scalar/Redocusaurus).
• Mkdocs-with-pdf: PDF generation plugin (using weasyprint).
• Mermaid: Diagramming and visualization (rendered via mmdc CLI).
• Python: Pre-processing scripts for validation and layout.