Technology Stack

Audience: contributors and evaluators comparing implementation choices. Start with Quickstart for normal usage.

Decision

Build the first implementation around Go, SQLite default local storage, ClickHouse central evidence storage, optional chDB local storage, and a React UI.

The stack should optimize for Kubernetes correctness, simple PoC iteration, single-binary local workflows, and a clean path from local evidence stores to a central service.

Recommended Stack

Layer	Choice	Reason
Collector and watcher	Go with client-go dynamic client	Best Kubernetes discovery, list/watch, resourceVersion, RESTMapper, and RBAC/SAR support.
Core API	Go	Shares filters, extractors, storage adapters, and authorization logic with the collector.
CLI	Go with Cobra	Single binary for local PoC, collection, ingestion, investigation, and benchmark commands.
Local storage	SQLite	Simple local PoC, single-file archives, deterministic tests, and easy support bundle workflows.
SQLite driver	modernc.org/sqlite	Pure Go driver keeps local PoC builds independent from CGO and system SQLite libraries.
Logging	charm.land/log/v2 behind slog	More readable terminal logs now, while preserving structured logging calls for future Bubble Tea agent UI integration.
Compression and tiering	SQLite local storage plus ClickHouse columnar codecs/S3 tiering experiments	Keeps proof storage cheap while preserving portable version semantics.
Central evidence backend candidate	ClickHouse	Append-heavy evidence storage, columnar compression, JSON/search support, and hot/cold S3 tiering fit the storage-cost profile.
Optional metadata/control storage	PostgreSQL or CockroachDB	Consider later for transactional control-plane metadata if ClickHouse owns historical evidence.
UI	TypeScript, React	Fits investigation views, timelines, topology graphs, and resource diffs.
API protocol	REST first	Easy to debug, stable for CLI and UI, and sufficient for MVP query flows.
Observability	OpenTelemetry plus Prometheus metrics	Required for watcher, filter, extractor, storage, and query acceptance.
Packaging	Container image, Helm or Kustomize later	Keeps local PoC separate from production deployment concerns.

Implementation Phases

Phase 1: Local PoC

Use:

Go CLI + client-go + SQLite + Zstd

Build:

• global discovery watcher,
• configurable filter pipeline,
• P0 resource-specific extractors,
• resource history store,
• facts and topology edges,
• investigation CLI,
• benchmark command.

The CLI may read and write SQLite directly. A separate API service is optional until the evidence model is validated.

Phase 2: Central Service

Use:

Go API + ClickHouse evidence backend + same collector/extractor libraries

Build:

• multi-user query API,
• Kubernetes SAR/SSAR authorization,
• audit log,
• retention and maintenance jobs,
• ClickHouse append-only evidence tables,
• UI-facing investigation responses.

Phase 3: UI And Operations

Use:

React UI + REST API + OpenTelemetry + Prometheus

Build:

• service/workload search,
• evidence timeline,
• topology-at-time graph,
• resource version diff,
• operational dashboards for watcher and storage health.

Technology Boundaries

Keep these pieces backend-agnostic:

• object identity,
• filter pipeline,
• resource-specific extractor outputs,
• version materialization strategy,
• fact and edge semantics,
• investigation ranking,
• authorization decisions.

Allow backend-specific implementations for:

• SQL schema and indexes,
• interval query indexes,
• JSON hot indexes,
• batch ingestion behavior,
• hot/cold storage policies,
• storage maintenance jobs.

Alternatives

Rust is a reasonable future option for compression-heavy or embedded storage components, but it increases early Kubernetes client and team velocity risk.

Python is useful for one-off benchmark scripts, but it should not be the main watcher or storage engine because long-running watch reliability and deployment shape matter.

Kafka and stream processors should stay out of the MVP critical path. ClickHouse is the focused MVP central evidence backend; it stays behind clear storage interfaces so the SQLite default local path and chDB-enabled local variant remain stable.

Selection Criteria

Use these criteria when revisiting the stack:

1. Kubernetes watch and RBAC correctness.
2. Write throughput under high resource churn.
3. Historical reconstruction latency.
4. Fact and topology query latency.
5. Storage footprint and maintenance behavior.
6. Local PoC to central service migration cost.
7. Debuggability during incidents.
8. Team familiarity.