Operations

Monitoring, replication, backup, and deployment for production context optimization workloads.

Monitoring

Prometheus Metrics

Nocturnus exposes 20+ metrics at GET /metrics in Prometheus format:

Fact operations — nocturnusai_facts_asserted_total, nocturnusai_facts_retracted_total
Inference — nocturnusai_inference_duration_seconds, nocturnusai_rules_fired_total
MCP — nocturnusai_mcp_tool_call_duration_seconds, nocturnusai_mcp_sse_subscribers
LLM — nocturnusai_llm_call_duration_seconds, nocturnusai_llm_tokens_used
Memory — nocturnusai_memory_facts_total, nocturnusai_memory_decay_evicted

Track context payload efficiency from API responses too: monitor totalCharCount, totalFactsIncluded, and diff churn (added/removed) over time.

Grafana Dashboards

The Docker Compose profile includes pre-built Grafana dashboards:

# Start with monitoring stack
docker compose --profile monitoring up -d

# Open Grafana at http://localhost:3000
# Default credentials: admin / admin

Health Probes

Two endpoints for Kubernetes readiness and liveness:

# Liveness — confirms server is running
curl http://localhost:9300/health

# Readiness — confirms all subsystems are initialized
curl http://localhost:9300/health/ready

Logging

Nocturnus uses structured JSON logging via SLF4J. Configure via environment variables:

# .env
LOG_LEVEL=INFO          # DEBUG, INFO, WARN, ERROR
LOG_FORMAT=json         # json or plain

Every request gets a X-Request-ID response header for correlation.

Persistence

The in-memory Hexastore is persisted via Write-Ahead Log (WAL) and periodic snapshots:

WAL — every mutation is appended to a log file before being applied. On restart, the WAL is replayed to restore state.
Snapshots — periodic full-state dumps for faster recovery. Old WAL segments are deleted after a snapshot.

# .env
STORAGE_DIR=/var/lib/nocturnusai/data   # WAL + snapshot directory

Replication

Nocturnus supports leader/follower replication via WAL streaming. Deploy read replicas to scale query load.

Leader

# .env (leader)
REPLICATION_MODE=LEADER
# The leader runs normally — no additional config needed

Follower

# .env (follower)
REPLICATION_MODE=FOLLOWER
LEADER_URL=http://leader-host:9300
REPLICATION_POLL_INTERVAL_MS=5000

Followers automatically poll the leader's WAL and apply changes locally. Queries against followers are eventually consistent.

Kubernetes Deployment

Example Kubernetes deployment with health checks:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nocturnusai
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: nocturnusai
          image: nocturnusai/server:latest
          ports:
            - containerPort: 9300
          livenessProbe:
            httpGet:
              path: /health
              port: 9300
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 9300
            initialDelaySeconds: 5
            periodSeconds: 10
          env:
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: nocturnusai-secrets
                  key: anthropic-api-key
          volumeMounts:
            - name: data
              mountPath: /var/lib/nocturnusai/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: nocturnusai-data

Agent-to-Agent Discovery

Nocturnus serves an Agent2Agent Protocol discovery card at GET /.well-known/agent.json. Other agents can automatically discover and interact with your fact database.