Observability

MUXI emits structured events across the full request lifecycle - from initial routing through agent execution to final delivery. Stream these events to your logging infrastructure for debugging, auditing, and performance monitoring.

Why this matters: When a user reports "the agent gave me a wrong answer," you need to trace exactly what happened - which agent was selected, what tools were called, what context was retrieved from memory, and how the response was generated. Without observability, you're debugging blind.

Example use cases:

• Debugging: Trace why an agent selected the wrong tool or returned unexpected results
• Auditing: Log all tool calls and data access for compliance requirements
• Performance: Identify slow LLM calls, memory lookups, or tool executions
• Alerting: Get notified when error rates spike or latency exceeds thresholds

Event System

MUXI emits 356 typed events across the system lifecycle.

Event Categories

Event Format

{
  "event_type": "conversation.request.received",
  "timestamp": "2025-01-08T10:00:00.123Z",
  "formation_id": "my-assistant",
  "session_id": "sess_abc123",
  "user_id": "user_123",
  "data": {
    "message_length": 42,
    "agent": "assistant"
  }
}

Event Stream

Subscribe to events:

curl http://localhost:8001/v1/events \
  -H "X-Muxi-Admin-Key: fma_..." \
  -H "Accept: text/event-stream"

Events stream via SSE:

event: conversation.request.received
data: {"session_id": "sess_123", ...}

event: conversation.llm.call.started
data: {"model": "GPT-5", ...}

event: conversation.response.completed
data: {"tokens": 150, ...}

Key Events

Request Lifecycle

Tool Execution

Memory

Logging configuration (schema-aligned)

Two-tier config matching the formation schema:

logging:
  system:
    level: info
    destination: stdout        # or file path
  conversation:
    enabled: true
    streams:
      - transport: stdout      # stdout | file | stream | trail
        level: info
        format: jsonl          # jsonl | text | msgpack | datadog_json | splunk_hec | ...
        events: ["request.*", "agent.*", "tool.*"]
      - transport: stream
        protocol: http         # http/https/zmq/websocket
        destination: https://logs.example.com/ingest
        format: jsonl
        auth:
          type: bearer
          token: "${{ secrets.LOG_TOKEN }}"

• System tier: infrastructure events (startup/shutdown, MCP, A2A, errors) → single destination.
• Conversation tier: multi-stream with per-stream level/format/event filters; use events to scope (e.g., request.*, memory.*).

PII & secret redaction

Every event is redacted before it reaches any sink (stdout, file, stream, or external integration). This applies to all event tiers, including system, MCP, and workflow events - not just user-facing ones - so secrets carried in any payload are scrubbed by default.

Redaction runs in two layers:

• Luhn validation: a 16-digit run is masked only when it passes the Luhn checksum, so order IDs, timestamps, and other long digit runs are preserved. Placeholders are length-accurate (one **** group per four digits).
• Consistent tokens: entity matches are replaced with indexed tokens such as [PERSON_1], [ORG_1], [ADDRESS_1], [DOB_1]. Repeated mentions of the same value reuse the same token within an event.
• Dates of birth: generic timestamps are left intact; a date is only masked as [DOB_1] when birth context (e.g. "born", "date of birth") precedes it.

Configuration

logging:
  redaction:
    entities: true   # default; set false to disable the entity layer only

• logging.redaction.entities toggles the entity layer. The regex layer is always on and cannot be disabled.
• Entity redaction is a built-in capability (Microsoft Presidio + spaCy en_core_web_sm) baked into the default images - no extra install step.
• If the NLP model is unavailable at startup, the entity layer degrades gracefully to regex-only and logs a one-time warning; secret/structured-PII redaction is unaffected.

Metrics

Available Metrics

Prometheus Export

curl http://localhost:7890/metrics

Integration

Datadog

Forward logs:

logging:
  format: json
  output: stdout  # Datadog agent picks up

Elastic

Use Filebeat to ship logs:

filebeat.inputs:
  - type: log
    paths:
      - /var/log/muxi/*.log
    json.keys_under_root: true

Custom Webhook

Forward events:

observability:
  webhook:
    url: https://your-service.com/events
    events:
      - "error.*"
      - "conversation.completed"

Debugging

Enable Debug Logs

logging:
  level: debug

Trace Requests

Add trace ID:

curl -H "X-Trace-Id: trace_123" ...

Trace ID appears in all related events.