Knowledge & RAG

MUXI's knowledge system lets agents answer questions from your documents - PDFs, markdown, spreadsheets, images, and more. Automatic indexing, semantic search, and multimodal support built in.

How It Works

sequenceDiagram
    participant U as User
    participant A as Agent
    participant K as Knowledge Index
    participant D as Your Documents

    U->>A: "What's our refund policy?"
    A->>K: Semantic search
    K->>D: Retrieve relevant chunks
    D-->>K: "Refunds within 30 days..."
    K-->>A: Context injected
    A->>U: "Our refund policy allows..."

sequenceDiagram
    participant U as User
    participant A as Agent
    participant K as Knowledge Index
    participant D as Your Documents

    U->>A: "What's our refund policy?"
    A->>K: Semantic search
    K->>D: Retrieve relevant chunks
    D-->>K: "Refunds within 30 days..."
    K-->>A: Context injected
    A->>U: "Our refund policy allows..."

1. You add documents to knowledge/ directory
2. MUXI indexes them at startup (embeddings + chunking)
3. User asks a question related to your docs
4. Agent searches the index semantically
5. Relevant context injected into the prompt
6. Agent answers using your domain knowledge

Supported Formats

Office documents convert with anydoc as the primary engine, with MarkItDown as a per-file fallback and as the converter for everything anydoc does not claim (HTML, images, audio, and more). PDFs route to a dedicated pdf-inspector engine, with MarkItDown as fallback. Conversion runs in a sandboxed subprocess with resource limits and a wall-clock timeout; hostile or malformed files are quarantined and skipped instead of affecting the runtime.

Common Formats

Source rules & validation

• Paths must stay within the formation (relative paths only; no ..); keep sources under knowledge/.
• Large/recursive sources can be limited with max_files_per_source, file_limit, allowed_extensions, and max_file_size.
• Files are chunked and cached with MD5; unchanged files are skipped on restart, only deltas are re-embedded.

Reasoning RAG (tree retrieval)

Vector chunking works well for short passages but loses the structure of long documents. For large files, MUXI can index a document as a hierarchical tree and reason over it at query time instead of matching isolated chunks. A per-file gate (knowledge.reasoning_threshold) decides automatically; a per-source retrieval: override forces the mode:

• tree (Method A) - an LLM navigates the compressed tree and selects the relevant nodes. No embeddings, best structural understanding.
• tree-vector (Method B) - per-node chunk embeddings scored with the PageIndex formula. No per-query LLM calls, cheaper than Method A.
• hybrid - both run in parallel and a sufficiency evaluator decides whether to fetch more, trading cost for recall.

Every mode falls back to plain vector search on failure, so a tree never breaks a turn. See the knowledge reference for the knowledge.tree settings and per-source configuration.

Multimodal Support

MUXI understands images, not just extracts text from them:

knowledge/
├── architecture-diagram.png    ← Vision model analyzes
├── product-screenshot.jpg      ← UI elements recognized
├── chart.png                   ← Data extracted
└── handwritten-notes.jpg       ← OCR + interpretation

When a user asks about your architecture, the agent can reference the diagram directly:

User:  "How does data flow through our system?"
Agent: "Based on the architecture diagram, data flows from..."

Agent-Specific Knowledge

Different agents can access different knowledge bases:

# agents/support.afs
schema: "1.0.0"
id: support
name: Support Agent
description: Customer support

system_message: You're a customer support specialist...

knowledge:
  enabled: true
  sources:
    - path: knowledge/faq/
    - path: knowledge/troubleshooting/

# agents/sales.afs
schema: "1.0.0"
id: sales
name: Sales Agent
description: Sales advisor

system_message: You're a sales advisor....

knowledge:
  enabled: true
  sources:
    - path: knowledge/pricing/
    - path: knowledge/features/

Support agent knows troubleshooting; sales agent knows pricing. No overlap, no confusion.

How Indexing Works

Automatic at Startup

muxi up
# Indexing knowledge...
# ✓ 47 documents indexed
# ✓ 1,234 chunks created
# ✓ Embeddings cached

Incremental Updates

When files change:

1. MD5 hash detects modifications
2. Only changed files re-indexed
3. Cache preserved for unchanged content

Force full reindex:

muxi knowledge rebuild

Lazy Loading

First query triggers loading - startup stays fast even with large knowledge bases.

Why This Matters

The result: agents that know your domain, not generic chatbots that need everything explained.

Quick Setup

agents:
  - id: assistant
    knowledge:
      enabled: true
      sources:
        - path: knowledge/docs/
          description: Product documentation

Add your docs to knowledge/docs/, restart, done.

Learn More

• Configure knowledge - YAML syntax
• Add Knowledge Guide - Step-by-step tutorial