From 1b3374129ee24bd159413fddeedff9d189b31dc3752480f0cfe12c8e3a0dbc57 Mon Sep 17 00:00:00 2001 From: Claude Code Date: Thu, 25 Jun 2026 06:50:25 +0000 Subject: [PATCH] Add saiop-rag-ingestion skill: network gotchas and verification method --- .../saiop-rag-ingestion/SKILL.md | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 skills/saiop-infrastructure/saiop-rag-ingestion/SKILL.md diff --git a/skills/saiop-infrastructure/saiop-rag-ingestion/SKILL.md b/skills/saiop-infrastructure/saiop-rag-ingestion/SKILL.md new file mode 100644 index 0000000..236a91b --- /dev/null +++ b/skills/saiop-infrastructure/saiop-rag-ingestion/SKILL.md @@ -0,0 +1,48 @@ +--- +name: saiop-rag-ingestion +description: "Run the SAIOP document ingestion pipeline (scripts/ingestion/ingest.py) to load docs into Qdrant — network gotchas and how to verify it actually worked." +version: 1.0.0 +author: Claude Code (SAIOP ops session) +license: MIT +platforms: [linux] +prerequisites: + env_vars: [QDRANT_API_KEY] + commands: [docker] +metadata: + hermes: + tags: [SAIOP, RAG, Qdrant, ingestion, Ollama] +--- + +# RAG ingestion into Qdrant (SAIOP) + +## When to use + +Loading new SOPs, runbooks, or infrastructure docs into Qdrant so they're +retrievable via RAG (Deployment Guide §6.2). + +## Key facts + +- The reference bundle's `scripts/ingestion/ingest.py` has a real bug: it + calls `sys.exit(1)` but never `import sys`. Add `import sys` before + running it, or it crashes if invoked with neither `--file` nor + `--directory`. +- It embeds via **Ollama directly** (`OLLAMA_URL/api/embeddings`, + `nomic-embed-text` model) — NOT through the AI Router. Point `OLLAMA_URL` + at `http://host.docker.internal:11434` if running the script in a + container (Ollama is a host systemd service, not a container). +- **UFW only allows port 11434 from specific docker subnets** + (`172.17.0.0/16`, `172.19.0.0/16` — the `ai-internal` network). A + container on a *different* network (e.g. `knowledge`, where Qdrant lives) + will silently time out reaching Ollama via `host.docker.internal`, even + though the hostname resolves fine. Fix: attach the ingestion container to + **both** `ai-internal` (for Ollama) and `knowledge` (for Qdrant) — + `docker run -d --network ai-internal ...` then + `docker network connect knowledge `. +- The five collections are `atu_infrastructure`, `atu_operations`, + `atu_incidents`, `atu_governance`, `atu_ai` — pick by content type (the + script's `doc_type_map` already does chunk-size tuning per collection). +- **Don't trust "uploaded N chunks" alone as proof it worked.** Verify with + a real semantic search: embed a realistic question via the same Ollama + endpoint, POST it to `/collections//points/search`, and check the + returned `chunk_text` is actually relevant (score > ~0.7 and on-topic) — + not just that points exist.