Skip to main content
Knowledge connectors let experts access documents on external platforms like Feishu, Yuque, and IMA. But “direct search” relies on each platform’s native search API — quality varies wildly, and some platforms can’t even return full text. The RAG pipeline upgrades connectors from “search channels” to “collection layers”: documents are periodically synced locally, chunked into vector-indexed segments, and retrieved through hybrid search. Precision improves significantly, and every statement in a response can be traced back to its source paragraph.

​
Two Modes, Side by Side

Each knowledge connection can independently choose its mode:

Direct Search (Default)

No indexing, no storage. Each query calls the platform API in real time. Best for small document sets with frequent updates and low precision requirements.

RAG Mode

When enabled, the system periodically syncs documents, chunks them, and builds vector indexes. Queries run local hybrid retrieval (vector + keyword), far surpassing native platform search quality.
Both modes can coexist across a single user’s connections. For example: Feishu with RAG enabled (many documents, precision matters), DingTalk staying on direct search (few documents, supplementary use only).

​
Enabling RAG Indexing

1

Connect a Platform

Bind platform credentials in Settings → Connectors (see Knowledge Connectors).
2

Enable RAG Toggle

Turn on “Enable RAG Indexing” on the connection card.
3

Initial Sync

Click “Sync Now” to trigger the first full sync. The card displays sync progress and status.
4

Start a Conversation

After sync completes, enable the Knowledge plugin and start chatting. Search automatically routes through the RAG pipeline, with source citations in responses.
After the initial sync, the system automatically runs incremental syncs every 24 hours (processing only new and changed documents). You can also trigger a manual sync at any time.

​
RAG Pipeline Architecture

The entire pipeline runs inside the Core service — zero additional containers, zero extra deployment configuration.

​
Sync Mechanism

​
Incremental Sync

The sync engine doesn’t reprocess everything each time. Each document’s SHA-256 content hash is computed on first sync. Subsequent syncs compare hashes — unchanged documents are skipped entirely, processing only additions and modifications.

​
Document Processing Flow

1

Fetch Document List

Retrieve full document metadata (title, ID, knowledge base) via platform API.
2

Change Detection

Compare against locally indexed documents’ content_hash to identify new and changed documents.
3

Fetch Full Text

Call platform API to get Markdown full text for changed documents.
4

Smart Chunking

MarkdownTextSplitter splits at heading boundaries (#/##/###) as preferred break points, with sentence/newline-aware secondary splits for long paragraphs. 100-character overlap preserves context. Each chunk is approximately 1,500 characters.
5

Vector Embedding

Each chunk is embedded via DashScope text-embedding-v3 into a 1024-dimensional vector (same model as the memory system).
6

Atomic Write

Within a database transaction: UPSERT document record → delete old chunks → insert new chunks. Guarantees data consistency.

​
Concurrency & Fault Tolerance

  • Single-instance worker: The sync worker uses PostgreSQL Advisory Locks to ensure single-instance execution across replicas — no duplicate syncs
  • Per-document resilience: A single document failure doesn’t block others; errors are recorded in the connection’s sync_error field
  • Embedding concurrency control: Batch embedding caps concurrency (5 parallel) to avoid exhausting DashScope API quotas

​
Hybrid Retrieval

RAG retrieval isn’t pure vector search — it’s vector + keyword dual-channel retrieval, fused via RRF (Reciprocal Rank Fusion).

Vector Search

Embeds the user query as a vector and retrieves the Top-30 semantically closest chunks via pgvector cosine similarity. Excels at synonyms and semantic relationships.

Keyword Search

Uses PostgreSQL full-text search (tsvector + tsquery) to retrieve Top-30 keyword-matching chunks. Excels at exact terminology and proper nouns.
Results are fused using the RRF formula: score(d)=∑r∈lists1k+rankr(d)\text{score}(d) = \sum_{r \in \text{lists}} \frac{1}{k + \text{rank}_r(d)} where k=60k=60 is a smoothing constant. Benefits of this fusion approach:
  • Vector search’s semantic matching and keyword search’s exact matching complement each other
  • Results ranking highly in both lists get boosted scores (consensus amplification)
  • Single-channel noise gets diluted by the other channel (noise suppression)
After deduplication, the Top-K results (default 10) are returned to the expert.

​
Source Attribution

RAG results integrate with the citation pipeline. Citation markers [1] [2] in expert responses link to source cards at the bottom of the conversation, each showing:
  • Original document title and platform source
  • Matching paragraph preview
  • Knowledge base name
  • Original document link (if available)

​
Sync Status

The connection card displays real-time sync status: A sync failure doesn’t affect existing indexes — previously indexed documents remain fully searchable. The next sync will automatically retry.

​
Platform Support

​
Billing

  • Sync process: Document embedding consumes DashScope embedding API quota (internal platform cost, not charged to user credits)
  • Retrieval process: Query embedding is the same — no additional user credit charge
  • Conversation cost: Retrieved chunk content is injected into conversation context, billed by model tokens — same as direct search mode, see in-product display for details
The primary cost of RAG mode is the embedding calls during the initial full sync. Incremental syncs only process changed documents, keeping ongoing costs minimal.