> ## Documentation Index
> Fetch the complete documentation index at: https://docs.profy.cn/llms.txt
> Use this file to discover all available pages before exploring further.

# Knowledge RAG Pipeline

> Connector ingestion → chunking & embedding → hybrid retrieval: how knowledge search upgrades from direct-search to RAG

Knowledge connectors let experts access documents on external platforms like Feishu, Yuque, and IMA. But "direct search" relies on each platform's native search API — quality varies wildly, and some platforms can't even return full text.

The RAG pipeline upgrades connectors from "search channels" to "collection layers": documents are periodically synced locally, chunked into vector-indexed segments, and retrieved through hybrid search. Precision improves significantly, and every statement in a response can be traced back to its source paragraph.

## Two Modes, Side by Side

Each knowledge connection can independently choose its mode:

<CardGroup cols={2}>
  <Card title="Direct Search (Default)" icon="magnifying-glass">
    No indexing, no storage. Each query calls the platform API in real time. Best for small document sets with frequent updates and low precision requirements.
  </Card>

  <Card title="RAG Mode" icon="database">
    When enabled, the system periodically syncs documents, chunks them, and builds vector indexes. Queries run local hybrid retrieval (vector + keyword), far surpassing native platform search quality.
  </Card>
</CardGroup>

Both modes can coexist across a single user's connections. For example: Feishu with RAG enabled (many documents, precision matters), DingTalk staying on direct search (few documents, supplementary use only).

## Enabling RAG Indexing

<Steps>
  <Step title="Connect a Platform">
    Bind platform credentials in **Settings → Connectors** (see [Knowledge Connectors](/en/documentation/capabilities/knowledge-connectors)).
  </Step>

  <Step title="Enable RAG Toggle">
    Turn on "Enable RAG Indexing" on the connection card.
  </Step>

  <Step title="Initial Sync">
    Click "Sync Now" to trigger the first full sync. The card displays sync progress and status.
  </Step>

  <Step title="Start a Conversation">
    After sync completes, enable the Knowledge plugin and start chatting. Search automatically routes through the RAG pipeline, with source citations in responses.
  </Step>
</Steps>

After the initial sync, the system automatically runs incremental syncs every 24 hours (processing only new and changed documents). You can also trigger a manual sync at any time.

## RAG Pipeline Architecture

```mermaid theme={null}
flowchart LR
  subgraph Collection
    A[Platform API<br/>Feishu/Yuque/IMA] -->|fetch docs| B[Sync Engine]
  end

  subgraph Processing
    B -->|Markdown| C[Text Chunking<br/>heading-aware split]
    C -->|chunk × N| D[Vector Embedding<br/>DashScope 1024-dim]
  end

  subgraph Storage
    D -->|write| E[(PostgreSQL<br/>pgvector)]
  end

  subgraph Retrieval
    F[User Query] -->|embed query| G[Vector Search<br/>cosine similarity]
    F -->|raw text| H[Keyword Search<br/>tsvector]
    G --> I[RRF Fusion]
    H --> I
    I -->|Top-K chunks| J[Expert Response<br/>with citations]
  end

  E --> G
  E --> H
```

The entire pipeline runs inside the Core service — zero additional containers, zero extra deployment configuration.

## Sync Mechanism

### Incremental Sync

The sync engine doesn't reprocess everything each time. Each document's SHA-256 content hash is computed on first sync. Subsequent syncs compare hashes — unchanged documents are skipped entirely, processing only additions and modifications.

### Document Processing Flow

<Steps>
  <Step title="Fetch Document List">
    Retrieve full document metadata (title, ID, knowledge base) via platform API.
  </Step>

  <Step title="Change Detection">
    Compare against locally indexed documents' content\_hash to identify new and changed documents.
  </Step>

  <Step title="Fetch Full Text">
    Call platform API to get Markdown full text for changed documents.
  </Step>

  <Step title="Smart Chunking">
    MarkdownTextSplitter splits at heading boundaries (`#/##/###`) as preferred break points, with sentence/newline-aware secondary splits for long paragraphs. 100-character overlap preserves context. Each chunk is approximately 1,500 characters.
  </Step>

  <Step title="Vector Embedding">
    Each chunk is embedded via DashScope text-embedding-v3 into a 1024-dimensional vector (same model as the memory system).
  </Step>

  <Step title="Atomic Write">
    Within a database transaction: UPSERT document record → delete old chunks → insert new chunks. Guarantees data consistency.
  </Step>
</Steps>

### Concurrency & Fault Tolerance

* **Single-instance worker**: The sync worker uses PostgreSQL Advisory Locks to ensure single-instance execution across replicas — no duplicate syncs
* **Per-document resilience**: A single document failure doesn't block others; errors are recorded in the connection's sync\_error field
* **Embedding concurrency control**: Batch embedding caps concurrency (5 parallel) to avoid exhausting DashScope API quotas

## Hybrid Retrieval

RAG retrieval isn't pure vector search — it's **vector + keyword dual-channel** retrieval, fused via RRF (Reciprocal Rank Fusion).

<CardGroup cols={2}>
  <Card title="Vector Search" icon="brain">
    Embeds the user query as a vector and retrieves the Top-30 semantically closest chunks via pgvector cosine similarity. Excels at synonyms and semantic relationships.
  </Card>

  <Card title="Keyword Search" icon="font">
    Uses PostgreSQL full-text search (tsvector + tsquery) to retrieve Top-30 keyword-matching chunks. Excels at exact terminology and proper nouns.
  </Card>
</CardGroup>

Results are fused using the RRF formula:

$\text{score}(d) = \sum_{r \in \text{lists}} \frac{1}{k + \text{rank}_r(d)}$

where $k=60$ is a smoothing constant. Benefits of this fusion approach:

* Vector search's semantic matching and keyword search's exact matching **complement** each other
* Results ranking highly in both lists get boosted scores (**consensus amplification**)
* Single-channel noise gets diluted by the other channel (**noise suppression**)

After deduplication, the Top-K results (default 10) are returned to the expert.

## Source Attribution

RAG results integrate with the [citation pipeline](/en/documentation/capabilities/knowledge-connectors). Citation markers `[1]` `[2]` in expert responses link to source cards at the bottom of the conversation, each showing:

* Original document title and platform source
* Matching paragraph preview
* Knowledge base name
* Original document link (if available)

## Sync Status

The connection card displays real-time sync status:

| Status          | Meaning                                        |
| --------------- | ---------------------------------------------- |
| **Ready**       | Last sync succeeded, waiting for next schedule |
| **Syncing**     | Currently fetching, chunking, embedding        |
| **Sync Failed** | Last sync errored, showing error reason        |

A sync failure doesn't affect existing indexes — previously indexed documents remain fully searchable. The next sync will automatically retry.

## Platform Support

| Platform | Collection Method                  | Content Format  | Notes                                                        |
| -------- | ---------------------------------- | --------------- | ------------------------------------------------------------ |
| Feishu   | Wiki Spaces → Nodes → Raw Content  | Markdown        | Auto-refreshes tenant\_access\_token                         |
| Yuque    | Repos → Docs → Raw Body            | Markdown        | Enterprise can configure custom host                         |
| IMA      | KBs → Knowledge List → Doc Content | Structured text | Only note-type items (media\_type=11)                        |
| DingTalk | —                                  | —               | RAG not supported (no full-text API), stays on direct search |

## Billing

* **Sync process**: Document embedding consumes DashScope embedding API quota (internal platform cost, not charged to user credits)
* **Retrieval process**: Query embedding is the same — no additional user credit charge
* **Conversation cost**: Retrieved chunk content is injected into conversation context, billed by model tokens — same as direct search mode, see in-product display for details

<Note>
  The primary cost of RAG mode is the embedding calls during the initial full sync. Incremental syncs only process changed documents, keeping ongoing costs minimal.
</Note>