Extraction Pipeline
FluidRAG can extract Slack conversations into your knowledge graph on a schedule. This makes historical Slack discussions available for RAG queries alongside your other ingested documents.
How It Works
Section titled “How It Works”The pipeline runs in three stages:
-
Thread Extraction — FluidRAG fetches threads from selected channels, identifies new and updated threads using delta logic (new parent messages or threads with new replies), and saves each thread as a markdown file with metadata (channel, timestamps, author, reply count).
-
LLM Preprocessing — Each thread is processed by an LLM that generates a structured document with a topic title, concise summary, and detailed information extracted from the conversation. Non-viable threads (e.g., bot messages, empty threads) are filtered out.
-
Knowledge Graph Ingestion — Processed documents are ingested into FalkorDB with entity extraction, embedding generation, and relationship creation, making them searchable via graph and vector search.
Default Schedule
Section titled “Default Schedule”Extraction is manual-trigger-only by default — no schedule is set automatically. You must configure a schedule in the Admin UI (e.g., every 8 hours) or trigger it manually.
Configuring Extraction via Admin UI
Section titled “Configuring Extraction via Admin UI”Enable Slack Extraction
Section titled “Enable Slack Extraction”- Open the Admin UI at
http://localhost/config-ui - Navigate to the Connectors page
- Find the Slack connector card
- Toggle the switch on the card to Enabled
- Click Configure (gear icon) to open the connector settings modal
Select Channels
Section titled “Select Channels”- In the connector settings modal, switch to the Source Config tab
- Click Discover Channels to see all channels the bot has access to
- Select the channels you want to extract from
- Save your selection
Configure Schedule
Section titled “Configure Schedule”| Setting | Default | Description |
|---|---|---|
| Schedule | (empty) | Cron expression (e.g., 0 */8 * * * for every 8 hours) |
| Timeout | 3600 seconds | Maximum extraction runtime |
| Max Retries | 3 | Retry attempts on transient errors |
Manual Actions
Section titled “Manual Actions”| Action | How |
|---|---|
| Trigger extraction now | Connectors → Slack card → Sync Now |
| Run a specific stage | Connectors → Slack card → Sync Now dropdown → Extract Only / Preprocess Only / Ingest Only |
| Reset sync state | Connectors → Slack card → Configure → Reset Sync State — clears last sync timestamp so the next run extracts everything |
| Test connection | Connectors → Slack card → Test — verifies the bot token is valid |
API Endpoints
Section titled “API Endpoints”All endpoints require admin authentication.
| Endpoint | Method | Description |
|---|---|---|
/api/connectors/slack | GET | Get Slack connector status and configuration |
/api/connectors/slack | PATCH | Enable/disable Slack connector |
/api/connectors/slack/live-query | PATCH | Toggle live query routing |
/api/connectors/slack/schedule | PATCH | Update extraction schedule |
/api/connectors/slack/schedule | DELETE | Clear extraction schedule |
/api/connectors/slack/test-connection | POST | Verify bot token is valid |
/api/connectors/slack/extraction/trigger | POST | Trigger extraction (supports stages) |
/api/connectors/slack/extraction/reset | POST | Reset sync state for full resync |
/api/connectors/slack/discovery | GET | List all bot-accessible channels |
/api/connectors/slack/source-config | GET / PATCH | Read or update selected channels |
/api/extraction/status | GET | Get extraction status for all sources |