Scraping Pipeline

DocShark processes documentation through a six-phase pipeline:

URL → Discover → Fetch → Extract → Convert → Chunk → Index

1. Discovery

DocShark uses a cascading strategy to find all pages in a documentation site:

• Sitemap.xml (preferred) — Parse /sitemap.xml for page URLs
• Navigation-aware extraction — Extract URLs from the site’s navigation structure
• Link crawl (fallback) — BFS from root URL, following internal links
• robots.txt — Respect disallowed paths via robots-parser

2. Fetching

• Default: Native fetch with proper User-Agent header
• Incremental: Sends If-None-Match / If-Modified-Since to skip unchanged pages (HTTP 304)
• Rate limiting: Configurable delay (default 500ms between requests)
• Retry: Exponential backoff on failure (3 attempts)
• Timeout: 30 seconds per request
• JS rendering: Auto-detects SPA sites and upgrades to puppeteer-core when needed

3. Content Extraction

Uses @mozilla/readability with linkedom (lighter than jsdom):

• Strips navigation, sidebars, footers, and ads
• Extracts article title and main content
• Pre-processes the DOM to rescue complex elements like <pre>, <table>, and <details> before Readability processing

4. HTML → Markdown

turndown configured for documentation-friendly output:

• ATX-style headings (# h1, ## h2)
• Fenced code blocks with language tags
• Preserved language-* classes on code blocks
• Table support via GFM plugin
• <details> and <summary> elements preserved

5. Chunking

Recursive heading-based splitting preserves document structure:

1. Split on # h1 → major sections
2. Within h1, split on ## h2 → subsections
3. Within h2, split on ### h3 → fine-grained sections
4. If section exceeds max tokens → split on paragraphs
5. Never split mid-code-block (atomic units)
6. Heading hierarchy preserved as breadcrumbs: "Getting Started > Installation"
7. Target: 500–1,500 tokens per chunk
8. Minimum: 50 tokens (skip tiny fragments)

6. Indexing

• Insert chunks into SQLite chunks table
• FTS5 index synced via database triggers
• Library stats updated (page count, chunk count)

Dependencies

Core footprint: ~375KB — No LangChain, no heavy browser runtimes by default.