资讯动态

python脚本通过docling解析文档qrdant-edge索引使用RAG-MCP填充资料

发布时间:2026/10/13 1:20:17 来源:尧图企业网站定制
这是一个用来快速提取文档关键值的脚本可以用来批量提取一批相同类型但是不同型号的公共参数值可以避免人工大量重复性的搜索和来回翻阅工作例如快速提取esp32_datasheet_cn.pdf的以下字段| **型号** | **封装类型** | **工作温度(°C)** | **CPU 架构** | **核心数** | **最高频率(MHz)** | **SRAM(KB)** | **PSRAM(支持)** | **内置Flash(MB)** | **Wi-Fi 协议** | **蓝牙规格** | **典型GPIO数** | **USB接口** | **以太网MAC** | **CAN总线** | **工作电压(V)** | **深度睡眠功耗(μA)** | **安全启动/Flash加密** |可以得到| **型号** | **封装类型** | **工作温度(°C)** | **CPU 架构** | **核心数** | **最高频率(MHz)** | **SRAM(KB)** | **PSRAM(支持)** | **内置Flash(MB)** | **Wi-Fi 协议** | **蓝牙规格** | **典型GPIO数** | **USB接口** | **以太网MAC** | **CAN总线** | **工作电压(V)** | **深度睡眠功耗(μA)** | **安全启动/Flash加密** | |----------|--------------|------------------|--------------|------------|-------------------|--------------|-----------------|-------------------|----------------|--------------|----------------|-------------|---------------|-------------|-----------------|----------------------|--------------------------| | ESP32 | QFN 5*5 | -40 ~ 125 | Xtensa LX6 32-bit | 双核 | 240 | 520 | 支持外部最大 8 MB | | IEEE 802.11b/g/n | 蓝牙 v4.2 BR/EDR 和 Bluetooth LE | 34 | | 支持IEEE 802.3需外部 PHY | 支持TWAI兼容 ISO 11898-1 | 2.3 ~ 3.6 | 10 | 支持安全启动、flash 加密 |流程图依赖[project] name rag-filler version 0.1.0 description Add your description here readme README.md requires-python 3.11 dependencies [ docling2.121.0, langchain1.3.16, langchain-core1.6.0, langchain-mcp-adapters0.3.2, langchain-openai1.6.0, langchain-text-splitters1.1.2, mcp1.29.0, qdrant-client1.19.0, ]在命令行中接受三个参数其他参数通过环境变量来输入def parse_args(argv: list[str] | None None) - argparse.Namespace: parser argparse.ArgumentParser( progmain.py, add_helpFalse, descriptionFill a reference Markdown document from a PDF manual via RAG an LLM agent., ) parser.add_argument(--in, destpdf, requiredTrue, metavarPDF, help用来索引的PDF) parser.add_argument(--reference, requiredTrue, metavarMD, help需要填充的markdown文件) parser.add_argument(--out, requiredTrue, metavarMD, help输出的markdown文件) return parser.parse_args(argv) def load_config() - dict: required (LLM_MODEL, EMBEDDING_MODEL) missing [name for name in required if not os.environ.get(name)] if missing: raise ConfigError( 缺少必需的环境变量 , .join(missing) ) return { llm_api_base: os.environ.get(LLM_API_BASE, DEFAULT_LLM_BASE), llm_api_key: os.environ.get(LLM_API_KEY, not-needed), llm_model: os.environ[LLM_MODEL], embedding_api_base: os.environ.get(EMBEDDING_API_BASE, DEFAULT_EMBED_BASE), embedding_api_key: os.environ.get(EMBEDDING_API_KEY, not-needed), embedding_model: os.environ[EMBEDDING_MODEL], embedding_dimension: int(os.environ.get(EMBEDDING_DIMENSION, DEFAULT_EMBED_DIMENSION)), }脚本通过mcp来暴露查询方法而不是将文档塞入上下文用本地模型也可以跑这里使用FastMCP来创建mcp服务def make_search_server(qdrant: QdrantClient, embed: OpenAIEmbeddings) - FastMCP: server FastMCP(manual-search) server.tool() def search_manual(query: str, top_k: int 5) - list[dict]: 在已索引的手册中检索与 query 最相关的段落并返回含相关度分数。填写表格单元格之前请调用此工具从手册中查找事实依据。 k max(1, min(int(top_k), 20)) # PyCharm 解析到旧版 qdrant-client 签名会误报 query 参数类型;1.19 运行时接受 list[float] vector embed.embed_query(query) hits qdrant.query_points(collection_nameCOLLECTION, queryvector, limitk).points # type: ignore[arg-type] return [ { text: (hit.payload or {}).get(text, ), section: (hit.payload or {}).get(section, ), score: round(float(hit.score), 4), } for hit in hits ] return server然后通过 docling库来解析PDF注意docling这个库会下载一个小模型来解析PDF解析到的数据通过langchain提供的方法来分块分块后通过Qwen3-Embedding-0.6B来embedding存入qrdant-edge建议embedding模型在本地跑chat模型使用api这样可以节省大量token性能要求也不高async def run(args: argparse.Namespace, cfg: dict) - int: embed OpenAIEmbeddings( modelcfg[embedding_model], openai_api_basecfg[embedding_api_base], openai_api_keycfg[embedding_api_key], check_embedding_ctx_lengthFalse, show_progress_barFalse, ) reference Path(args.reference) ref_text reference.read_text(encodingutf-8) print(f正在索引 {args.pdf} ...) qdrant QdrantClient(:memory:) count index_pdf(args.pdf, embed, qdrant, cfg[embedding_dimension]) print(f已将 {count} 个文本块索引到内存向量数据库) llm ChatOpenAI( modelcfg[llm_model], base_urlcfg[llm_api_base], api_keycfg[llm_api_key], temperature0, ) server make_search_server(qdrant, embed) try: async with create_connected_server_and_client_session(server) as session: tools await load_mcp_tools(session) if not any(getattr(tool, name, None) search_manual for tool in tools): raise FatalError(向量检索 MCP 工具 search_manual 未绑定到 agent) agent create_agent(llm, tools, system_promptSYSTEM_PROMPT) print(正在填充参考文档 ...) response await agent.ainvoke( {messages: [HumanMessage(content( 下面是需要填充的参考文档。请依据手册填充其中留空或占位的内容 手册内容一律通过 search_manual 检索获取不要把 PDF 全文当作已知信息。 输出文档的最终形式由你判断但必须内容完整、可直接使用。\n\n 参考文档\nmarkdown\n ref_text \n ))]}, configRunnableConfig(recursion_limit120), ) filled extract_content(response[messages][-1].content) except FatalError: raise except Exception as exc: raise FatalError(fAPI 调用失败{format_api_error(exc)}) from exc write_output(Path(args.out), filled) print(f已写入 {args.out}) return 0实测例子这里RX100M3说明书.pdf是一个135页的pdf⚡wo ❯❯ $env:LLM_MODELdeepseek-v4-flash; $env:EMBEDDING_MODELQwen3-Embedding-0.6B; $env:LLM_API_BASEhttps://api.deepseek.com/v1; $env:LLM_API_KEYsk-44f7ab82118c4767b44bf89f807c418b; uv run python main.py --in res/RX100M3说明书.pdf --reference res/ref_cam.md --out result_cam.md 正在索引 res/RX100M3说明书.pdf ... [INFO] 2026-08-23 15:26:07,927 [RapidOCR] base.py:23: Using engine_name: torch [INFO] 2026-08-23 15:26:07,946 [RapidOCR] device_config.py:57: Using CPU device [INFO] 2026-08-23 15:26:07,958 [RapidOCR] download_file.py:60: File exists and is valid: E:\work\working\py\RagFiller\.venv\Lib\site-packages\rapidocr\models\PP-OCRv6_det_small.pth [INFO] 2026-08-23 15:26:07,958 [RapidOCR] main.py:50: Using E:\work\working\py\RagFiller\.venv\Lib\site-packages\rapidocr\models\PP-OCRv6_det_small.pth [INFO] 2026-08-23 15:26:08,114 [RapidOCR] base.py:23: Using engine_name: torch [INFO] 2026-08-23 15:26:08,114 [RapidOCR] device_config.py:57: Using CPU device [INFO] 2026-08-23 15:26:08,118 [RapidOCR] download_file.py:60: File exists and is valid: E:\work\working\py\RagFiller\.venv\Lib\site-packages\rapidocr\models\ch_ptocr_mobile_v2.0_cls_mobile.pth [INFO] 2026-08-23 15:26:08,118 [RapidOCR] main.py:50: Using E:\work\working\py\RagFiller\.venv\Lib\site-packages\rapidocr\models\ch_ptocr_mobile_v2.0_cls_mobile.pth [INFO] 2026-08-23 15:26:08,191 [RapidOCR] base.py:23: Using engine_name: torch [INFO] 2026-08-23 15:26:08,193 [RapidOCR] device_config.py:57: Using CPU device [INFO] 2026-08-23 15:26:08,208 [RapidOCR] download_file.py:60: File exists and is valid: E:\work\working\py\RagFiller\.venv\Lib\site-packages\rapidocr\models\PP-OCRv6_rec_small.pth [INFO] 2026-08-23 15:26:08,208 [RapidOCR] main.py:50: Using E:\work\working\py\RagFiller\.venv\Lib\site-packages\rapidocr\models\PP-OCRv6_rec_small.pth Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 770/770 [00:0000:00, 6633.26it/s] E:\work\working\py\RagFiller\.venv\Lib\site-packages\torch\nn\modules\conv.py:560: UserWarning: Using paddingsame with even kernel lengths and odd dilation may require a zero-padded copy of the input be created (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\aten\src\ATen\native\Convolution.cpp:1102.) return F.conv2d( RapidOCR returned empty result! RapidOCR returned empty result! RapidOCR returned empty result! [WARNING] 2026-08-23 15:28:36,526 [RapidOCR] main.py:132: The text detection result is empty RapidOCR returned empty result! RapidOCR returned empty result! RapidOCR returned empty result! RapidOCR returned empty result! [WARNING] 2026-08-23 15:29:07,453 [RapidOCR] main.py:132: The text detection result is empty RapidOCR returned empty result! 已将 375 个文本块索引到内存向量数据库 E:\work\working\py\RagFiller\.venv\Lib\site-packages\pydantic_settings\sources\utils.py:47: IncompleteFieldDefinitionWarning: Field lifespan has an incomplete definition: its annotation contains an unresolved forward reference, so settings sources may fail to correctly resolve its value. Call model_rebuild() on the model where the field is defined, once all the referenced types are defined. warnings.warn( [08/23/26 15:31:19] INFO Processing request of type ListToolsRequest server.py:733 正在填充参考文档 ... INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:21] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 [08/23/26 15:31:22] INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:24] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:26] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:28] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:30] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:32] INFO Processing request of type CallToolRequest server.py:733 [08/23/26 15:31:33] INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:35] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:38] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:41] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 [08/23/26 15:31:42] INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:44] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:47] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:49] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:51] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 [08/23/26 15:31:52] INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:54] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:56] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:31:59] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:32:05] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:32:07] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 [08/23/26 15:32:08] INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 [08/23/26 15:32:14] INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO Processing request of type CallToolRequest server.py:733 INFO HTTP Request: POST http://127.0.0.1:7838/v1/embeddings HTTP/1.1 200 OK _client.py:1085 INFO HTTP Request: POST https://api.deepseek.com/v1/chat/completions HTTP/1.1 200 OK _client.py:1923 已写入 result_cam.md可以得到| **型号** | **发布时间** | **传感器类型** | **有效像素(MP)** | **传感器尺寸** | **镜头焦距(等效)** | **最大光圈** | **光学变焦** | **对焦系统** | **连拍速度(fps)** | **视频规格** | **取景器** | **屏幕** | **机身尺寸(mm)** | **重量(g)** | **电池型号** | |----------|--------------|----------------|------------------|----------------|---------------------|--------------|--------------|--------------|-------------------|--------------|------------|----------|------------------|--------------|--------------| | Sony Cyber-shot DSC-RX100M3 | | Exmor R™ CMOS | 约2010万约20.1 MP | 13.2 mm × 8.8 mm1.0型 | f8.8 mm 至 25.7 mm24 mm 至 70 mm35 mm 胶片等效 | F1.8W至 F2.8T | 2.9倍 | 对焦模式单次AF / 连续AF / DMF / 手动对焦对焦区域广域 / 中间 / 自由点 / 锁定AF支持眼控AF、中央锁定AF、AF辅助照明 | | 文件格式XAVC S50p 50M、25p 50M、100p 50M100p 为 1280×720 高速记录可记录 100 fps 动态影像/ AVCHD50i 24M(FX)、50i 17M(FH)、50p 28M(PS)、25p 24M(FX)、25p 17M(FH)/ MP41440×1080 12M、VGA 3M兼容 1080 50i / 1080 50p | 电子取景器有机EL弹出式总点数 1 440 000 点取景率 100%倍率约 0.59 倍眼点约 20 mm | 7.5 cm3.0 型TFT 液晶总点数 1 228 800 点可向上旋转约 180 度 | 约 101.6 × 58.1 × 41.0符合 CIPA 标准 | 约 290 g包括电池 NP-BX1、Memory Stick PRO Duo | NP-BX1 |

读完文章,也想定制专属网站?

尧图设计师 24 小时内与您沟通定制方案

免费获取报价 →
↑