graphify 增量更新与纯聚类实操--update/--cluster-only的完整流水线与易错点【免费下载链接】graphifyTurn any codebase, with its docs, SQL schemas, configs, and PDFs, into a queryable knowledge graph. A /graphify skill for Claude Code, Cursor, Codex, and Gemini CLI: local deterministic AST parsing, every edge explained, no vector store.项目地址: https://gitcode.com/GitHub_Trending/graph/graphifygraphify 的更新与维护并不总是需要全量重建。tools/skillgen/fragments/references/shared/update.md定义了 Agent 在用户传入--update增量重抽取或--cluster-only仅重聚类时应遵循的操作手册runbook只在自上次运行后新增/修改过的文件上工作从而节省 Token 与时间。本文以该 runbook 为骨架结合 graphify/detect.py、graphify/build.py、graphify/analyze.py 与 graphify/cli.py 的源码实现逐段还原增量合并、删除剪枝、manifest 打点与仅聚类四条关键链路并解释每段脚本背后的 issue 编号所对应的工程权衡。一、该 runbook 的定位什么时候才需要读它片段文件自述得非常清楚首次全量构建永远不会读取这份文件。它只在用户显式传入以下两个模式之一时被加载--update增量重抽取incremental re-extraction--cluster-only只对已有图重新跑社区聚类。它是 skill 生成体系tools/skillgen/中“共享参考片段”的一部分会被聚合进各平台agents、codex、claude、opencode 等的 skill 文档里例如 graphify/skills/agents/references/update.md。注意该文档中频繁出现的“Step 3A/3B/3C/4–8/9”指代主流程 runbook 的阶段编号——本 runbook 负责的是如何把增量状态注入那些既有阶段而不是重开一条新流水线。三个全局约定先记住graphify-out/.graphify_python由构建过程写入的 Python 解释器路径文件。所有脚本都通过$(cat graphify-out/.graphify_python) -c ...运行保证使用的解释器版本与上次构建一致。INPUT_PATH与IS_DIRECTED代码中的占位符——前者替换为实际扫描根路径后者在用户给了--directed时为True否则为False。中间文件都是graphify-out/下带.前缀的隐藏文件.graphify_detect.json、.graphify_incremental.json、.graphify_extract.json、.graphify_old.json它们只在本次更新会话内存活主流程的 Step 9 清理会删除它们。二、--update增量重抽取完整流水线2.1 第 1 步用detect_incremental找出“真正变了”的文件runbook 首先执行差异检测并把结果落到磁盘$(cat graphify-out/.graphify_python) -c import sys, json from graphify.detect import detect_incremental, save_manifest from pathlib import Path result detect_incremental(Path(INPUT_PATH)) new_total result.get(new_total, 0) print(json.dumps(result, indent2, ensure_asciiFalse)) Path(graphify-out/.graphify_incremental.json).write_text(json.dumps(result, ensure_asciiFalse), encoding\utf-8\) deleted list(result.get(deleted_files, [])) if new_total 0 and not deleted: print(No files changed since last run. Nothing to update.) raise SystemExit(0) if deleted: print(f{len(deleted)} deleted file(s) to prune.) if new_total 0: print(f{new_total} new/changed file(s) to re-extract.) 源码层面detect_incremental定义于 graphify/detect.py关键设计是两个 hash 字段区分两种变更语义kindast只有当文件的ast_hash缺失或内容变化才判为“已变更”用于graphify updatekindsemantic默认按semantic_hash判定用于graphify extract——这样被update纯 AST动过的文件会在下次extract时被重新做语义抽取。其判定策略是一条分层的“快/慢路径”快速路径下 mtime 未变且 hash 一致即视为未变除 stat 外零磁盘 IO慢速路径下 mtime 变了则回退到 MD5 内容校验再决定是否重抽。它同时兼容旧版 manifest 中存储纯 float mtime 或{mtime, hash}的遗留 schemagraphify/detect.py。值得留意的一个边界mtime 与内容 hash 都一致也不代表安全——若 mtime 与写入发生在同一文件系统时间片内同长度编辑可能不触发 mtime 移动因此_mtime_may_hide_a_rewrite会针对这个窄窗口额外做一次 hash 校验防止图里一直服务旧内容。一个与用户直觉相反的设计是删除与排除被严格区分deleted_files只收录“磁盘上已不存在”的 manifest 行其缓存节点是真正的幽灵数据而文件还在磁盘、只是被.gitignore/.graphifyignore/--exclude排除的会进入excluded_files绝不能当作删除上报graphify/detect.py。这对应 runbook 末尾给save_manifest传scan_corpus的原因。2.2 第 2 步填充.graphify_detect.jsondetect_incremental的结果只反映“变化集”但主流程的 Step 3A–6 是无条件读取.graphify_detect.json的。因此必须把增量状态翻译成它们认识的格式$(cat graphify-out/.graphify_python) -c import json from pathlib import Path r json.loads(Path(graphify-out/.graphify_incremental.json).read_text(encoding\utf-8\)) Path(graphify-out/.graphify_detect.json).write_text(json.dumps({ files: r.get(new_files, {}), all_files: r.get(files, {}), total_files: r.get(new_total, 0), total_words: r.get(total_words, 0), skipped_sensitive: r.get(skipped_sensitive, []), needs_graph: True, }, ensure_asciiFalse), encoding\utf-8\) 这里有一个必须理解的双字段设计files承载变更子集驱动 Step 3AAST 抽取与 Step 3B0cache 命中检查只处理变化文件all_files承载完整语料供任何需要“全语料上下文”的阶段使用例如判断某个符号是否真的没有外部引用、超边是否需要跨文件合并。2.3 第 3 步按“变更内容”分三条支路支路 A——纯代码变更跳过语义抽取零 LLM先探测本次变更是否全部是代码文件$(cat graphify-out/.graphify_python) -c import json from pathlib import Path result json.loads(open(graphify-out/.graphify_incremental.json, encodingutf-8).read()) if Path(graphify-out/.graphify_incremental.json).exists() else {} code_exts {.py,.ts,.js,.go,.rs,.java,.cpp,.c,.rb,.swift,.kt,.cs,.scala,.php,.cc,.cxx,.hpp,.h,.kts,.lua,.toc,.f,.F,.f90,.F90,.f95,.F95,.f03,.F03,.f08,.F08} new_files result.get(new_files, {}) all_changed [f for files in new_files.values() for f in files] code_only all(Path(f).suffix.lower() in code_exts for f in all_changed) print(code_only:, code_only) 若code_only为True打印[graphify update] Code-only changes detected - skipping semantic extraction (no LLM needed)只对变化文件跑 Step 3AAST整体跳过 Step 3B不派发任何语义子代理随后直接进入 merge 与 Step 4–8。说明runbook 里这份扩展名集合是一个保守子集实际判定代码文件的权威集合是 graphify/detect.py 的CODE_EXTENSIONS约上百种覆盖 ts/tsx/jsx/vue/svelte/zig/ps1/sql/fortran 等文档/论文/图片/Office/视频则分别由DOC_EXTENSIONS、PAPER_EXTENSIONS、IMAGE_EXTENSIONS、OFFICE_EXTENSIONS、VIDEO_EXTENSIONS定义。扩展名只用于“要不要走语义抽取”的路由最终“是不是可建图源码”由classify_file与_is_graphable_source决定graphify/detect.py。支路 B——非纯代码变更先转写视频再走完整语义管线若任何变更文件属于new_files[video]必须先执行 graphify/skills/agents/references/transcribe.md即主流程的 Step 2.5对它们做转写然后重写.graphify_detect.json把转写产物路径放进files[document]、删掉files[video]。否则裸.mp4/.mp3路径会被当作不可读媒体直接喂给语义子代理runbook 引用 issue #1392。完成后再按正常流程跑完整的 Step 3A–3C。支路 C——只有删除构造空 extraction 供 merge 剪枝if [ ! -f graphify-out/.graphify_extract.json ]; then echo [graphify update] Only deletions -- creating empty extraction for merge. $(cat graphify-out/.graphify_python) -c import json from pathlib import Path Path(graphify-out/.graphify_extract.json).write_text(json.dumps({nodes:[],edges:[],hyperedges:[],input_tokens:0,output_tokens:0}), encodingutf-8) fi此时没有新文件需要抽取但 merge 步骤需要一份“本轮 extraction”作为输入空 extraction 让build_merge拿到零新增、只执行删除侧的工作。2.4 第 4 步build_merge——合并语义与三个“不要踩的坑”这是整个 runbook 技术含量最高的部分$(cat graphify-out/.graphify_python) -c import json from pathlib import Path from graphify.build import build_merge from graphify.detect import save_manifest # Load new extraction and incremental state new_extraction json.loads(Path(graphify-out/.graphify_extract.json).read_text(encoding\utf-8\)) incremental json.loads(Path(graphify-out/.graphify_incremental.json).read_text(encoding\utf-8\)) deleted list(incremental.get(deleted_files, [])) # prune_sources is ONLY for genuinely DELETED files. Changed/re-extracted files are # handled by build_merges replace-on-re-extract (#1344): every source_file in # new_chunks is dropped from the base before merge, so old/stale nodes dont survive. # Do NOT add changed here: with root passed, prune_set relativizes to the same base # as the freshly merged nodes and would DELETE the re-extracted content (#1178 is moot # now that replace — not the dedup pass — reconciles changed files). prune list(deleted) or None # Use build_merge() — reads graph.json directly without NetworkX round-trip # so edge direction (calls, implements, imports) is always preserved (#801). # Pass root so prune_sources (absolute paths from detect_incremental) are # relativized to match the graphs relative source_file values; without it # nothing is pruned and stale nodes accumulate on every update (#1361). # directedIS_DIRECTED: replace IS_DIRECTED with True if --directed was given, else # False. Without it a --directed --update silently rebuilds undirected and collapses # reciprocal A-B edges (#1392). G build_merge( [new_extraction], graph_pathgraphify-out/graph.json, prune_sourcesprune, rootINPUT_PATH, directedIS_DIRECTED, ) print(f[graphify update] Merged: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges) 对照 graphify/build.py 的build_merge签名new_chunks, graph_path, prune_sources, *, directed, dedup, dedup_llm_backend, root可以印证 runbook 注释里的四件事prune_sources只能放真正删除的文件。变更/重抽的文件由build_merge的 replace-on-re-extract 机制接管源码注释 #2333/#2336new_chunks中出现的每个source_file其旧节点/边会按 tierAST vs 语义见_is_ast_tier被丢弃后重写因此重抽文件不会累积陈旧节点而prune_sources是不分 tier 的删除。若把 changed 文件误放进 prune 集合由于root会将其相对化到与合并后节点相同的基准上等于把刚重抽出来的内容又删掉——这正是历史上 #1178 的教训现在的版本中 replace 而非 dedup 负责对账变更文件所以该问题已经不存在。直接读graph.json合并不做 NetworkX 往返序列化从而保证边的方向calls / implements / imports在合并后不被破坏#801。root必须传detect_incremental返回的是绝对路径而图中节点的source_file是相对根路径存储的build_merge会把 prune 源相对化到同一基准才能命中源码注释 #932/#1571/#1361。漏传会导致删除永不生效、陈旧节点每次更新都残留。directedIS_DIRECTED必须显式源码中directedNone时会继承图上已有的directed标志#2342但 runbook 出于确定性要求显式传值——否则一个--directed --update会悄悄把有向图重建为无向图把互指的 A↔B 边折叠成一条#1392。2.5 第 5 步把合并结果写回.graphify_extract.jsonmerge 返回的nx.Graph不会落盘build_merge只读入graph_path写盘由调用方负责。因此 runbook 手动把合并后的全量图序列化回.graphify_extract.json供 Step 4分析/聚类读取# Write merged result back to .graphify_extract.json so Step 4 sees the full graph merged_out { nodes: [{id: n, **d} for n, d in G.nodes(dataTrue)], edges: [ # Explicit source/target last so they win over any stale attrs in d. {**{k: val for k, val in d.items() if k not in (_src, _tgt, source, target)}, source: d.get(_src, u), target: d.get(_tgt, v)} for u, v, d in G.edges(dataTrue) ], # G.graph[\hyperedges\] holds hyperedges from both existing graph.json # and new_extraction (build_merge combines them). Falling back to # new_extraction only would silently drop prior-run hyperedges (#801). hyperedges: list(G.graph.get(hyperedges, [])), input_tokens: new_extraction.get(input_tokens, 0), output_tokens: new_extraction.get(output_tokens, 0), } Path(graphify-out/.graphify_extract.json).write_text(json.dumps(merged_out, ensure_asciiFalse), encoding\utf-8\) print(f[graphify update] Merged extraction written ({len(merged_out[\nodes\])} nodes, {len(merged_out[\edges\])} edges))三个细节值得展开边重建时source/target放在 dict 最后保证它们压过d中任何陈旧的同名属性NetworkX 内部以_src/_tgt保存的端点信息成为权威来源hyperedges必须读G.graph[hyperedges]build_merge会把已有graph.json与新 extraction 的超边合并存于图属性中若退回new_extraction单侧就会静默丢掉上一轮运行的超边#801每次 merge 都会重新打印节点/边计数用于 Agent 自我核对合并是否按预期发生。2.6 第 6 步save_manifest——防止幽灵节点与内容丢失的三个收尾更新成功不算完成manifest 必须被正确推进到“今天”的状态否则下一次--update会以旧基线做 diff产生假变更/幽灵节点报告# Save manifest so next --update diffs against todays state, not the # prior runs baseline (prevents ghost-node reports on subsequent updates). # root matches the build_merge call above so the manifest keys stay relative to # the scan root — portable across clones/machines, so --update keeps matching # cached files instead of missing every one after a move (#1417). # # Only stamp semantic files (docs/papers/images) that ACTUALLY produced output # THIS run (new_extraction is this runs fresh extraction, read above before the # merge overwrote the file): a changed doc whose chunk failed must stay unstamped # so the next --update re-queues it, otherwise it is marked done and its content # is lost forever (#2015). Mirrors the library extract path # (cli._stamped_manifest_files clear_semantic scan_corpus). from graphify.cli import _stamped_manifest_files _manifest_files _stamped_manifest_files(incremental[files], new_extraction, Path(INPUT_PATH)) # Changed semantic files dispatched this run but NOT stamped had their chunk fail # or be omitted; clear any stale semantic_hash so they are re-queued (#1948). _sem_types (document, paper, image) _dispatched {f for t, fl in incremental.get(new_files, {}).items() if t in _sem_types for f in fl} _stamped {f for fl in _manifest_files.values() for f in fl} _cleared _dispatched - _stamped # scan_corpus the RAW full corpus so in-root files newly excluded since last run # are dropped rather than masquerading as deletions; untouched rows preserved (#1908). _scan {f for fl in incremental[files].values() for f in fl} save_manifest(_manifest_files, rootINPUT_PATH, scan_corpus_scan, clear_semantic_cleared or None) print([graphify update] Manifest saved.)源码侧 graphify/cli.py 的_stamped_manifest_files解释了“只给真正产出结果的语义文件打点”的细则#933语义文件document/paper/image只有在sem_result的nodes/hyperedges集合里出现了对应source_file才打点chunk 失败、被截断partial_source_files或只产出边没有节点/超边#2927的文件一律不打点从而在下次--update时被重新排队超边也计入有效产出#1920因为某些文档的 chunk 唯一结果可能就是一条连接 3 个以上节点的超边命中判定会先按root解析路径#1897/#1890避免“fresh extraction 存相对路径、detect 给绝对路径”导致的永不匹配。save_manifest本身定义于 graphify/detect.py支持三种kindastupdate 用只打ast_hash保留内容未变文件的semantic_hash、semanticextract 用、both全量管线。runbook 用到的两个扩展参数都有明确语义scan_corpus#1908传完整的 raw detect 语料而非 stamp 过滤后的子集。这样在根目录内、仍活在磁盘上但已被新增 ignore 规则排除的文件会被丢弃而不是被detect_incremental当成“删除”未触及的行会被保留。注意只保存子集文件如 hook、changed_paths 场景时该参数必须为None。clear_semantic#1948把“本轮派发了但没产出 stamp”的语义文件的旧semantic_hash清空防止 seed 循环把上一轮的 hash 原样复制过来、掩盖本次失败副作用是让detect_incremental(kindsemantic)把它们再次排队。而root的一致性merge 与 manifest 都传INPUT_PATH保证 manifest 的键始终是相对扫描根的正斜杠路径跨机器、跨克隆位置可移植#777/#1417。2.7 第 7 步跑 Step 4–8并用graph_diff汇报差异合并完成后对合并后的图正常执行主流程 Step 4–8。在 Step 4 之后展示图差异的脚本是# After Step 4, show the graph diff $(cat graphify-out/.graphify_python) -c import json from graphify.analyze import graph_diff from graphify.build import build_from_json from networkx.readwrite import json_graph import networkx as nx from pathlib import Path # Load old graph (before update) from backup written before merge old_data json.loads(Path(graphify-out/.graphify_old.json).read_text(encoding\utf-8\)) if Path(graphify-out/.graphify_old.json).exists() else None new_extract json.loads(Path(graphify-out/.graphify_extract.json).read_text(encoding\utf-8\)) G_new build_from_json(new_extract, directedIS_DIRECTED) if old_data: G_old json_graph.node_link_graph(old_data, edgeslinks) diff graph_diff(G_old, G_new) print(diff[summary]) if diff[new_nodes]: print(New nodes:, , .join(n[label] for n in diff[new_nodes][:5])) if diff[new_edges]: print(New edges:, len(diff[new_edges])) 配套的备份动作有两步必须在 merge 之前做、在收尾后做# Before the merge step, save the old graph cp graphify-out/graph.json graphify-out/.graphify_old.json # Clean up after rm -f graphify-out/.graphify_old.jsongraph_diff定义于 graphify/analyze.py返回结构是{new_nodes, removed_nodes, new_edges, removed_edges, summary}其中summary形如3 new nodes, 5 new edges, 1 node removed。实现上通过edge_key把边的有向性纳入比较有向图按(u, v, relation)无向图按排序后的(min, max, relation)因此“互指边被折叠”“方向被翻转”这类回归也能在 diff 里暴露出来。三、--cluster-only只重聚类绝不重跑 5–9当用户只想更新社区划分、命名与可视化产物而不想动抽取结果时runbook 要求跳过 Step 1–3直接执行graphify cluster-only .该命令是自包含的它读取已有图重新聚类源码层面对应 graphify/cluster.py 的cluster及其 Leiden 划分、hub 命名、label_communities_by_hub、score_all等逻辑并基于现有图重新生成GRAPH_REPORT.md、graph.json与graph.html。在 CLI 实现里graphify/cli.py 的 cluster 分支它会在完成后打印社区数量并确认三个产物已刷新。runbook 特别强调不要重跑 Step 5–9。原因非常直接——这些步骤读取的是.graphify_extract.json、.graphify_detect.json、.graphify_analysis.json等中间文件而上一次全量构建的 Step 9 清理早已把它们删除重跑只会得到FileNotFoundError#1392。命令结束后把刷新后的GRAPH_REPORT.md摘要照常呈现给用户即可。四、两类模式的快速对照维度--update--cluster-only触发动作添加/修改文件后重抽变化文件已有图上重跑聚类是否重抽取仅变化文件AST 必跑纯代码则跳过语义不抽取是否用 LLM出现 doc/paper/image/video 变化时使用不使用关键产物合并后的graph.json、更新的 manifest重刷GRAPH_REPORT.md/graph.json/graph.html最容易踩的坑prune_sources误加 changed 文件、漏传root/directed重跑 Step 5–9 触发FileNotFoundError五、从 runbook 到源码你可以继续深入的文件增量差异检测实现detect_incremental的 fast/slow 路径、ast/semantic 双 hash 语义、delete 与 exclude 的区分manifest 读写save_manifest的kind/scan_corpus/clear_semantic/clear_ast参数与可移植相对键合并与剪枝实现build_merge的 replace-on-re-extract#2333/#2336、tier 判定、root/directed的兜底逻辑图差异对比graph_diff的有向/无向edge_key归一化与 summary 生成仅打点真正产出语义的文件_stamped_manifest_files对 #933/#1920/#2927/#1897 的完整处理聚类与社区命名cluster与 Leiden 划分、hub 标签推断的实现。在自动化测试侧tests/test_incremental.py、tests/test_incremental_mtime_collision.py 与 tests/test_dedup_remaps_hyperedges.py 覆盖了 mtime 碰撞、超边在合并中的保留等增量场景是对本文所述语义最直接的回归验证。六、总结--update的本质是“把全量构建压缩到变化子集上”detect_incremental负责精确定位变化code-only 快速路径负责把纯代码变更的代价压到零 LLMbuild_merge负责用 replace-on-re-extract 而非 prune 来对账变更、用 prune 只剪真删除、用root/directed保住相对路径与边方向最后save_manifest用“只打点真实产出 清空失败语义 全语料扫描”三重保险确保下一次 diff 不会失真。--cluster-only则是完全独立的自包含捷径前提是尊重“只读图、别碰中间文件”的边界。对 Agent 而言遵守这份 runbook 意味着更新是幂等、可解释且可验证的——每一轮更新的图 diff 都能精确回答“这轮改了什么”。【免费下载链接】graphifyTurn any codebase, with its docs, SQL schemas, configs, and PDFs, into a queryable knowledge graph. A /graphify skill for Claude Code, Cursor, Codex, and Gemini CLI: local deterministic AST parsing, every edge explained, no vector store.项目地址: https://gitcode.com/GitHub_Trending/graph/graphify创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考