Aspire 定时工作流故障监视器monitor-scheduled-workflows 的设计、配置与源码解析【免费下载链接】aspireAspire is the tool for code-first, extensible, observable dev and deploy.项目地址: https://gitcode.com/GitHub_Trending/as/aspireAspire 仓库中有大量无人值守的定时 GitHub Actions 工作流刷新 manifest、更新依赖、清理部署、回合并 release 等。一旦它们失败GitHub 只会邮件通知最后修改该工作流文件的人故障可能数天无人察觉。本文以 docs/ci/monitor-scheduled-workflows.md 为核心完整讲解monitor-scheduled-workflows看门狗watchdog的调度机制、观察列表配置、Issue 去重契约、关闭策略与权限模型并深入 monitor-scheduled-workflows.js 与共享引擎 tracking-issue.js 的源码实现及其单元测试帮助你理解并复用这套每工作流一个去重 Issue的自动化故障上报模式。要解决的问题定时任务失败的静默盲区文档开篇即点明动机仓库中多个定时工作流generate diffs、refresh manifests/SDKs、update models/dependencies、clean up deployments、retrain the labeler、backmerge releases 等长期无人值守运行。当其中一个失败时GitHub 仅向最后编辑该工作流文件的人发送邮件——在大型团队中这个人往往早已不再关注这条流水线一个坏掉的定时任务可以沉默地躺上数天。monitor-scheduled-workflows就是为此设立的看门狗它监视一组指定工作流对每个工作流维护唯一一个去重deduplicated的 GitHub Issue在失败时创建file、在重复失败时追加评论update、在恢复成功时关闭close。文档明确指出这是仓库内部 AzDO build notifier 在 GitHub Actions 侧的对应物两者复用同一套file → update → close-on-green契约区别仅在于按workflow而非按 branch作为键。调度每 2 小时轮询 3 小时回溯窗口 dry-run 手动触发工作流定义在 monitor-scheduled-workflows.yml核心调度与执行结构如下on: schedule: - cron: 0 */2 * * * # every 2 hours workflow_dispatch: inputs: dry_run: description: Inspect and log intended issue actions without mutating GitHub required: false default: false type: boolean permissions: contents: read # checkout to require the local .js module actions: read # list workflow runs / conclusions issues: write # file / comment / close automation-broken issues # One watchdog run at a time; a backlog would only re-evaluate the same state. concurrency: group: monitor-scheduled-workflows cancel-in-progress: false jobs: monitor: runs-on: ubuntu-latest if: ${{ github.repository_owner microsoft }} steps: - uses: actions/checkoutde0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: ref: main persist-credentials: false - name: Evaluate scheduled workflows uses: actions/github-script3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: DRY_RUN: ${{ github.event_name workflow_dispatch inputs.dry_run }} with: script: | const dryRun process.env.DRY_RUN true; await require(./.github/workflows/monitor-scheduled-workflows.js).run({ github, context, core, dryRun });几个关键设计点每 2 小时一次cron: 0 */2 * * *并支持workflow_dispatch手动触发。手动触发可传入dry_run布尔值此时工作流只记录本应采取的 Issue 操作would file / would comment / would close不修改 GitHub 上任何内容——这在调试监视器自身行为时非常有用。concurrency组串行化同一时刻只允许一个看门狗运行在途。注释写得很直白——排队积压的运行只会重复评估同一状态毫无收益。dry_run通过环境变量传递而非直接插值进脚本避免用户输入被内联进github-script脚本内只做process.env.DRY_RUN true的安全判断。为什么用 3 小时轮询窗口这是该设计中最容易被忽视、却决定正确性的细节。看门狗每次运行时只处理3 小时轮询窗口内更新过的已完成定时运行并按时间从旧到新处理。源码中窗口是硬编码常量monitor-scheduled-workflows.js// Conclusions that count as the workflow is broken. cancelled is excluded: // operator cancellation (and concurrency-superseded runs) is not a workflow defect, // and firing on it would create noise. timed_out is NOT excluded — a run that // hits its timeout is treated as broken. const FAILURE_CONCLUSIONS new Set([failure, timed_out, startup_failure]); const SUCCESS_CONCLUSIONS new Set([success]); const WORKFLOW_RUN_PAGE_SIZE 100; // The watchdog runs every two hours. Look back three hours so ordinary GitHub // schedule/queue delay cannot create a gap where a completed run is never seen. const POLLING_WINDOW_MS 3 * 60 * 60 * 1000;窗口3 小时刻意大于调度间隔2 小时目的是让相邻两次看门狗 tick 之间存在重叠区这样即使 GitHub 的调度/排队出现延迟也不会出现某次已完成运行永远落在两个 tick 之间的缝里的盲区。文档还强调重叠不会导致重复上报——因为每次失败运行记录为评论时携带独立的隐藏运行标记重复观察到的同一运行会被去重见下文 Dedup 一节。窗口过滤与排序由纯函数selectRunsForPollingWindow实现按updated_at ?? run_started_at ?? created_at取时间戳落在[now - 3h, now]内的运行被保留并升序排序function selectRunsForPollingWindow(runs, { now new Date(), pollingWindowMs POLLING_WINDOW_MS } {}) { const nowTimestamp now.getTime(); const cutoff nowTimestamp - pollingWindowMs; return (runs ?? []) .filter(run { const timestamp getRunTimestamp(run); return timestamp ! null timestamp cutoff timestamp nowTimestamp; }) .sort((left, right) getRunTimestamp(left) - getRunTimestamp(right)); }观察列表配置与代码分离两类条目语义不同文档强调观察列表不在工作流脚本里而是独立的配置文件 monitor-scheduled-workflows.config.json条目格式为{ file, name, enabled?, selfReports?, labels? }数组。当前仓库的实际观察列表为{ watched: [ { file: generate-api-diffs.yml, name: Generate API Diffs }, { file: generate-ats-diffs.yml, name: Generate ATS Diffs }, { file: refresh-manifests.yml, name: Refresh Manifests }, { file: update-dependencies.yml, name: Update Dependencies }, { file: update-ai-foundry-models.yml, name: Update AI Foundry Models }, { file: update-azure-vm-sizes.yml, name: Update Azure VM Sizes }, { file: update-aspire-skills-bundle.yml, name: Update Aspire Skills Bundle }, { file: deployment-cleanup.yml, name: Deployment Cleanup }, { file: labeler-cache-retention.yml, name: Labeler Cache Retention }, { file: warm-cli-e2e-image-cache.yml, name: Warm CLI E2E Image Cache }, { file: locker.yml, name: Lock Threads }, { file: backmerge-release.yml, name: Backmerge Release to Main }, { file: sync-main-to-release-14.yml, name: Sync Main to Release 14.0 }, { file: tests-outerloop.yml, name: Outerloop Tests, selfReports: true }, { file: tests-quarantine.yml, name: Quarantined Tests, selfReports: true }, { file: tests-daily-smoke.yml, name: Daily CLI Smoke Tests, selfReports: true, labels: [area-cli] }, { file: deployment-tests.yml, name: Deployment E2E Tests, selfReports: true, labels: [area-testing, deployment-e2e] } ] }文件头部的$schema-note字段本身就是一份内联使用手册条目默认处于监视状态watched by defaultenabled: false可在不删除条目的情况下停止监视selfReports: true表示该工作流会在流水线内自行上报其常规失败看门狗随后只兜底startup_failure和timed_out可选的labels[]会附加到所创建的 Issue 上。配置文件中的两类条目对应两种完全不同的监视语义全监视条目full-watchfailure结论无歧义意味着坏了的工作流。看门狗在任何 failure 结论下都会提 Issue。包括generate-api-diffs、generate-ats-diffsrefresh-manifestsupdate-dependencies、update-ai-foundry-models、update-azure-vm-sizes、update-aspire-skills-bundledeployment-cleanuplabeler-cache-retentionwarm-cli-e2e-image-cachelockerbackmerge-release兜底条目backstopselfReports: true这些工作流自身内置了if: failure()的门控 reporter job会在流水线内为自己的常规失败提 Issue例如 specialized-test-failure-issues.md 描述的 outerloop/quarantine 测试失败 reporter以及 pipeline-failure-issues.md 描述的每日冒烟/部署测试 reporter。对这类工作流看门狗只记录startup_failure和timed_out两种结论——恰好是流水线内 reporter 无法捕获的两种——并且绝不记录普通failure如果记录就会在第二个标记名下重复提 Issuedouble-file。tests-outerloop、tests-quarantinetests-daily-smoke、deployment-tests增删改观察条目的操作规则文档给出四条明确的配置操作规范操作做法添加全监视工作流加一个{ file, name }条目缺省即被监视为自上报工作流添加兜底加selfReports: true给所提 Issue 附加标签automation-broken之外加labels: [ ... ]数组停止监视某工作流不删除设enabled: false禁用条目会被跳过并记录日志禁用即跳过的逻辑在源码中是一个简单的过滤函数enabled缺省视为启用monitor-scheduled-workflows.jsfunction selectEnabled(config) { const watched Array.isArray(config?.watched) ? config.watched : []; return watched.filter(entry entry typeof entry.file string entry.enabled ! false); }为什么兜底条目只记录 startup_failure / timed_out这一节是理解整个设计的核心。流水线内 reporter 本质是一个门控在if: failure()上的 job而 GitHub 有两种结论会绕过它startup_failure运行根本没有启动任何 job坏掉的 YAML、无法解析的uses:、失效的 runner于是 reporter job 自身也不会运行timed_outjob 级超时属于cancelled-class状态failure()表达式为 falsereporter job 因此不会运行。而普通failure包括以failure形式呈现的 step 级超时会触发 reporter所以看门狗把这些留给流水线内 reporter 处理。源码中用常量把这层语义固化下来monitor-scheduled-workflows.jsconst BACKSTOP_CONCLUSIONS new Set([startup_failure, timed_out]);决策入口是纯函数decideAction传入结论、现有 Issue 与哪些结论算坏的集合返回record/close/noop三选一function decideAction({ conclusion, issue, failureConclusions FAILURE_CONCLUSIONS }) { const normalized typeof conclusion string ? conclusion.toLowerCase() : null; if (normalized ! null failureConclusions.has(normalized)) { return issue ? { action: record, reason: latest run concluded ${normalized}; recording on issue #${issue.number} } : { action: record, reason: latest run concluded ${normalized}; no open issue }; } if (normalized ! null SUCCESS_CONCLUSIONS.has(normalized)) { return issue ? { action: close, reason: latest run concluded success; closing issue #${issue.number} } : { action: noop, reason: latest run succeeded; nothing open }; } // null (no completed run yet), cancelled, skipped, neutral, etc. return { action: noop, reason: latest conclusion ${normalized ?? none} is not actionable }; }注意run()主循环中如何按条目类型切换失败集合const failureConclusions wf.selfReports ? BACKSTOP_CONCLUSIONS : FAILURE_CONCLUSIONS;——selfReports条目在普通failure时落入noop分支正好实现绝不为它提第二个 Issue的契约。有意排除在观察范围之外的工作流文档还专门列出了刻意不监视的对象及理由这部分同样是设计约束的一部分backmerge-release此处为全监视但它还会在green成功运行下提merge conflicts Issue——那是其自身工作流内处理的另一件事与看门狗无关。workflow_call构件如tests.yml、run-tests.yml它们的失败会反映在调用方caller上信号属于调用方不应在看门狗处重复上报。ci.ymlred-main 的 push 失败由ci.yml自己提 Issue并自关闭见 ci-failure-issues.md。Agentic 的*.lock.yml工作流gh-aw有自己的错误上报机制。会创建什么样的 Issue标题、标签、标记与评论当一个被监视工作流在main分支、看门狗轮询窗口内出现了结论为failure、timed_out或startup_failure的已完成定时运行时会触发如下 Issue 契约标题Scheduled workflow failing: display name标签automation-broken由工作流幂等地创建正文首行标记隐藏的 HTML 注释!-- automation-broken:workflow-file --作为去重键几个实现层面的关键约定每个工作流同时至多存在一个 open Issue。正文是提 Issue 时一次性写定的固定描述隐藏标记 说明文字之后不再改写正文每次新观察到的失败运行都记录为一条评论携带运行链接、commit 与结论——正是评论触发了 通知。评论按运行去重扫描器可能在多个 tick 观察到同一次失败运行。每条评论内嵌隐藏的!-- run:id --标记已存在对应评论的运行是 no-op直到更新的运行完成。cancelled与skipped被刻意忽略——操作者取消和跳过的运行不是工作流缺陷。反之timed_out被视为失败并会提 Issue。对backstop 条目selfReports: true普通failure同样被忽略其流水线内 reporter 拥有它只有startup_failure和timed_out会提 Issue且正文措辞会明确说明这一点以区别于流水线内 reporter 提的 Issue。Issue 携带automation-broken加上条目级labels并被打上autoClose:true戳。源码中的 Issue 内容构造标记、标题与正文的构造函数都是纯函数。正文链接指向仓库 Actions 页面的工作流视图并按selfReports切换导语措辞monitor-scheduled-workflows.jsfunction buildIssueBody({ marker, displayName, workflowFile, selfReports false }) { const link \${workflowFile}\; const lead selfReports ? The scheduled workflow ${link} (**${displayName}**) had a run that **failed to start or timed out**. Its normal failures are reported separately by an in-pipeline job; this issue backstops runs that never produced a result. : The scheduled workflow ${link} (**${displayName}**) is failing.; return tracking.buildBody({ marker, autoClose: true, lead, note: [ Filed and updated automatically by the scheduled-workflow watchdog. Each, failed run is added as a comment below, and the issue is **closed, automatically** on the next successful run., See [docs/ci/monitor-scheduled-workflows.md](https://link.gitcode.com/i/d122c64f7c7478ab949782b3b825ff4b)., ], }); }每次失败运行产生的评论格式formatComment// Comment recorded per newly-observed failed run. // failure: { runUrl, runNumber, sha, conclusion } function formatComment({ runUrl, runNumber, sha, conclusion }) { const runLink runUrl ? run #${runNumber ?? ?} : run #${runNumber ?? ?}; const shaPart sha ? (commit \${String(sha).slice(0, 8)}\) : ; return The scheduled run concluded \${conclusion}\ in ${runLink}${shaPart}.; }单元测试 MonitorScheduledWorkflowsTests 精确锁定了这些输出例如断言buildMarker(generate-api-diffs.yml)必须等于!-- automation-broken:generate-api-diffs.yml --、标题必须为Scheduled workflow failing: Generate API Diffs、selfReports条目正文必须包含failed to start or timed out与reported separately、且所有正文都必须包含!-- autoclose:true --戳。什么会被关闭close-on-green当轮询窗口内最新的已完成定时运行结论为success时该工作流对应的 openautomation-brokenIssue 会被追加一条 latest run succeeded 评论并以state_reason: completed关闭。源码中的关闭路径有两个前置守卫monitor-scheduled-workflows.jsconst autoClose tracking.readAutoClose(issue.body); if (autoClose ! true) { core.info(Issue #${issue.number} for ${wf.file} does not opt into auto-close; leaving it open.); continue; } // ... await tracking.closeIssue(github, owner, repo, issue.number); await tracking.addComment(github, owner, repo, issue.number, Latest run succeeded (run #${newest.run_number}). Closing automatically.);readAutoClose是防御式解析正文中找不到或解析不出!-- autoclose:true/false --戳时返回null调用方必须把null当作不自动关闭处理——这样人工编辑过的正文、或在引入戳记之前创建的旧 Issue都不会被看门狗从 triager 手中抢着关掉。共享引擎 tracking-issue.js 中的实现function readAutoClose(body) { if (typeof body ! string) { return null; } const match /!--\s*autoclose:(true|false)\s*--/i.exec(body); if (match null) { return null; } return match[1].toLowerCase() true; }去重Dedup为什么刻意不用 Search API这是文档中一个很有工程判断力的决定Issue 查找使用GET /issues?labelsautomation-brokenstateopen强一致的 list API加上本地正文标记过滤而不是Search API——因为 Search API 的最终一致性窗口可能让近乎同时的两次运行各自看到0 命中从而重复提 Issue。源码 tracking-issue.js 的注释与实现印证了这一点并且顺手处理了 REST 模型中PR 也是 issue的坑// Uses the (strongly-consistent) list // API rather than Search, whose eventual-consistency window would let // near-simultaneous pollers each see 0 hits and file duplicates. async function listIssuesByLabel(github, owner, repo, label, { state all } {}) { const items await github.paginate(github.rest.issues.listForRepo, { owner, repo, labels: label, state, per_page: 100, }); // listForRepo returns pull requests too (they are issues in the REST model). // Exclude them so a labeled PR whose body happens to carry a tracking marker is // never mistaken for the managed issue and then commented/closed in its place. return items.filter(item !item.pull_request); }若极端情况下出现两个 open Issue 携带同一标记例如有人手动创建了一个编号最小最旧的视为 canonical由 TrackingIssueTests 中的FindOpenIssueForMarkerReturnsOldestMatch用例固化三个候选 issue 11/40/88 中必须返回 11。此外看门狗一次性列出含 closed 状态的 Issue 并跨条目复用原因是某工作流再次失败时应当重开既有 canonical tracker而不是提一个重复的新 IssuerecordRun引擎对 closed 状态命中会自动reopenIssue。dry-run 模式下也会模拟这套状态迁移占位 issue 推入本地列表、closed 置为 open保证 dry-run 日志反映真实的多 tick 行为。权限与鉴权最小 token不触发下游文档给出的权限模型job 使用默认GITHUB_TOKEN仅需actions: read列出运行/结论与issues: write提 Issue/评论/关闭contents: read用于 checkout 本地的.js模块脚本通过require(./.github/workflows/monitor-scheduled-workflows.js)加载因此 YAML 只有一层薄壳逻辑全部在被测试的 JS 模块里。无需 App tokenIssue 创建在同一仓库内且看门狗刻意不触发下游自动化——避免监视器自己制造的 Issue 又触发别的机器人的反馈环。标签automation-broken由ensureLabel幂等创建422 视为已存在并静默忽略。为什么它从不喧闹地失败run()主循环中对每个工作流的运行列表查询都包在try/catch里单个无法读取的工作流只会记一条 warning 并continue跳过而不是让整个看门狗运行失败monitor-scheduled-workflows.js} catch (error) { core.warning(Could not list runs for ${wf.file}: ${error.message}); continue; }同时查询运行列表时显式带event: schedule过滤——源码注释解释得很清楚被监视的工作流常常同时有workflow_dispatch个别如warm-cli-e2e-image-cache.yml还有push:触发器。若不加该过滤轮询窗口内的一次手动/push 成功运行会自动关闭真实的定时失败 Issue掩盖这个看门狗专门要抓的静默故障而一次手动/push 失败则会提一个假 Issue。const runs await github.rest.actions.listWorkflowRuns({ owner, repo, workflow_id: wf.file, branch: main, event: schedule, status: completed, per_page: WORKFLOW_RUN_PAGE_SIZE, });逻辑分层与测试可复用的 tracking-issue 引擎文档最后一节描述了代码分层这也是整个方案可迁移性的来源通用引擎tracking-issue.js仓库无关repo-agnostic的 tracking-issue 机制——标记去重查找、按运行去重的评论记录循环、octokit 原语ensureLabel / listByLabel / create / comment / close / reopen / hasCommentForRun以及唯一承载去重契约的编排函数recordRun。它不关心任何仓库、标签、工作流或产品刻意保持纯净。看门狗专属模块monitor-scheduled-workflows.js标记命名空间automation-broken:前缀、Issue 标题/正文、每运行的失败评论、record/close/noop 决策均为纯函数以及run()编排器——读配置、循环轮询、驱动引擎。由工作流的github-script步骤调用。这套机制在仓库中被多个 reporter 共享specialized-test 失败 reporter、nightly-pipeline 失败 reporter、red-main CI reporter 都复用同一引擎各自行使自己的标记与文案策略。测试链路是Node harness C# xUnit的组合层文件作用引擎单测TrackingIssueTests通过 tracking-issue.harness.js 以 Node 子进程调用纯 helper断言标记去重、recordRun的 find-or-create 评论去重等契约看门狗单测MonitorScheduledWorkflowsTests通过 monitor-scheduled-workflows.harness.js 驱动buildMarker/buildIssueTitle/decideAction/selectEnabled/buildIssueBody/formatComment六个操作集成测试MonitorScheduledWorkflowsIntegrationTests通过 monitor-scheduled-workflows.integration.harness.js 以 fake 驱动run()全流程harness 的形态值得借鉴C# 测试把{ operation, payload }写成 JSON 文件Node 以 CLI 方式读取并回传 JSON 结果从而把 JS 纯函数纳入仓库统一的 xUnit 测试体系。decideAction的关键用例矩阵包括普通条目failure/timed_out/startup_failure且无 Issue →record有 Issue → 仍record已记录的去重交给下游recordRun扫评论完成success 有 Issue →closesuccess 无 Issue →noopcancelled/skipped/null→ 一律noopselfReports条目startup_failure→record、timed_out→record、failure→noop避免 double-file、success→close。修改时的对齐清单文档结尾给维护者的约束同样值得记住当你改动工作流的 job/step 名称或模块的导出契约时必须保持以下五者对齐——工作流 YAML、.js模块、harness、测试、以及本文所对应的文档 docs/ci/monitor-scheduled-workflows.md。这是该仓库对脚本 文档 测试三位一体的强制约定。总结一个可复用的无人值守自动化故障上报模式纵观 docs/ci/monitor-scheduled-workflows.md 及其实现这套看门狗沉淀了一套清晰可复用的模式观察列表外置为 JSON 配置与脚本解耦支持enabled/selfReports/labels三个正交维度每主题一个 canonical Issue以正文中的隐藏 HTML 标记为去重键查找走强一致的 list API 而非最终一致的 Search API正文只写一次失败以评论累积评论内嵌每运行标记实现跨 tick 去重评论本身即通知载体结论分类学驱动决策failure/timed_out/startup_failure算坏success触发关闭cancelled/skipped刻意忽略对自上报工作流只兜底流水线内 reporter 够不着的两种结论调度间隔 轮询窗口用重叠消除静默缝隙用运行级去重抵消重叠带来的重复观察dry-run 手动触发 单实例并发 逐工作流容错让监视器自身安静、可验证、不会成为新的故障源。如果你在自己的仓库中也有失败后无人知晓的定时任务这套 file → update → close-on-green 的契约、以及纯决策函数 Node harness xUnit的测试组织方式都可以直接从 Aspire 仓库的 .github/workflows/monitor-scheduled-workflows.js 与 .github/workflows/tracking-issue.js 中借鉴移植。【免费下载链接】aspireAspire is the tool for code-first, extensible, observable dev and deploy.项目地址: https://gitcode.com/GitHub_Trending/as/aspire创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考