Python后端中间件专题26只看平均响应时间会误判——Prometheus RED 与低基数100 个请求里99 个耗时 20 ms一个耗时 10 s平均约 120 ms报一个平均数既看不出尾部用户的十秒等待也不知道有多少请求失败。TicketFlow 在请求完成时累计请求数、状态类别和直方图再给 Outbox 与 Worker 加能反映异步滞后的指标。要掌握的量与前置本课要从 Counter/Histogram 计算 R请求速率、E错误比例、D延迟分布用 route 模板而非工单 ID 控制序列数量解释 Outbox Gauge 与 broker 队列长度并非同一量。前置是 23 的单入口、25 的业务 ID 日志和 Prometheusrate()的窗口概念。安装项目开发依赖即可运行本地 selector真正 scrape API A/B、relay、Worker 属于远端验收。上一课练习答案答案 EX-25-01 [code]在project/保存为ex25_log_probe.pyimportjsonfromticketflow.observability.loggingimportJsonEventLogger lines[]loggerJsonEventLogger(lines.append)private_fields{broker_password:rabbit-private,context:{authorization:Bearer bearer-private},}logger.event(outbox_publish_failed,request_idreq-25,ticket_idticket-25,**private_fields,)assertlen(lines)1eventjson.loads(lines[0])assertevent[request_id]req-25assertevent[ticket_id]ticket-25assertevent[broker_password][REDACTED]assertevent[context][authorization][REDACTED]assertrabbit-privatenotinlines[0]andbearer-privatenotinlines[0]print(event[event],event[ticket_id],event[broker_password])运行命令PYTHONPATHsrc python ex25_log_probe.pyPowerShell$env:PYTHONPATHsrc; python ex25_log_probe.py。预期输出outbox_publish_failed ticket-25 [REDACTED]这是内存 sink 的脱敏合同不是三个生产进程的日志样本。答案 EX-25-02 [prose]先从入口request_id与返回的ticket_id查询 SQL 工单和同事务 Outbox拿稳定event_id确认 Outbox 是 pending 还是已确认发布再看 relay 的尝试/错误、RabbitMQ 任务、Worker 的消费 receipt 与通知 fake sink。业务 ID 可在有访问控制、保留期限的日志里帮助定点查事故若把每张工单或每租户作为指标标签一次创建就会增加长期时间序列存储、查询和告警都会失控还暴露不必要的业务维度。当前逐事件埋点已经存在API 成功创建写api_ticket_createdrelay 成功/可重试失败分别写outbox_event_published、outbox_event_retryableWorker 成功、可重试失败和终止失败分别写worker_event_handled、worker_event_retryable、worker_event_failed。本地合同已经验证这些事件的字段结构以及凭据、Authorization 等敏感字段的递归脱敏。尚未取得的是一次真实 APIrelayWorker 三进程运行中的关联日志样本第 25 课远端 selector 对这一项仍是PENDING不能把本地合同写成远端链路已通过。用可聚合的量描述请求ticketflow_http_requests_total{method,route,status_class}用方法、路由模板如/tickets/{ticket_id}和5xx聚合ticketflow_http_request_duration_seconds_bucket{method,route,le}给出直方图桶。粗略五分钟查询sum(rate(ticketflow_http_requests_total[5m])) sum(rate(ticketflow_http_requests_total{status_class5xx}[5m])) / sum(rate(ticketflow_http_requests_total[5m])) histogram_quantile(0.95, sum by (le) (rate(ticketflow_http_request_duration_seconds_bucket[5m])))分母为零时 PromQL 不能定义正常错误率看板为了避免除零采用clamp_min(...,0.000001)低流量/无流量时数值并不能证明 SLO 达标要同时看 request rate 与 scrapeup。p95 是按桶插值得到的估计多个服务实例要先按le聚合本文的全局 p95 不保留 route某条慢路径可能被总量遮住需要按 route 分组进一步定位。Histogram 目前没有 status class 标签不能从它直接拆“仅 5xx 的 p95”。异步侧ticketflow_outbox_pending表示 SQL 已提交、尚未确认发布的数量ticketflow_outbox_oldest_age_seconds表示最老未发布事件年龄。计数可能因为吞吐量上涨而短时增加年龄持续增大则更像 relay/broker 故障。ticketflow_worker_tasks_total{outcome}、ticketflow_outbox_publish_total{outcome}、搜索请求时延和缓存命中都有枚举 outcome不能用异常消息、trace ID 或 tenant 当标签。Outbox pending 不等于 RabbitMQ queue depth前者恢复为零时Worker 仍可能积压。TicketFlowMetrics.record_cache/record_outbox_publish/record_search/record_worker对标签值做白名单observe_http本身没有检查 route 必为模板所以安全性还依赖 API 中间件从路由对象取模板未匹配时统一写unmatched。若调用者改成scope[path]每个工单 URL 都能变成新序列修改埋点时需测试这一点。从单测到抓取的证据层正式本地 selectorpython tools/verify_executable_revisions.py --lesson 26 --python python独立安装并运行 baseline 与 09–26 的全部历史合同。它检查独立CollectorRegistry、route 模板、cache/outbox/search/worker 样本但没验证 Prometheus 真抓到了两台 API。远端配置列出api-a:8000、api-b:8000、relay:9101、worker:9102需经 live target/sample 检查。Worker 的 Compose 进程采用 Celery prefork任务在子进程记录 Counter、父进程提供/metrics运行时启动前清理专用 multiprocess 目录并由父进程合并子进程文件不能用普通父进程内存 registry 代替。授权远端设置TICKETFLOW_PROMETHEUS_URL后运行python scripts/run_remote_selector.py --checkpoint 26 --selector tests/integration/test_worker_metrics_scrape.py::test_running_prefork_worker_completed_task_is_scraped创建工单、等通知完成并要求 Prometheus 中 Worker 的successCounter 相对基线增长未实际运行时保持 PENDING不能声称抓取 PASS。若 rate 为空先看/api/v1/targets的健康与 scrape URL再看 exporter/metrics、请求是否发生若 p95 奇怪检查桶聚合与窗口样本不要把“无流量”报告为 0 ms 成功。本课练习练习 EX-26-01 [code]在独立CollectorRegistry里用TicketFlowMetrics记录 97 个 200、3 个 503相同 GET/路由模板再设置 pending7从 sample 中读取两种状态类别的 Counter断言分别为 97/3、没有业务 ID 标签打印2xx97 5xx3 pending7。不要把这当真实流量。练习 EX-26-02 [prose]给一次“HTTP 错误率 5% 且 Outbox pending 增至 120”的事故设计排查顺序哪张面板先看请求量和 p95哪些指标区分 broker/relay 与 API 故障假设业务 SLO 为 99.5% 成功率解释它与配置的五分钟 5% 告警阈值并非一回事以及低流量下比率的陷阱。收尾指标可以发现趋势却不会自动告诉值班人员去哪一页、执行什么安全动作。27 用已 provision 的 Grafana 看板、告警规则和 runbook 走一遍事故它也会指出静态规则与实际故障语义之间的差距。完整核心模块有界标签的业务指标from__future__importannotationsfromprometheus_clientimportCollectorRegistry,Counter,Gauge,HistogramclassTicketFlowMetrics:Own metric contracts so high-cardinality business IDs cannot become labels.def__init__(self,*,registry:CollectorRegistry)-None:self.http_requestsCounter(ticketflow_http_requests_total,Completed TicketFlow HTTP requests,(method,route,status_class),registryregistry,)self.http_secondsHistogram(ticketflow_http_request_duration_seconds,End-to-end HTTP response duration,(method,route),registryregistry,)self.cache_operationsCounter(ticketflow_cache_operations_total,Cache outcomes by bounded operation,(operation,outcome),registryregistry,)self.outbox_pendingGauge(ticketflow_outbox_pending,Events committed in PostgreSQL but not yet confirmed by the broker,registryregistry,)self.outbox_oldest_ageGauge(ticketflow_outbox_oldest_age_seconds,Age of the oldest unpublished Outbox event,registryregistry,)self.outbox_publishCounter(ticketflow_outbox_publish_total,Outbox publish attempts by bounded outcome,(outcome,),registryregistry,)self.search_requestsCounter(ticketflow_search_requests_total,Search requests by outcome,(outcome,),registryregistry,)self.search_secondsHistogram(ticketflow_search_request_duration_seconds,Search request latency,(outcome,),registryregistry,)self.worker_tasksCounter(ticketflow_worker_tasks_total,Worker task outcomes,(outcome,),registryregistry,)defobserve_http(self,*,method:str,route:str,status_code:int,duration_seconds:float,)-None:status_classf{status_code//100}xxself.http_requests.labels(method.upper(),route,status_class).inc()self.http_seconds.labels(method.upper(),route).observe(duration_seconds)defrecord_cache(self,*,operation:str,outcome:str)-None:allowed_operations{ticket_get,ticket_put,ticket_invalidate}allowed_outcomes{hit,miss,success,error,degraded}ifoperationnotinallowed_operationsoroutcomenotinallowed_outcomes:raiseValueError(cache metric labels must come from the bounded contract)self.cache_operations.labels(operation,outcome).inc()defset_outbox_pending(self,count:int)-None:ifcount0:raiseValueError(outbox pending count cannot be negative)self.outbox_pending.set(count)defset_outbox_stats(self,*,pending:int,oldest_age_seconds:float)-None:self.set_outbox_pending(pending)self.outbox_oldest_age.set(max(0.0,oldest_age_seconds))defrecord_outbox_publish(self,outcome:str)-None:ifoutcomenotin{confirmed,retryable,failed}:raiseValueError(invalid Outbox outcome)self.outbox_publish.labels(outcome).inc()defrecord_search(self,*,outcome:str,duration_seconds:float)-None:ifoutcomenotin{success,unavailable,rejected}:raiseValueError(invalid search outcome)self.search_requests.labels(outcome).inc()self.search_seconds.labels(outcome).observe(duration_seconds)defrecord_worker(self,outcome:str)-None:ifoutcomenotin{success,retry,failure,duplicate}:raiseValueError(invalid worker outcome)self.worker_tasks.labels(outcome).inc()