Keep 集成 VictoriaMetrics从 Docker 部署到 Provider 配置与指标告警工作流实战【免费下载链接】keepThe open-source AIOps and alert management platform项目地址: https://gitcode.com/GitHub_Trending/kee/keep本篇技术指南聚焦 Keep 开源 AIOps 告警管理平台与 VictoriaMetrics 监控体系的集成实践完整覆盖三大部分使用 Docker 快速部署 VictoriaMetrics 全家桶victoriametrics / vmagent / vmalert / alertmanager / grafana、在 Keep 中正确配置 VictoriaMetrics ProviderVMAlert 与 VM Backend 双端点认证模型、以及通过工作流完成指标查询、阈值判断与告警创建的端到端方案。读完本文你将掌握 Keep 与 VictoriaMetrics 从环境准备、配置接入到告警闭环的全部实战能力。一、集成架构概览Provider 与 VMAlert / VM Backend 的双端点模型在 Keep 的源码中VictoriaMetrics 集成由 victoriametrics_provider.py 实现其核心设计是同时面向两个服务端点VMAlert默认端口8880负责根据规则评估指标并触发告警Keep 通过其 HTTP API/api/v1/alerts拉取当前处于 firing / pending 等状态的告警VM Backend默认端口8428VictoriaMetrics 时序数据库本体Keep 通过其 Prometheus 兼容 API/api/v1/query与/api/v1/query_range查询指标数据。从源码的validate_config()victoriametrics_provider.py可以看到两者至少必须配置其一否则会抛出At least one of VMAlert or VM Backend must be configured异常。这决定了两种典型用法场景必须配置的端点用途只拉取/接收告警VMAlertHost/Port 或 URL告警进入 Keep 告警中心、触发工作流只查询指标VM BackendHost/Port 或 URL工作流中以 step 形式执行指标查询完整接入两者都配置告警 指标查询双能力此外该 Provider 还通过WebhookAlertmanager 推送方式接收告警形成推拉结合的告警接入能力详见下文第四节。二、第一步使用 Docker 部署 VictoriaMetrics 全家桶keep/providers/victoriametrics_provider/README.md 给出了官方的本地部署路径。由于 Keep 前端服务默认占用3000端口官方部署步骤专门对 docker-compose 做了端口规避处理完整操作如下。1. 克隆 VictoriaMetrics 仓库# 克隆 VictoriaMetrics 官方仓库仓库地址请以 VictoriaMetrics 官方文档为准 git clone VictoriaMetrics 官方仓库地址2. 进入 docker 部署目录VictoriaMetrics 官方仓库内置了完整的 docker-compose 编排文件聚合了 victoriametrics、vmagent、vmalert、alertmanager、grafana 五个组件cd deployment/docker3. 修改端口避免与 Keep 服务冲突VictoriaMetrics 全家桶中 Grafana 默认监听3000与 Keep 前端keep-ui冲突。官方给出的处理方式是直接用sed在 compose 文件中完成全局替换sed -i -e s/3000:3000/3001:3000/ -e s/127.0.0.1:3000/127.0.0.1:3001/ docker-compose.yml该命令将 Grafana 的宿主机映射端口从3000改为3001同时修正127.0.0.1绑定地址避免与 Keep 服务抢占端口。4. 启动全部服务docker-compose up -d5. 服务端口一览启动完成后各组件分别监听以下端口以本 README 说明为准服务访问地址说明victoriametricshttp://localhost:8428时序数据库后端VM Backendgrafanahttp://localhost:3001可视化面板已避开 Keep 的 3000 端口vmagenthttp://localhost:8429指标抓取代理vmalerthttp://localhost:8880告警规则评估引擎alertmanagerhttp://localhost:9093告警聚合与路由提示8428VM Backend与8880VMAlert正是 Keep Provider 认证配置中的两个默认端口与源码中VMAlertPort默认8880、VMBackendPort默认8428一一对应见 victoriametrics_provider.py 与 victoriametrics_provider.py。三、Provider 认证配置参数详解与双端点配置方式VictoriaMetrics Provider 的完整认证参数由VictoriametricsProviderAuthConfigvictoriametrics_provider.py定义官方文档与自动生成片段 docs/snippets/providers/victoriametrics-snippet-autogenerated.mdx 对其做了完整说明。全部参数均可选required: False但受至少配置一个端点的约束3.1 VMAlert 端点配置参数默认值说明VMAlertHost无VMAlert 运行的主机名或 IP如http://localhost、http://192.168.1.100VMAlertPort8880VMAlert 监听端口VMAlertURL无VMAlert 完整 URL是 Host/Port 组合的替代写法如http://vmalert.mydomain.com:88803.2 VM Backend 端点配置参数默认值说明VMBackendHost无VictoriaMetrics 后端主机名或 IPVMBackendPort8428后端监听端口VMBackendURL无后端完整 URL是 Host/Port 组合的替代写法如http://vm.mydomain.com:84283.3 认证与校验配置参数默认值说明BasicAuthUsername无Basic 认证用户名BasicAuthPassword无Basic 认证密码sensitive: True属于敏感配置会走 Keep 的密钥管理SkipValidationfalse设为true时跳过认证连通性校验insecurefalse跳过 TLS 证书校验布尔开关3.4 配置示例以 VMAlert 为主provider: type: victoriametrics authentication: VMAlertURL: http://localhost:8880 # 或使用 Host Port 的组合写法 # VMAlertHost: http://localhost # VMAlertPort: 8880 BasicAuthUsername: admin BasicAuthPassword: secret3.5 源码层面的连接行为从源码可以观察到三个值得注意的实现细节URL 与 Host/Port 互斥优先vmalert_host/vmbackend_host属性victoriametrics_provider.py优先使用完整 URL未提供 URL 时才拼接Host:Port并对结果做rstrip(/)去尾斜杠处理。HTTPS 优先、自动回退 HTTP当 Host 未显式携带http://或https://前缀时Provider 会先尝试 HTTPS 握手捕获requests.exceptions.SSLError后回退为 HTTP并缓存探测结果。TLS 控制所有 HTTP 请求均携带verifynot self.authentication_config.insecure即开启insecure可跳过自签名证书校验。四、连接校验与 Scopeconnected 作用域Provider 定义了唯一的强制作用域connectedThe user can connect to the client由validate_scopes()victoriametrics_provider.py实现若SkipValidation True直接返回{connected: True}跳过一切连通性检查若配置了 VMAlert则向其发起GET请求200视为连接成功若配置了 VM Backend同样发起GET请求并校验200任一检查失败会在返回结果中拼接形如VMAlert error: 404/VM Backend error: 500的错误信息Keep 侧据此判断 Provider 未连通。这一机制保证了在 UI 中保存 Provider 时Keep 能实时校验配置的端点是否真实可达。五、接收告警的两条路径VictoriaMetrics 生态的告警链路通常是victoriametrics 存储指标 → vmalert 基于规则评估 → alertmanager 聚合路由。Keep 提供两种接入方式。5.1 路径一Alertmanager Webhook 推送omnidirectionalProvider 源码中的webhook_templatevictoriametrics_provider.py声明该 Provider 利用 Prometheus Alertmanager 的可配置 Webhook 能力完整模板如下route: receiver: keep group_by: [alertname] group_wait: 15s group_interval: 15s repeat_interval: 1m continue: true receivers: - name: keep webhook_configs: - url: {keep_webhook_api_url} send_resolved: true http_config: basic_auth: username: api_key password: {api_key}其中{keep_webhook_api_url}对应 Keep 的 Webhook 接收地址形如KEEP_BACKEND_URL/alerts/event/victoriametrics见 victoriametrics-snippet-autogenerated.mdx并推荐使用api_key作为 Basic 认证用户名、Keep API Key 作为密码。send_resolved: true保证告警恢复事件也能推送进 Keep从而正确闭环告警状态。5.2 路径二从 VMAlert 拉取告警pull_get_alerts()victoriametrics_provider.py调用GET {vmalert_host}/api/v1/alerts解析data.alerts数组并映射为 Keep 的AlertDto。映射关系揭示了 Keep 与 VictoriaMetrics 字段的对应规则VMAlert 字段Keep AlertDto 字段说明namename告警名称idid告警 IDannotations.descriptiondescription告警描述annotations.summarymessage告警摘要信息statestatus状态映射见下方状态表labels.severityseverity严重级别映射见下方严重级别表activeAtstartedAt告警触发时间sourceurl告警来源链接rule_idevent_id关联的规则 IDlabelslabels原始标签透传严重级别映射SEVERITIES_MAPvictoriametrics_provider.pyVictoriaMetrics 值Keep 严重级别criticalCRITICALhighHIGHwarningWARNINGlowLOWtest/infoINFO状态映射STATUS_MAPvictoriametrics_provider.pyVMAlertstate值Keep 告警状态firingFIRINGresolvedRESOLVEDacknowledgedACKNOWLEDGEDsuppressedSUPPRESSEDpendingPENDING未命中的状态/级别分别回退为FIRING与LOW。同时Provider 暴露的 Webhook 事件解析路径_format_alert()victoriametrics_provider.py针对 Alertmanager 推送的alerts数组将labels.alertname映射为name、annotations.description映射为description、annotations.summary映射为message并使用fingerprint作为告警去重指纹与 ID。六、查询指标query 与 query_range 两种查询类型官方文档 docs/providers/documentation/victoriametrics-provider.mdx 明确VictoriaMetrics Provider 支持通过query和query_range两种类型查询指标。底层实现在_query()victoriametrics_provider.py对应 VictoriaMetrics 的 Prometheus 兼容 HTTP API6.1query类型瞬时查询对应GET /api/v1/query返回查询时刻的单个数据点。参数必填说明与示例query是要执行的 PromQL 查询如sum(rate(http_requests_total{jobapi-server}[5m]))queryType是固定传querystart否查询的时间点如2024-01-01T00:00:00Z透传为 API 的time参数6.2query_range类型范围查询对应GET /api/v1/query_range返回一段时间序列上的数据点。参数必填说明与示例query是要执行的 PromQL 查询如sum(rate(http_requests_total{jobapi-server}[5m]))queryType是固定传query_rangestart是起始时间如2024-01-01T00:00:00Zend是结束时间如2024-01-01T00:00:00Zstep是采样步长如15s6.3 源码实现细节_query()首先校验vmbackend_enabled若未配置 VM Backend 端点则直接抛出VM Backend is not configured瞬时查询返回response[data][result]范围查询返回完整 JSON 响应工作流中可通过results.data.result路径取值两者均支持携带 Basic Auth 与insecureTLS 选项queryType传非法值时会抛出Invalid query type。在 Provider 自测入口victoriametrics_provider.py中还演示了通过环境变量注入端点地址的调用方式VMALERT_HOST/VMALERT_USER/VMALERT_PASSWORD控制 VMAlert 拉取VMBACKEND_HOST/VMBACKEND_USER/VMBACKEND_PASSWORD控制指标查询默认地址分别为http://localhost:8880与http://localhost:8428。七、工作流实战从指标查询到告警创建仓库提供了三个可直接运行的 VictoriaMetrics 工作流示例examples/workflows 目录官方文档也收录了对应的告警评估示例docs/alertevaluation/examples/victoriametricssingle.mdx、docs/alertevaluation/examples/victoriametricsmulti.mdx。7.1 基础形态工作流中的查询 Step在 Keep 工作流中VictoriaMetrics Provider 以step形式执行查询模板如下来自 victoriametrics-snippet-autogenerated.mdxsteps: - name: Query victoriametrics provider: victoriametrics config: {{ provider.my_provider_name }} with: query: {value} start: {value} end: {value} step: {value} queryType: {value}7.2 示例一单指标阈值告警create_alert_from_vm_metric.yml 演示了查询平均 CPU 使用率 → 超过阈值自动创建告警的完整链路# This workflow queries VictoriaMetrics metrics and creates alerts based on CPU usage workflow: # Unique identifier for this workflow id: victoriametrics-cpu-alert # Display name shown in the UI name: VictoriaMetrics CPU Alert # Brief description of what this workflow does description: Monitors CPU usage metrics from VictoriaMetrics and generates alerts based on configurable thresholds. # Define how the workflow is triggered triggers: - type: manual # Can be triggered manually from the UI # Steps to execute in order steps: - name: victoriametrics-step provider: # Use VictoriaMetrics provider config defined in providers.vm config: {{ providers.vm }} type: victoriametrics with: # Query average CPU usage rate query: avg(rate(process_cpu_seconds_total)) queryType: query # Actions to take based on the query results actions: - name: create-alert provider: type: keep with: # Create alert if CPU usage exceeds threshold if: {{ value.1 }} 0.0040 alert: name: High CPU Usage description: [Single] CPU usage is high on the VM (created from VM metric) # Set severity based on CPU usage thresholds severity: {{ value.1 }} 0.9 ? critical : {{ value.1 }} 0.7 ? warning : info # Alert labels for filtering and routing labels: environment: production app: myapp service: api team: devops owner: alice关键点瞬时查询结果结构为data.result[0].value[1]时间戳、指标值因此{{ value.1 }}即指标数值if条件与severity中的三元表达式共同构成分级告警逻辑CPU 使用率 0.9 为 critical、 0.7 为 warning否则为 info。7.3 示例二多服务批量告警按标签去重create_multi_alert_from_vm_metric.yml 展示了基于sum(...) by (job)的多指标场景将每个 job 单独生成一条告警workflow: # Unique identifier for this workflow id: multi-service-cpu-monitor # Display name shown in the UI name: Multi-Service CPU Monitor # Brief description of what this workflow does description: Creates separate alerts for different services based on VictoriaMetrics CPU metrics with customizable thresholds. triggers: # This workflow can be triggered manually from the UI - type: manual steps: # Query VictoriaMetrics for CPU metrics - name: victoriametrics-step provider: # Use the VictoriaMetrics provider configuration config: {{ providers.vm }} type: victoriametrics with: # Query that returns the sum of CPU usage for each job # Example response: # [ # {metric: {job: victoriametrics}, value: [1737808021, 0.022633333333333307]}, # {metric: {job: vmagent}, value: [1737808021, 0.009299999999999998]} # ] query: sum(rate(process_cpu_seconds_total)) by (job) queryType: query actions: # Create an alert in Keep based on the query results - name: create-alert provider: type: keep with: # Only create alert if CPU usage is above threshold if: {{ value.1 }} 0.01 # Alert must persist for 1 minute for: 1m # Use job label to create unique fingerprint for each alert fingerprint_fields: - labels.job alert: # Alert name includes the specific job name: High CPU Usage on {{ metric.job }} description: CPU usage is high on the VM (created from VM metric) # Set severity based on CPU usage thresholds: # 0.9 critical # 0.7 warning # else info severity: {{ value.1 }} 0.9 ? critical : {{ value.1 }} 0.7 ? warning : info labels: # Job label is required for alert fingerprinting job: {{ metric.job }} # Additional context labels environment: production app: myapp service: api team: devops owner: alice该示例的关键技巧通过{{ metric.job }}将查询结果的 job 标签写入告警名称与标签并通过fingerprint_fields: [labels.job]保证每个 job 生成独立的告警指纹避免多条服务告警被合并for: 1m要求指标持续超阈值 1 分钟才真正告警有效抑制抖动。7.4 示例三阈值判断 多渠道通知query_victoriametrics.yml 演示了将查询结果与threshold条件类型结合同时向 Slack 与 Ntfy 推送通知workflow: id: victoriametrics-threshold-monitor name: VictoriaMetrics Threshold Monitor description: Monitors VictoriaMetrics metrics with threshold-based alerts, sending notifications to both Slack and Ntfy. triggers: - type: manual steps: - name: victoriametrics-step provider: config: {{ providers.victoriametrics }} type: victoriametrics with: query: avg(rate(process_cpu_seconds_total)) queryType: query actions: - name: trigger-slack1 condition: - name: threshold-condition type: threshold value: {{ steps.victoriametrics-step.results.data.result.0.value.1 }} compare_to: 0.0050 alias: A compare_type: gt provider: type: slack config: {{ providers.slack }} with: message: Result: {{ steps.victoriametrics-step.results.data.result.0.value.1 }} is greater than 0.0040! - name: trigger-slack2 if: {{ A }} provider: type: slack config: {{ providers.slack }} with: message: Result: {{ steps.victoriametrics-step.results.data.result.0.value.1 }} is greater than 0.0040! - name: trigger-ntfy if: {{ A }} provider: type: ntfy config: {{ providers.ntfy }} with: message: Result: {{ steps.victoriametrics-step.results.data.result.0.value.1 }} is greater than 0.0040! topic: ezhil这里展示了两种条件复用方式在 action 内直接内联threshold条件compare_type: gt且compare_to: 0.0050并为条件设置别名A后续 action 通过if: {{ A }}引用该条件结果避免重复计算。八、Provider 自测与调试若需脱离 UI 单独验证 Provider 的连通性与查询能力可直接运行源码自测入口victoriametrics_provider.py# 默认使用 http://localhost:8880 拉取 VMAlert 告警 python keep/providers/victoriametrics_provider/victoriametrics_provider.py脚本逻辑分别以 VMAlert 与 VM Backend 两组配置实例化 Providerprovider.get_alerts()拉取告警provider.query(queryavg(rate(process_cpu_seconds_total)), queryTypequery)执行一次瞬时查询并打印结果。端点与凭据可通过VMALERT_HOST、VMALERT_USER、VMALERT_PASSWORD、VMBACKEND_HOST、VMBACKEND_USER、VMBACKEND_PASSWORD环境变量覆盖便于在不修改任何代码的前提下针对实际环境做连通性自检。九、总结Keep 与 VictoriaMetrics 的集成为监控告警体系提供了完整的存储 — 评估 — 告警 — 处置闭环部署沿用官方 docker-compose 全家桶victoriametrics / vmagent / vmalert / alertmanager / grafana仅需将 Grafana 端口从3000调整为3001即可与 Keep 共存接入Provider 支持 VMAlert 与 VM Backend 双端点模型告警可经 Alertmanager Webhook 推送带api_keyBasic 认证也可由 Keep 主动调用/api/v1/alerts拉取查询工作流中以 step 形式执行query/query_range两类 PromQL 查询结果经{{ value.1 }}、{{ metric.job }}等模板变量参与阈值判断处置结合 keep provider 创建分级告警、fingerprint_fields去重、for抑制抖动并可联动 Slack、Ntfy 等通知渠道。如需进一步深入可阅读 victoriametrics_provider.py 源码、官方文档 docs/providers/documentation/victoriametrics-provider.mdx、自动生成配置片段 docs/snippets/providers/victoriametrics-snippet-autogenerated.mdx以及 examples/workflows 目录下的完整可运行工作流。【免费下载链接】keepThe open-source AIOps and alert management platform项目地址: https://gitcode.com/GitHub_Trending/kee/keep创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考