Monitoring and Alerting Improvement Plan
This document records observations, proposed rules, and partial implementation progress. Its checked and unchecked items represent the state recorded in the source document; they should not be read as a claim that the entire migration was completed. Platform resource identifiers tied to a cloud account are generalized in this English edition.
1. Starting Point and Goal
StablePay already had production SLS logs, APM, traces, and Kubernetes metrics. The main business-alert path was SLS log -> processing -> MetricStore -> SLS alert -> notification target. That path worked, but metric coverage, rule depth, notification ownership, and noise control needed work. The plan aimed to improve those layers and gradually consolidate alerts in Alibaba Cloud Monitoring 2.0, without rebuilding all telemetry collection at once.
The confirmed business metric families included unified_action_outcome, unified_biz_count, and unified_error_count in the production MetricStore. Production SLS had raw logs, metric data, alert history, and ETL logs. The important distinction was that formal business alerts did not all query raw logs: their primary decision often came from metrics, while raw logs were joined only to add a sample message, trace_id, or error detail. A success-rate rule used that metrics-plus-sample pattern; a five-minute system-error burst could be decided directly from unified_error_count{error_class="system"}. The alert history held fields such as alert.alert_name, project, policy IDs, status, and severity, allowing rule inventory to be checked against more than chat messages.
Read-only inspection showed a relatively small, uneven set of formal business alerts, concentrated on success/failure rates, error volume, reason codes, and metric interruption. A May 11 notification sample confirmed success-rate, processing-ratio, and deployment replica alerts had reached production notification channels. SLS business and ACK/infrastructure alerts appeared in the same production group. The sample also contained a stablepay-pre namespace deployment alert, showing that non-production events had entered the production group. Notification delivery existed, but consistent severity, owner, escalation, repeated reminders, and recovery handling had not yet been established.
The observed problems were limited metric and service coverage; too few consistently defined rules; unclear action ownership; and Cloud Monitoring 2.0 not yet serving as the formal unified production alert entry. The intended outcome was reusable core metrics and rules, a traceable rule -> severity -> recipient -> responder chain, and staged consolidation of SLS, ARMS, infrastructure, and custom alerts.
2. Scope and Architecture
The work covered inventory of current metrics, rules, and notification paths; improvement of core metrics and alert rules; notification ownership and actions; and a migration path into Cloud Monitoring 2.0. It did not include a one-shot rewrite, an immediate switch of every alert, or final thresholds for every service before historical traffic and false positives had been assessed.
Business logs
-> SLS log storage
-> log processing and metric extraction
-> MetricStore
-> alert evaluation
-> notification targets
-> Cloud Monitoring 2.0 for consolidated alert management
The processing layer turns logs into stable metrics; the rule layer selects thresholds and trigger windows; notification routing assigns severity, handling, and escalation; the platform layer provides a unified management view. Cloud Monitoring 2.0 was intended to consolidate alerts, not replace all underlying SLS and APM data pipelines.
3. Metric Inventory
The plan organized metrics into three reusable layers:
| Layer | First concerns | Priority |
|---|---|---|
| General service outcomes | Request volume, success/failure rate, error count, latency, metric interruption | First |
| System stability | Pod restarts, Ready state, CPU/memory pressure, OOM, exits, available replicas | First |
| Shared business outcomes | Business/reason codes, upstream/downstream errors, processing ratio, stalled state transitions | Establish pattern, then extend |
Confirmed data sources were SLS business metrics (unified_action_outcome, unified_biz_count, unified_error_count); ARMS/APM service metrics such as request_count, error_count, slow_count, avg_request_latency_seconds, and error_rate; APM exception metrics; Kubernetes pod and state metrics for CPU, memory, restart, Ready and termination; Kubernetes events; and log/trace fields such as trace_id, span_id, biz_code, error, caller, and __topic__.
First Metric and Rule Candidates
| Candidate | Suggested level | Source state in the assessment |
|---|---|---|
| Five-minute success-rate drop | P1 | Existing SLS outcome metric and alert direction |
| Five-minute failure-rate or error-volume rise | P1 | Existing SLS and/or APM data; some rules already present |
| Five-minute average latency rise | P2 | Partly available in APM; rule needed review |
| Five-minute P95 latency rise | P1 | To be added |
| Ten-minute metric interruption | P1 | Existing metric-interruption direction |
| Five-minute abnormal request-volume drop | P1 | To be added |
| Five-minute abnormal Pod restart count | P2 | To be added |
| Pod not Ready, CrashLoop, or abnormal exit | P1 | Some related alert behavior observed |
| Five-minute high CPU or memory working set | P2 | To be added |
| Five-minute OOM risk | P1 | To be added |
| Available replica mismatch | P1 | Related infrastructure alert observed |
Five-minute non-0000 business/reason-code rise | P1 | Existing direction |
| Upstream or system error burst | P1 | Existing rules/directions |
| Processing-state share rise | P2 | Existing rule direction |
| Long-pending or nonterminal share rise | P2 | To be added |
| Abnormal state-transition count | P1 | To be added |
| Dependency failure-rate rise | P1 | Partly available |
| Reconciliation exception rise | P0/P1 | High-value future business metric |
The first version should favor already available sources and cross-service patterns. The aim was to support formal rule construction and migration, rather than decorate dashboards or prematurely define every business-specific metric.
4. Rule Structure and Threshold Review
First improve existing directions: success, failure, system/upstream error, metric interruption, reason-code anomalies, and processing ratio. Then extend the pattern to business-specific signals through service owners.
The proposed organizing model was a shared rule template with a small number of service overrides. A rule_template defines what failure is detected and gives default threshold, duration, reminder interval, and severity. A system_rule_override changes only necessary fields for a particular service, such as threshold, minimum sample size, duration, severity, or notification action. One global rule would be too rigid for critical or low-volume services; full copies for every service would cause rule proliferation.
Thresholds were to be reviewed in batches using history, volume, false positives, and service importance. Customer- or money-impacting outcomes warrant stricter handling; resource and trend signals can be looser. P0/P1/P2/P3 should imply different windows and actions. Once live, thresholds require review against actual alerts. The goal is detection with minimal duplication: keep the rule best expressing user impact; retire tests and longstanding noise; use consistent names and metric semantics; and avoid formal high-priority rules with no response action.
5. Severity, Ownership, and Notification
StablePay retained P0/P1/P2/P3 language internally, while the first Cloud Monitoring 2.0 configuration mapped directly to Alibaba Cloud's four severities:
| Platform severity | Internal meaning | Suggested action |
|---|---|---|
| Critical | P0/P1 | Immediate group notification and on-call/owner response |
| Error | P2 | Group notice and daytime/team handling |
| Warning | P3 | Lower-priority response and review |
| Normal | Observation or very-low-priority item | Dashboard or infrequent notice |
The platform configuration would not force a separate native severity for P0 and P1; any internal distinction could live in labels, names, or response process. High-priority rules need an on-call responder and escalation path; P2 needs a team or default owner; warnings can be reviewed periodically. Cross-team rules need one first responder. Owner mapping could follow initial rule classification and gray rollout, but must be complete before switching a rule into the final primary notification channel.
Two-Track Group Notification
A proof of concept showed the native Feishu alert card could reach the group and preserve ChatOps links for rule, history, claim, close, and mute. Native bot plus contact group did not reliably mention the correct owner in the group. The recommended path therefore had two tracks:
- Alibaba Cloud's native Feishu bot posts the full standard alert card.
- A lightweight webhook route looks up the owner and adds a short
@ ownermessage in the same official group.
The routing layer only receives a webhook, reads system/service, looks up a mapping, and sends the short reminder. It should not duplicate full-card rendering, rule evaluation, a complex deduplication engine, multi-group distribution, or platform state management. A missing owner should produce a fallback message, not drop the alert. Direct messages, acknowledgement buttons, and feedback synchronization could be considered in a later phase.
The proposed owner mapping needed system, optional service_scope, primary name and Feishu open_id, backup name and open_id, team, and enabled status. Formal rules need either a stable owner mapping or an explicit fallback. The PoC acceptance checks were reliable system parsing, correct mapping, successful mention in the same group, fallback on a miss, and enough short-message context to show rule, severity, system, link, and troubleshooting cue.
The source recorded a more concrete progress snapshot as of May 19: native production card bot, custom webhook for lightweight owner mentions, business-result template and notification policy, and the first business-result rule had been created. Stability and core-business templates/policies were still pending. The native card and supplemental webhook were configured as separate notification paths; action integration and escalation policies were not yet configured.
Noise controls proposed short-window grouping of repeat alerts, selective recovery messages, downgrade of unactionable low-priority rules to observation, and restraint on P0/P1 volume. A feedback ledger should classify valid, false, duplicate, insufficient-information, and not-applicable alerts; record rule, service, severity, trigger frequency, false-positive rate, and adjustment; and be reviewed weekly. Low-volume services can receive minimum-sample gates or overrides.
6. Cloud Monitoring 2.0 Migration Boundary
The consolidation target was not an optional research topic, but Cloud Monitoring 2.0 was not yet the production alert hub at the time of the source's initial assessment. Existing SLS log processing, MetricStore, and rules would remain while formal alerts were inventoried and migrated. ARMS, ACK/infrastructure, and other custom alerts would be included progressively.
The sequence was: inventory current formal production rules and their real sources; select a small first batch; define common names, severity, labels, templates, lifecycle, and ownership; create a replacement; run it in gray mode; switch primary notifications after validation; then retire the old rule. At least one overlapping observation period should avoid a detection gap, but duplicate production notifications should not persist after cutover. Test, idle, and noisy rules should be retired or marked as observations instead of migrated automatically.
The first inventory should focus on rules that actually notified production. Its ledger needed old name, source platform, monitored service or infrastructure scope, metric definition, active/test/observation state, current recipient, proposed keep/replace/retire decision, and new-rule mapping. Recent group messages and the alert-history store could cross-check each other. SLS business results and ACK infrastructure were both in the first denominator; stablepay-pre events in the production group made environment scoping a first-order issue. A full historical inventory of all unused rules was outside the first batch.
First Migration Candidates
| Rule family | Original example | Proposed handling |
|---|---|---|
| Business outcome | Five-minute success-rate drop | Rebuild with shared business-result template |
| Business outcome | Five-minute payment failure-rate rise | Rebuild with payment scope and sample gate |
| Business outcome | Five-minute chaincore/channelcore failure-rate rise | Rebuild and reduce repeat noise |
| Business outcome | Five-minute system errors greater than 10 | Rebuild with metric query and contextual sample if useful |
| Stability | Deployment available-replica mismatch | Fix production namespace boundary before formal migration |
| Stability | Pod not healthy | Fix production namespace boundary before formal migration |
Processing-ratio, non-0000 reason count, metric interruption, and upstream-error bursts were cited as examples to clean up or leave as observations when their current action value or noise did not justify formal migration. A specific test rule and generic test alerts also belonged outside the formal denominator. That decision should be recorded per rule rather than inferred from its name alone.
For existing Cloud Monitoring 2.0 items, the plan called for consistent environment, scope, metric meaning, severity, source, owner team, and lifecycle labels. An item with no recent notification, owner, or clear business purpose should become “needs confirmation,” not silently enter the formal count. The migration should consolidate duplicate alerts for the same problem rather than copy old configurations verbatim.
7. Business-Result Query and Rule Template
First-wave SLS business rules should be understood as metric-driven. System-error bursts can be decided directly from metrics. Success and failure-rate rules use stablepay-log-metrics for the primary condition and may join stablepay-log for notification or debugging context. The source recommended establishing the MetricStore query, grouping, and threshold before adding optional raw-log enrichment.
Because query fields become alert payload current_value/data, a formal rule should return:
| Field | Requirement | Reason |
|---|---|---|
service | Required | Stable route key and group display field |
| Primary value | Required | Show the actual success/failure rate or error count |
| Abnormal count | Required | Explain impact and noise |
| Total count | Strongly recommended | Make ratio meaningful and support a minimum-sample gate |
sys_code, biz, reason_code | Recommended | Help locate system, business area, and failure kind |
env, source_platform, alert_domain, and lifecycle can come from rule labels rather than query output. Ratio rules should return numerator and denominator, not only a percentage.
The proposed rule name format was [environment][category][scope] meaning severity, with human-readable Chinese names in the internal console. Examples were [生产][基础通用][默认] 成功率下降 P1 and [生产][系统稳定性][StablePay集群] Pod 不健康 P2. Threshold, window, lifecycle, and migration batch belong in labels or the governance ledger, not the title. An override is distinguished by scope, not by adding “override” to its name.
The template fields included lifecycle (gray, formal, observe, off), native severity, mandatory env, and labels such as alert_domain, source_platform, service_scope, migration_batch, and lifecycle. Required per-rule configuration included source old rule, monitored scope, metric expression or PromQL, severity, minimum sample threshold where needed, duration, repeat interval, notification action, gray observation period (suggested 3–7 days), and explicit old-rule retirement condition.
The first candidates used SLS for four business rules and ACK/Prometheus for two stability rules. Business-result rules initially used Critical severity and gray lifecycle; stability rules were proposed at Error/observation until production-only scoping was proven. As the source progressed, three business-result rules were recorded as created: general success-rate drop, payment failure-rate rise, and system-error burst. The first two had been observed entering the production group through the new path; the new system-error rule still awaited a natural trigger. Deployment-replica and Pod-health rules remained to be built. The business-result notification template existed (1/3 template classes); stability and core-business templates were not created.
Old/new comparison found the new success-rate rule could cover the old five-minute meaning and add service, sys_code, biz, and reason_code for display and routing. The payment-failure old rule had been seen in a payment/refund group, while the new rule reached the StablePay production group. The system-error replacement retained the five-minute value > 10 condition, but its new notification path still needed a natural-trigger check. “Can cover” in this comparison was an assessment, not permission to retire the old alert without gray validation.
8. Migration Acceptance and Action Ledger
The intended acceptance criteria were: a clear inventory of formal rules in Cloud Monitoring 2.0; each rule tied to service scope, severity, recipient, and owner before primary cutover; old SLS/ARMS rules replaced or explicitly retired; test and noisy rules outside the formal denominator; no sustained duplicate production notifications; and the team able to identify the unified production alert entry.
The suggested execution rhythm was to classify seven days of production messages, export current SLS and Cloud Monitoring/ACK configuration into one ledger, mark keep/replace/retire, remove obvious tests and non-production noise, then create templates and replacements. New rules would run in gray mode for three to seven days before primary notification switched, followed by retirement of old rules and another review of duplicates and false positives. In parallel, the owner-routing line needed an official group ID, minimum payload fields (env, system, service, severity, rule_name, source_platform, status), a first owner map, webhook PoC, fallback, and an end-to-end exercise.
The source checklist showed meaningful partial progress:
- Inventory completed in part: production group and recent messages confirmed, mixed SLS/ACK notifications and non-production sample identified, SLS stores and alert-history source checked, first active-rule list and governance ledger created. Full cross-source reverse lookup, including ARMS and idle rules, remained open.
- First rule selection completed: business and stability candidates identified, keep/replace/retire decisions made for the first group, and a V0.1 candidate list produced.
- Templates and gray rollout partial: common naming, severity, and label templates existed and old/new mappings were drafted; the source separately recorded several individual business rules as created. The broader checklist for rebuilding the entire first business and stability batch and finishing gray observation remained unchecked.
- Cutover not completed: primary switch, old-rule retirement, duplicate and non-production rechecks, and first noise review remained unchecked.
- Owner route partial: a lightweight webhook PoC for an added owner mention was checked. Official group ID, minimum payload, owner and backup map, missing-owner fallback, final integration, and end-to-end exercise remained unchecked.
The checklist and individual rule progress use different levels of granularity. An individual “created” rule does not imply the first batch passed gray validation or switched to formal notifications.
9. Open Risks and Expected Outputs
Historical metric data still needed to validate definitions, windows, and formal-alert suitability. Services differed in log semantics and metric completeness, which could limit one template's coverage. Owner mapping could be completed in stages, but a rule without accountable routing should remain gray rather than enter the final primary path. Cloud Monitoring 2.0's source coverage, switch timing, and event-center use also needed decisions.
The Feishu low-code route was treated as optional supporting orchestration. Its ability to look up an owner from system, send a dynamic @ to a member in the same group, and provide fallback/observability required proof. If a critical part failed, the proposed alternative was a minimal self-hosted webhook relay, not forcing an unreliable low-code path into the primary notification role.
Expected artifacts were a layered metric inventory with source and severity; a first formal-rule list distinguishing existing, new, and retired rules; notification/owner/escalation design; a route design for native card plus short owner mention and fallback; and the first Cloud Monitoring 2.0 migration scope and extension path. The source documented partial progress toward these artifacts, not a completed platform-wide migration.