Automating SLA Breach Alerts for Multi‑Region APIs with Lescopr

Marketing team discussing marketing terms in front of a white board with post-its
A step‑by‑step guide that shows how to build an automated SLA breach detection pipeline for multi‑region APIs using Lescopr dashboards, from design to production.

Introduction

Ensuring that a distributed API meets its Service Level Agreement (SLA) across several geographic regions is a constant challenge for SRE teams. A single region can slip under the latency or error‑rate threshold, yet the overall average may still look acceptable. This guide walks you through a complete project—from initial design to production launch—that automates SLA breach detection using Lescopr dashboards. By the end you will have a repeatable pipeline that raises alerts the moment any region violates the agreed SLA, letting you react before customers feel the impact.

1. Project Planning and Requirements

1.1 Define SLA Metrics

  • Latency: 95th‑percentile response time ≤ 250 ms per region.
  • Error Rate: ≤ 0.1 % HTTP 5xx responses per region.
  • Availability: ≥ 99.9 % uptime per region.

These metrics become the basis for the detection logic. Lescopr’s Metric Explorer lets you create region‑specific views, which we will later embed in a dashboard.

1.2 Choose the Monitoring Stack

We built the pipeline on top of:

  1. FastAPI services deployed in three AWS regions (us‑east‑1, eu‑central‑1, ap‑southeast‑2).
  2. OpenTelemetry instrumentation sending traces and metrics to Lescopr.
  3. LesCopr Alerting Engine for rule evaluation.

1.3 Outline Milestones

Milestone Deliverable Owner
M1 – Metric collection OpenTelemetry agents configured DevOps
M2 – Dashboard prototype Lescopr dashboard with regional panels SRE
M3 – Alert rule definition SLA breach rule in Lescopr SRE
M4 – Automation script Webhook that creates tickets Engineer
M5 – Production rollout Canary deployment + monitoring Team

2. Implementing Metric Collection

2.1 Instrument FastAPI with OpenTelemetry

Add the following snippet to each service:

from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.exporter.lesCopr import LesCoprSpanExporter

app = FastAPI()
FastAPIInstrumentor().instrument_app(app)

exporter = LesCoprSpanExporter(endpoint="https://api.lescopr.com")
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(exporter)
)

The exporter forwards latency, error counts, and custom attributes (including the region tag) to Lescopr in real time.

2.2 Verify Data in Lescopr

Navigate to Metrics → Explorer, filter by region label, and confirm that latency percentiles and error rates appear for each deployment. This verification step ensures the data pipeline is healthy before we move on to alerting.

3. Building the SLA Dashboard

3.1 Create Regional Panels

  1. Open Dashboards → New.
  2. Add a Line chart for the 95th‑percentile latency, filtered by region = us-east-1.
  3. Duplicate the chart for eu-central-1 and ap-southeast-2.
  4. Add a Bar chart for error‑rate percentages, again scoped per region.

3.2 Combine Panels into a Single View

Arrange the three latency charts side‑by‑side and place the error‑rate bar chart below them. Name the dashboard Multi‑Region SLA Overview.

3.3 Enable Real‑Time Refresh

Set the refresh interval to 30 seconds so that any breach becomes visible instantly. This real‑time view is the visual backbone of the automated detection system.

4. Defining Automated Alert Rules

4.1 SLA Breach Logic

In Alerts → New Rule, use the following expression (LesCopr uses a PromQL‑like syntax):

max_over_time(latency_p95{region!=""}[5m]) > 250

This rule triggers when the 95th‑percentile latency for any region exceeds 250 ms for five consecutive minutes.

4.2 Notification Channels

Configure two channels:

  • Webhook: Sends a JSON payload to an internal incident‑management service.
  • Slack: Posts a concise alert to the #sre‑alerts channel.

Both channels include the region label, allowing downstream automation to act on the specific offending region.

5. Automating Incident Creation

5.1 Webhook Payload Example

{
  "title": "SLA Breach Detected",
  "region": "{{labels.region}}",
  "metric": "latency_p95",
  "value": "{{value}}",
  "threshold": 250,
  "timestamp": "{{timestamp}}"
}

5.2 Script to Open a Ticket

A small Python script consumes the webhook and creates a ticket in the internal system:

import requests, json

def create_ticket(payload):
    ticket = {
        "summary": f"SLA breach in {payload['region']}",
        "description": (
            f"Latency {payload['metric']} reached {payload['value']} ms, "
            f"exceeding the 250 ms threshold."
        ),
        "priority": "high"
    }
    requests.post("https://tickets.internal/api/create", json=ticket)

# Flask endpoint to receive webhook
from flask import Flask, request
app = Flask(__name__)

@app.route('/lescopr-webhook', methods=['POST'])
def webhook():
    data = request.json
    create_ticket(data)
    return "OK", 200

Deploy this service in a single region; its low latency ensures the alert‑to‑ticket conversion happens within seconds.

6. Testing the End‑to‑End Flow

6.1 Simulate a Breach

Use a load‑testing tool (e.g., k6) to artificially increase response times for the eu-central-1 region:

k6 run --vus 50 --duration 2m script.js

Watch the Lescopr dashboard; the latency chart should cross the 250 ms line. After five minutes, the alert fires, the webhook triggers, and a ticket appears in the incident system.

6.2 Verify Alert Noise

Run the same test with a brief spike (under five minutes). No alert should be generated, confirming that the rule’s time window filters out transient glitches.

7. Production Rollout and Monitoring

7.1 Canary Deployment

Deploy the new instrumentation to 10 % of traffic in each region. Verify that metrics flow correctly and that alerts fire only when expected.

7.2 Full Rollout

After successful canary validation, enable the instrumentation for 100 % of traffic. Keep the dashboard and alert rules active, but add a silence window during scheduled maintenance to avoid false positives.

7.3 Ongoing Tuning

Periodically review the 5‑minute window and the 250 ms threshold. Adjust them based on evolving performance baselines and business requirements.

Conclusion

By following this step‑by‑step project—from metric collection to automated ticket creation—you can reliably detect SLA breaches for multi‑region APIs the moment they occur. The combination of Lescopr’s real‑time dashboards, flexible alert engine, and webhook integration turns raw latency data into actionable incidents, reducing MTTR and protecting your service‑level commitments.

To go further, Lescopr's documentation covers step‑by‑step setup.