Grafana

Receive alerts from Grafana unified alerting and legacy alerts

Overview

Grafana is a popular open-source platform for monitoring and observability. Easyalert supports both Grafana Unified Alerting (8+) and Legacy Alerting (7.x) formats.

ℹ️ Info: Grafana 8+ uses Unified Alerting by default. Legacy alerting is deprecated but still supported for older installations.

Requirements

Grafana 8.0+ (for Unified Alerting) or Grafana 7.x (Legacy)
Easyalert account and active tenant
Admin or Editor permissions in Grafana

Setup Instructions

Step 1: Create Integration in Easyalert

Go to Integrations page from left menu
Click Add Integration button
Select Grafana as Source Type
Enter a name (e.g., Grafana Production)
Click Create to save
Copy the generated Webhook URL

Example: https://api.easyalert.io/api/v1/webhooks/ingest/wh_abc123...

Step 2: Create Contact Point (Unified Alerting)

Go to Alerting → Contact points
Click Add contact point button
Fill in the form:

Field	Value
Name	`Easyalert`
Integration	Select `Webhook`
URL	Your copied Webhook URL
HTTP Method	`POST`

Click Test button to verify connection
Click Save contact point

Step 3: Configure Notification Policy

Go to Alerting → Notification policies
Edit Default policy or add New nested policy:

Field	Value
Contact point	Select `Easyalert`
Group by	`alertname`, `severity`
Group wait	`30s`
Group interval	`5m`
Repeat interval	`4h`

Step 4: Test the Integration

Use the Test button in the contact point configuration to send a test alert.

Legacy Alerting Configuration (Grafana 7.x)

For Grafana 7.x and earlier with legacy alerting:

Create Notification Channel

Go to Alerting → Notification channels
Click Add channel
Select webhook as the type

Configure Webhook

Name: Easyalert
URL: Your Easyalert webhook URL
HTTP Method: POST
Send on all alerts: Yes
Include image: Optional

Creating Alert Rules

Labels you define in alert rules become tags in Easyalert.

Add Labels (IMPORTANT)

In Section 3: Add annotations and labels, add labels for routing:

Label	Value	Description
`severity`	`warning` or `critical`	Alert severity
`customer`	`AcmeCorp`	Customer name (for MSPs)
`team`	`backend`	Responsible team
`environment`	`production`	Environment

Add Annotations

Annotation	Example Value
`summary`	`High CPU usage on {{ $labels.instance }}`
`description`	`CPU usage is {{ $values.A }}%`

Field Mapping

Unified Alerting

Grafana Field	Easyalert Field
`labels.alertname`	Title
`annotations.summary`	Title (fallback)
`annotations.description`	Description
`status`	Status (firing → problem, resolved → ok)
`labels.severity`	Severity
`labels.instance`	Host
`labels.job`	Service
`fingerprint`	Event ID

Legacy Alerting

Grafana Field	Easyalert Field
`ruleName`	Title
`message`	Description
`state`	Status
`ruleUrl`	URL
`evalMatches`	Metric details

Severity Mapping

Grafana Severity	Easyalert Severity
critical	Critical
error	High
warning	Warning
info	Info

Legacy State Mapping

Grafana State	Easyalert Status
alerting	Problem
ok	OK
pending	Problem (warning)
no_data	Problem (warning)

Label → Tag Conversion

All labels from Grafana are automatically available as tags in Easyalert.

Example Conversion

Labels in Alert Rule:

alertname = HighCPUUsage
severity = critical
customer = acme
team = backend
environment = production

Available Tags in Easyalert:

tags.alertname = "HighCPUUsage"
tags.severity = "critical"
tags.customer = "acme"
tags.team = "backend"
tags.environment = "production"

Routing Examples

Escalation Routing:

tags.customer equals "acme" → Acme Corp Policy
tags.team equals "database" → DBA Team Policy
tags.environment equals "production" → Critical Policy

Notification Rules:

tags.severity equals "critical" → call + sms + email
tags.severity equals "warning" → sms + email

Fingerprint and Duplicate Detection

Grafana generates a unique fingerprint for each alert. Easyalert uses this fingerprint to:

Not create new incident if same alert comes again
Auto-close incident when resolved status arrives

Example Alert Rules

CPU Alert

# Query (Prometheus)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Condition
IS ABOVE 80

# Labels
severity: warning
team: infra
customer: acme

# Annotations
summary: High CPU usage on {{ $labels.instance }}

Memory Alert

# Query
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Condition
IS ABOVE 85

# Labels
severity: critical
team: infra

Grafana-Specific Features

Dashboard Links

Grafana includes helpful links in alerts:

Field	Description
`dashboardURL`	Link to the dashboard
`panelURL`	Direct link to the panel
`silenceURL`	Quick link to silence the alert
`generatorURL`	Link to the alert rule

These are captured and displayed in the incident details.

Test

curl -X POST "YOUR_WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -d '{
    "receiver": "Easyalert",
    "status": "firing",
    "orgId": 1,
    "alerts": [{
      "status": "firing",
      "labels": {
        "alertname": "TestAlert",
        "severity": "warning",
        "customer": "test",
        "team": "backend"
      },
      "annotations": {
        "summary": "Test Alert",
        "description": "This is a test"
      },
      "startsAt": "2024-12-03T10:00:00Z",
      "fingerprint": "test-123"
    }],
    "externalURL": "http://grafana:3000"
  }'

Troubleshooting

Alerts not being received

Test the contact point from Grafana UI
Check Grafana server logs for webhook errors
Verify network connectivity to Easyalert
Ensure notification policy routes to correct contact point

Labels not becoming tags

Check Labels section in alert rule
Verify labels are properly defined
Review webhook samples in Easyalert

Resolved alerts not working

Verify "Disable Resolved Message" is NOT checked in contact point
Check notification policy includes resolved alerts
Verify alert rule has proper conditions for resolution

Missing dashboard links

Ensure externalURL is configured in Grafana
Check Grafana root URL settings

Legacy vs Unified format issues

Check which alerting system is enabled in Grafana
Easyalert auto-detects format, but verify payload structure
Review webhook samples for actual format being sent

Best Practices

Use Unified Alerting: Migrate to Unified Alerting for better features and consistency with Prometheus/Alertmanager format.

Include Runbook URLs: Add runbook_url annotation to help responders quickly find remediation steps.

Organize with Folders: Use Grafana folders to organize alerts and include folder name in labels.

Configure External URL: Set Grafana's external URL properly so dashboard links work correctly.

Use Labels for Routing: Add labels like customer, team, environment for escalation routing in Easyalert.