Prometheus

Receive alerts from Prometheus Alertmanager

Overview

Prometheus is a leading open-source monitoring and alerting toolkit. Easyalert integrates with Prometheus via Alertmanager webhooks, receiving alert notifications when your alerting rules fire.

ℹ️ Info: This integration receives webhooks from Alertmanager, not directly from Prometheus. Make sure you have Alertmanager configured.

Requirements

Prometheus and AlertManager installed
Easyalert account and active tenant
Access to AlertManager configuration file

Setup Instructions

Step 1: Create Integration in Easyalert

Go to Integrations page from left menu
Click Add Integration button
Select Prometheus as Source Type
Enter a name (e.g., Kubernetes Prometheus)
Click Create to save
Copy the generated Webhook URL

Example: https://api.easyalert.io/api/v1/webhooks/ingest/wh_abc123...

Step 2: Configure Alertmanager

Edit your alertmanager.yml file:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'easyalert'

receivers:
  - name: 'easyalert'
    webhook_configs:
      - url: 'YOUR_WEBHOOK_URL_HERE'
        send_resolved: true
        http_config:
          follow_redirects: true

Step 3: Reload Alertmanager

Apply the configuration:

curl -X POST http://alertmanager:9093/-/reload

Step 4: Test the Integration

Trigger a test alert and verify it appears in Easyalert.

Alertmanager Configuration

Route Parameters

Parameter	Description	Recommended Value
`group_by`	Labels to group by	`['alertname', 'severity']`
`group_wait`	Wait time after first alert	`30s`
`group_interval`	Interval for adding new alerts to group	`5m`
`repeat_interval`	Resend interval for same alert	`4h`

Multi-Tenant Configuration (for MSPs)

Route different customers to different webhooks:

route:
  receiver: "default"
  routes:
    - match:
        customer: "acme"
      receiver: "acme-easyalert"
    - match:
        customer: "beta"
      receiver: "beta-easyalert"

receivers:
  - name: "acme-easyalert"
    webhook_configs:
      - url: "https://api.easyalert.io/api/v1/webhooks/ingest/wh_acme..."
        send_resolved: true

  - name: "beta-easyalert"
    webhook_configs:
      - url: "https://api.easyalert.io/api/v1/webhooks/ingest/wh_beta..."
        send_resolved: true

Field Mapping

Easyalert automatically maps Alertmanager fields:

Alertmanager Field	Easyalert Field
`labels.alertname`	Title
`annotations.summary`	Title (fallback)
`annotations.description`	Description
`status`	Status (firing → problem, resolved → ok)
`labels.severity`	Severity
`labels.instance`	Host
`labels.job`	Service
`fingerprint`	Event ID
All labels	Tags

Severity Mapping

Prometheus Label	Easyalert Severity
`critical`	`critical`
`error`	`high`
`warning`	`warning`
`info`	`info`

Status Handling

Alertmanager Status	Easyalert Status	Action
firing	Problem	Creates/updates incident
resolved	OK	Resolves incident

💡 Tip: Set send_resolved: true in your webhook config to automatically resolve incidents when alerts clear.

Label → Tag Conversion

All labels from AlertManager are automatically converted to tags. This allows you to use any label for routing.

Example Conversion

Labels in alert:

labels:
  alertname: HighCPUUsage
  severity: warning
  customer: acme
  team: backend
  environment: production

Available tags in Easyalert:

tags.alertname = "HighCPUUsage"
tags.severity = "warning"
tags.customer = "acme"
tags.team = "backend"
tags.environment = "production"

Routing Examples

Escalation Routing:

tags.customer equals "acme" → Acme Corp Policy
tags.team equals "database" → DBA Team Policy
tags.environment equals "production" → Critical Policy

Notification Rules:

tags.severity equals "critical" → call + sms + email
tags.environment equals "staging" → email only

Fingerprint and Duplicate Detection

AlertManager generates a unique fingerprint for each alert. Easyalert uses this fingerprint to detect duplicate alerts.

How It Works

When alert arrives, fingerprint is checked
Same fingerprint + firing status = duplicate (skip)
Same fingerprint + resolved status = incident is closed
Different fingerprint = new incident

repeat_interval Setting

repeat_interval determines how often AlertManager resends the same alert:

route:
  repeat_interval: 4h # Resend every 4 hours

Thanks to duplicate detection, even if AlertManager resends the same alert, a new incident won't be created.

Alert Rule Examples

Host/VM Monitoring

groups:
  - name: host-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
          team: infrastructure
          customer: acme
        annotations:
          summary: "High CPU usage: {{ $labels.instance }}"
          description: 'CPU usage above 80% (current: {{ $value | printf "%.1f" }}%)'

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
          team: infrastructure
        annotations:
          summary: "High memory usage: {{ $labels.instance }}"
          description: "Memory usage above 85%"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 10m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Critical disk space: {{ $labels.instance }}"
          description: "Remaining disk space below 15%"

Kubernetes Alerts

groups:
  - name: kubernetes-alerts
    rules:
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Pod crash loop: {{ $labels.namespace }}/{{ $labels.pod }}"
          description: "Pod restarted more than 3 times in last 15 minutes"

      - alert: DeploymentReplicasMismatch
        expr: kube_deployment_status_replicas_available != kube_deployment_spec_replicas
        for: 10m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Deployment replica mismatch: {{ $labels.deployment }}"
          description: "Expected: {{ $value }}, Active: different"

Database Alerts

groups:
  - name: database-alerts
    rules:
      - alert: PostgreSQLDown
        expr: pg_up == 0
        for: 1m
        labels:
          severity: critical
          team: database
        annotations:
          summary: "PostgreSQL down: {{ $labels.instance }}"
          description: "PostgreSQL instance unreachable"

      - alert: PostgreSQLHighConnections
        expr: sum(pg_stat_activity_count) > 100
        for: 5m
        labels:
          severity: warning
          team: database
        annotations:
          summary: "High connection count: {{ $labels.instance }}"
          description: "Active connections: {{ $value }}"

Deployment Examples

Docker Compose

version: "3.8"
services:
  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"

Kubernetes ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    route:
      receiver: 'easyalert'
    receivers:
      - name: 'easyalert'
        webhook_configs:
          - url: 'WEBHOOK_URL'
            send_resolved: true

Troubleshooting

Alerts not being received

Check AlertManager logs for webhook errors
Verify the webhook URL is accessible from Alertmanager
Test connectivity: curl -X POST YOUR_WEBHOOK_URL -d '{}'
Check firewall rules between Alertmanager and internet
Test config with amtool

Labels not becoming tags

Verify alert rules include required labels
Check that labels aren't being dropped by Alertmanager routes
Review group_by settings

Resolved alerts not clearing incidents

Verify send_resolved: true in webhook config
Check that fingerprint matches between firing and resolved
Review Alertmanager routing to ensure resolved alerts reach webhook

Duplicate incidents

Check group_by configuration
Verify fingerprint is consistent
Review group_interval and repeat_interval settings

Connection refused

Check firewall/network settings
Verify AlertManager can reach Easyalert API
Test URL access from AlertManager server

Test Command

# Send test alert to AlertManager
curl -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "customer": "test"
    },
    "annotations": {
      "summary": "Test alert",
      "description": "This is a test alert"
    }
  }]'

Best Practices

Use Consistent Severity Labels: Standardize on severity: critical|warning|info across all alert rules for consistent mapping.

Include Good Annotations: Add meaningful summary and description annotations to help responders understand the alert.

Group Related Alerts: Use group_by to prevent notification storms from related alerts.

Enable Resolved Notifications: Always set send_resolved: true to automatically resolve incidents.

Add Runbook Links: Include runbook_url annotation for quick access to remediation steps.

Use Labels for Routing: Add labels like customer, team, environment for escalation routing in Easyalert.