Recurring Alerts

Identify noisy alerts for tuning, suppression rules, and alert optimization.

Overview

The Recurring Alerts page provides a comprehensive view of your most frequently triggered alerts. Use this data to identify candidates for tuning, create suppression rules, and reduce alert fatigue across your team.

Alert Ranking — See which alerts fire most frequently
Trend Analysis — Identify increasing or decreasing patterns
Export & Reports — Export data and send email reports
Quick Actions — Create suppression rules directly

Summary Statistics

Four metrics provide an overview of recurring alert patterns:

Metric	Description
Unique Alerts	Number of distinct alert types
Total Occurrences	Sum of all alert instances
Increasing Trend	Alerts getting worse over time
Avg Alert Share	Average percentage of total each alert represents

ℹ️ High "Increasing Trend" count indicates growing problems that need attention.

Alert List Table

The main table displays detailed information about each recurring alert:

Column	Description
#	Rank by occurrence count
Alert Title	Alert name and associated service
Count	Number of times alert fired
Trend	Increasing, stable, or decreasing
Volume	Percentage of total alerts
MTTA	Average acknowledgment time
MTTR	Average resolution time
Severities	Breakdown by severity level
Actions	Quick action menu

Understanding Trends

Trend	Icon	Meaning
Increasing	🔴 ↑	Alert firing more frequently
Stable	➖	No significant change
Decreasing	🟢 ↓	Alert firing less frequently

⚠️ Alerts with Increasing trends require investigation — they often indicate growing underlying problems.

Filtering and Search

Search

Search across multiple fields: alert title, service name, host names, and custom tags.

Trend Filter

Filter to see only:

All Trends — Show everything
Increasing — Alerts getting worse
Stable — Consistent alerts
Decreasing — Improving alerts

Group by Service

Toggle "Group by Service" to aggregate alerts by service:

Mode	Use Case
Off	See individual alert patterns
On	Identify noisy services overall

💡 Use "Group by Service" to find services that need comprehensive alert review.

Taking Action

From the Actions Menu

For each alert, you can:

View Incidents — Opens the incidents page filtered to this alert title.
Create Suppress Rule — Pre-populates an alert rule with this alert's details.

Export Options

CSV Export

Click the download button to export all filtered data to CSV. Included fields: Rank, Alert Title, Service, Count, Percent of Total, MTTA, MTTR, Severity breakdown, Trend information, Last occurrence, Affected services and hosts.

Email Report

Click the email button to send a report to yourself. The report includes: summary statistics, top 20 recurring alerts, trend highlights, and recommendations.

ℹ️ Reports are sent to your registered email address.

Identifying Tuning Candidates

High-Priority Candidates

Alerts that should be reviewed first:

High Volume, Low Severity

Pattern: Alert fires frequently but is mostly low/medium severity.

Action options: Increase threshold to reduce triggers, convert to informational alert, or suppress during non-business hours.

Increasing Trend

Pattern: Alert firing more frequently over time.

Action options: Investigate root cause of increase, fix underlying issue, or temporarily suppress while fixing.

High MTTR Alerts

Pattern: Alert takes a long time to resolve.

Action options: Create or improve runbook, automate remediation, or review if alert is actionable.

Auto-Resolved Alerts

Pattern: Alert fires and resolves quickly without action.

Action options: Increase alert delay/threshold, convert to warning level, or implement hysteresis.

Creating Effective Suppress Rules

Identify Pattern — Determine what makes this alert non-actionable: specific time windows, certain environments, or below a specific threshold.
Define Conditions — Create precise conditions that match: alert title patterns, source/service, or severity level.
Set Appropriate Action — Choose the right response: Suppress (don't create incident), Reduce Severity (lower priority), or Route Differently (send to different team).
Monitor Results — After implementing, verify: desired alerts are suppressed, important alerts still come through, and overall noise is reduced.

Best Practices

Regular Review Cadence

Schedule weekly or bi-weekly reviews of recurring alerts. Monday morning review of previous week, include in team standup agenda, and track progress on noise reduction.

Start with Top 10

Focus on the top 10 recurring alerts. These represent most of the noise, improvements have the biggest impact, and it's a more manageable scope.

Document Decisions

For each reviewed alert, document: the decision made (tune, suppress, keep as-is), reasoning, expected outcome, and review date.

Measure Improvement

Track metrics over time: total unique alerts, total occurrences, percentage of alerts suppressed, and team feedback on noise levels.

Don't Over-Suppress

Before suppressing, ask: Has this alert ever caught a real issue? Could we miss something important? Is there a better alternative (tuning vs. suppressing)?

Review Suppressions Periodically

Suppressions can become stale. Services change, thresholds should be reconsidered. Set reminders to review suppression rules quarterly.

Common Patterns and Solutions

Disk Space Alerts

Pattern: Frequent disk space warnings that self-resolve.

Solutions: Increase threshold (e.g., 80% → 90%), implement auto-cleanup scripts, add hysteresis (alert only after X minutes), or separate critical partition alerts from non-critical.

Connection Pool Alerts

Pattern: Brief spikes in connection pool usage.

Solutions: Increase pool size if appropriate, add averaging/smoothing to alert, or alert on sustained high usage instead of spikes.

Batch Job Failures

Pattern: Same job fails and succeeds on retry.

Solutions: Improve job retry logic, alert only after N failures, or separate transient vs. persistent failures.

Health Check Flapping

Pattern: Health checks failing/recovering rapidly.

Solutions: Add dead time between alerts, require multiple consecutive failures, or review health check timeout settings.

Deployment Noise

Pattern: Alerts during deployments.

Solutions: Implement deployment windows with suppression, use canary/gradual deployments, or improve deployment health checks.

Pagination and Large Datasets

For organizations with many alerts:

Pagination Controls

Rows per page: 10, 25, 50, or 100
Navigation: Previous/Next page buttons
Position indicator: "1-25 of 150"

Working with Large Lists

Use search and filters to narrow down. Export to CSV for offline analysis. Focus on top performers by incident count. Use "Group by Service" to aggregate.

Troubleshooting

No alerts appearing

Verify incidents exist in the selected date range. Check that incidents have title information. Ensure incidents are assigned to your tenant.

Trend data seems wrong

Trends compare current period to previous period. Short date ranges may show variable trends. Try a longer date range for more accurate trends.

Service grouping not working

Verify incidents have service metadata. Check integration is sending service information. Review alert payload configuration.

Export not including all data

Exports include current filter results. Clear filters to export all data. Maximum export is 100 alerts.

Email report not received

Check spam/junk folders. Verify email address in profile. Contact admin if email delivery issues persist.

URL Parameters

The page supports URL parameters for deep linking:

Parameter	Description	Example
`days`	Date range in days	`?days=14`
`groupByService`	Enable service grouping	`?groupByService=true`
`search`	Pre-fill search	`?search=database`
`trend`	Filter by trend	`?trend=increasing`

💡 Bookmark filtered views for quick access to specific alert categories.

Alert Rules — Create suppression rules → Alert Rules
Alert Analytics — Detailed incident metrics → Alert Analytics
Integrations — Configure alert sources → Integrations

How to detect recurring alerts ?