Event Management without good correlation rules is just a more expensive way to flood operators with alerts. The default rules ship as starting points, not endpoints. Real noise reduction comes from correlation tuned to your environment over months. Done well, you can cut alert volume by 80 percent and still catch every real incident.
Start with deduplication, not correlation
Before correlation, ensure raw event dedup is working. Two events from the same source with the same metric and similar timestamps should collapse to one event with a count. The Event Field Mapping rules drive this. Inspect the em_event table for clusters of near-identical events; if you see them, dedup is misconfigured.
Correlate on CI, not on text
Text-based correlation rules (“alerts containing ‘disk full’”) are brittle. CI-based correlation (“alerts on this server, this database, this application”) is durable. Spend the time to ensure events arrive with a populated cmdb_ci field, even if it requires preprocessing. Without CI, correlation is a string-matching exercise.
Use service maps to amplify CI correlation
If a service has 20 nodes and 15 of them alert simultaneously, that is a service-level event, not 15 node-level events. Configure correlation rules that watch for percentage-of-nodes thresholds against service maps and create one parent alert with the children attached. Operators triage one alert instead of 15.
Time windows are sensitivity dials
Correlation rules group events within a time window. Too narrow misses related events; too wide groups unrelated ones. Start at five minutes for most rules, tighter for fast-moving infrastructure (containers), wider for batch systems (overnight ETL). Track the false-merge rate (alerts that should not have been correlated) and adjust weekly.
Topology-aware correlation prevents storms
When a single network device fails, every downstream service alerts. Topology-aware correlation uses the CMDB relationship graph to identify the root cause and demote the downstream alerts to “secondary.” This requires accurate relationships in the CMDB; without that data, topology correlation guesses badly.
Prioritize known patterns
If you know “host down on patch Tuesday between 2am and 4am is expected,” encode it as a correlation rule that auto-acknowledges. If you know “database replica lag over 30 seconds during business hours is a P2,” encode it. The known patterns are the fastest wins; novel events get human attention.
Tune with outcome data, not gut feel
Every two weeks, review the top 50 alert clusters by volume. For each, ask: did this need to be 50 alerts, or could it have been 5? Did the operator action change because of the count? Use the answers to tune correlation rules. Outcome-based tuning beats theoretical tuning every time.
Watch for correlation suppressing real signal
Aggressive correlation can hide real problems behind a single parent alert. If parent alerts go unacknowledged for hours, operators may be missing the children. Build a Performance Analytics indicator for “child alerts on unacknowledged parents” and review weekly.
What to do this week: pull your top 20 alert sources by volume and check that each event arrives with a populated CI; fix any that do not.
