I recently came cross an issue where Splunk stopped alerting, honestly, no idea why.. Still investigating. But in the interim I wanted to get some additional context for the investigation. I will update this post once I have more details but I ended up building this dashboard that shows the past 7 days of alerting with some different contextual views with a timepicker defaulted to last 7 days:
- Total Alerts Sent (count)
- Alerts Sent (timechart)
- Alerts Sent by Host (timechart)
- Alerts Sent by App (timechart)
- Alerts Sent by Search (timechart)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
<form theme="light">
<label>Splunk Alerting Dashboard</label>
<description>Show me the alerts</description>
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>-7d@h</earliest>
<latest>now</latest>
</default>
</input>
</fieldset>
<row>
<panel id="TotalAlertsSent">
<title>Total Alerts Sent</title>
<html depends="$hiddenForCSS$">
<style>
#TotalAlertsSent{width: 20% !important;}
</style>
</html>
<single>
<search>
<query>index=_internal sourcetype=splunkd component=sendmodalert | stats count</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</single>
</panel>
<panel id="AlertsSentChart">
<title>Alerts Sent</title>
<html depends="$hiddenForCSS$">
<style>
#AlertsSentChart{width: 80% !important;}
</style>
</html>
<chart>
<search>
<query>index=_internal sourcetype=splunkd component=sendmodalert | timechart count</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
</search>
<option name="charting.axisTitleX.visibility">collapsed</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
<option name="charting.legend.placement">none</option>
<option name="height">155</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
<row>
<panel>
<title>Alerts Sent by Host</title>
<chart>
<search>
<query>index=_internal sourcetype=splunkd component=sendmodalert
| timechart count by host</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
<option name="charting.legend.placement">bottom</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
<row>
<panel>
<title>Alerts Sent by App</title>
<chart>
<search>
<query>index=_internal sourcetype=splunkd component=sendmodalert
| where isnotnull(app)
| timechart count by app</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
<option name="charting.legend.placement">bottom</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
<row>
<panel>
<title>Alerts Sent by Search</title>
<chart>
<search>
<query>index=_internal sourcetype=splunkd component=sendmodalert
| where isnotnull(search)
| timechart count by search useother=0</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
</form>