bash$ nagiosr --all --exclude 'g\dn' # report all HARD CRITICALs excluding compute nodes (gXnYY) Date & Time Down Date & Time Up Description Plugin Output Duration ---------------- ---------------- ------------------------------------------------------ -------- Tue Aug 1 00:38 Tue Aug 1 02:33 noc Load Average 17.50 loadave 1h55m Tue Aug 1 03:40 Tue Aug 1 04:00 caraway Load Average 8.97 loadave 19m59s Tue Aug 1 10:33 Tue Aug 1 10:54 glow-condor Condor Pool 0 nodes 20m28s Tue Aug 1 10:37 Tue Aug 1 13:42 mr1-netbotz Temperature 82 deg F 3h5m Tue Aug 1 11:29 Tue Aug 1 11:34 glow-condor Condor Pool 0 nodes 5m32s Tue Aug 1 11:53 Tue Aug 1 12:12 glow-condor Condor Pool 0 nodes 19m11s Tue Aug 1 11:59 Fri Aug 11 15:27 airport-wesley AirPort Usage snmpget failed 10d3h Tue Aug 1 12:00 Thu Aug 3 09:16 airport-conf-room AirPort Usage snmpget failed 1d21h Tue Aug 1 12:01 Mon Aug 7 09:35 airport-cms-lab AirPort Usage snmpget failed 5d21h Tue Aug 1 12:09 Mon Aug 7 10:34 airport-pheno-grads AirPort Usage snmpget failed 5d22h Tue Aug 1 12:28 Tue Aug 1 12:35 glow-condor Condor Pool 0 nodes 6m37s Tue Aug 1 22:34 Wed Aug 2 02:09 DESY Peering hop is wiscnet.net 3h34m Tue Aug 1 22:35 Wed Aug 2 02:10 Fermi Peering hop is wiscnet.net 3h34m Tue Aug 1 22:35 Wed Aug 2 02:11 SLAC Peering hop is wiscnet.net 3h35m Tue Aug 1 22:36 Wed Aug 2 02:10 CERN Peering hop is wiscnet.net 3h33m Tue Aug 1 22:36 Wed Aug 2 02:10 ESnet Peering hop is wiscnet.net 3h33m Wed Aug 2 03:00 Wed Aug 2 06:05 login01 Load Average 9.49 loadave 3h4m Wed Aug 2 03:18 Wed Aug 2 03:23 Fermi Ping 100% packet loss 4m53s Wed Aug 2 08:45 Thu Aug 17 13:12 CERN Ping 100% packet loss 15d4h Wed Aug 2 10:02 Wed Aug 2 10:21 login03 Load Average 29.89 loadave 19m49s Wed Aug 2 11:13 Wed Aug 2 11:23 login03 Load Average 15.60 loadave 9m59s Wed Aug 2 17:20 Thu Aug 3 10:38 airport-pheno-print AirPort Usage snmpget failed 17h18m Wed Aug 2 17:21 Thu Aug 3 10:35 lp5 HPJetDirect connection timed out 17h13m Wed Aug 2 21:56 Wed Aug 2 22:06 login03 Load Average 5.04 loadave 9m59s Thu Aug 3 02:21 Thu Aug 3 09:25 cs-condor Condor Pool 0 nodes 7h4m Thu Aug 3 04:54 Thu Aug 3 05:56 glow-condor Condor Pool 0 nodes 1h1m Thu Aug 3 06:20 Thu Aug 3 06:35 login03 Load Average 20.94 loadave 15m19s Fri Aug 4 00:57 Wed Aug 9 00:58 cilantro NSR Backups 0 GB, 0 saves 5d0h Fri Aug 4 14:50 Mon Aug 21 11:16 thyme AFS Volume Quotas varsha 93%, wangkai 99% 16d20h Fri Aug 4 19:04 Fri Aug 4 20:19 login02 Load Average 11.94 loadave 1h14m Sun Aug 6 10:16 Sun Aug 6 12:30 Rader-home Ping 100% packet loss 2h14m Sun Aug 6 11:46 Sun Aug 6 11:55 UW-Web Ping 100% packet loss 9m16s Sun Aug 6 12:05 Sun Aug 6 12:56 UW-Web Ping 100% packet loss 50m50s Sun Aug 6 13:21 Sun Aug 6 13:50 UW-Web Ping 100% packet loss 28m46s Sun Aug 6 14:11 Sun Aug 6 14:15 UW-Web Ping 100% packet loss 3m56s Sun Aug 6 16:40 Sun Aug 6 17:59 login02 Load Average 14.83 loadave 1h19m Sun Aug 6 22:35 Sun Aug 6 23:14 ESnet Ping 100% packet loss 39m8s Sun Aug 6 22:37 Sun Aug 6 23:37 SLAC Peering check timed out 59m43s Sun Aug 6 22:39 Sun Aug 6 23:37 ESnet Peering check timed out 57m50s Sun Aug 6 22:47 Mon Aug 7 00:02 login02 Load Average 15.58 loadave 1h14m Sun Aug 6 22:51 Sun Aug 6 23:35 Fermi Peering check timed out 44m8s Mon Aug 7 15:38 Mon Aug 7 16:18 login02 Load Average 11.81 loadave 40m10s Tue Aug 8 14:08 Tue Aug 8 15:03 anise AFS Volume Quotas herndon 96% 55m9s Tue Aug 8 22:08 Tue Aug 8 22:14 Fermi Ping 100% packet loss 5m18s Tue Aug 8 22:09 Tue Aug 8 22:22 Fermi Peering hop is wiscnet.net 12m28s Tue Aug 8 22:10 Tue Aug 8 22:20 DESY Peering hop is wiscnet.net 10m8s Tue Aug 8 22:11 Tue Aug 8 22:22 CERN Peering hop is wiscnet.net 10m8s Tue Aug 8 22:13 Tue Aug 8 22:18 SLAC Peering hop is wiscnet.net 4m49s Tue Aug 8 22:13 Tue Aug 8 22:19 ESnet Peering hop is wiscnet.net 5m18s Wed Aug 9 00:22 Wed Aug 9 00:32 anise Load Average snmpget failed 9m48s Wed Aug 9 04:08 Wed Aug 9 04:18 ginseng Load Average 2.91 loadave 10m2s Thu Aug 10 08:20 Thu Aug 10 08:29 fermi-cmsgrid Grid FTP socket timed out 9m25s Thu Aug 10 11:07 Thu Aug 10 15:07 cilantro NSR Tape 0 GB avail 4h0m Fri Aug 11 01:07 Fri Aug 11 21:07 cilantro NSR Backups 0 GB, 0 saves 19h59m Fri Aug 11 10:06 Fri Aug 11 11:31 glowserv02 SSH socket timed out 1h25m Fri Aug 11 17:44 *Still Critical* caraway CMS DB Web Service connection refused 12d16h Sat Aug 12 01:17 Mon Aug 14 09:17 cilantro NSR Tape 97 GB avail 2d8h Sat Aug 12 10:53 *Still Critical* rosemary AFS /vicepa check timed out 11d23h Sat Aug 12 10:55 Sat Aug 12 10:59 rosemary AFS /vicepd check timed out 3m19s Sat Aug 12 10:56 Sat Aug 12 11:00 rosemary AFS Fileserver Perf can not contact server 4m38s Sat Aug 12 10:56 Sat Aug 12 11:00 rosemary AFS /vicepb check timed out 3m59s Sun Aug 13 19:53 Sun Aug 13 20:12 Fermi Ping 100% packet loss 19m0s Mon Aug 14 00:21 Mon Aug 14 16:19 cilantro NSR Backups 0 GB, 0 saves 15h58m Mon Aug 14 11:44 Mon Aug 14 12:38 Smith-home Ping 100% packet loss 54m6s Mon Aug 14 19:45 Mon Aug 14 20:04 Smith-home Ping 100% packet loss 18m58s Tue Aug 15 03:21 Tue Aug 15 04:00 Smith-home Ping 100% packet loss 38m55s Tue Aug 15 09:49 Tue Aug 15 09:52 login02 Cfengine socket timed out 3m53s Tue Aug 15 09:49 Tue Aug 15 09:53 login02 SSH socket timed out 4m33s Tue Aug 15 09:49 Tue Aug 15 09:54 login02 LIDS snmpget failed 4m48s Tue Aug 15 10:16 Tue Aug 15 10:21 login03 SSH socket timed out 4m13s Tue Aug 15 10:25 Tue Aug 15 10:29 login01 SSH socket timed out 3m53s Tue Aug 15 13:27 Mon Aug 21 11:32 cilantro NSR Tape 99 GB avail 5d22h Tue Aug 15 15:09 Tue Aug 15 15:12 Smith-home Ping 100% packet loss 2m54s Thu Aug 17 13:12 Thu Aug 17 13:42 s5n03 RAID Status snmpget failed 29m21s Thu Aug 17 13:13 Thu Aug 17 13:43 s5n03 /space.1 snmpget failed 30m1s Thu Aug 17 13:14 Thu Aug 17 13:43 s5n03 SSH socket timed out 29m50s Thu Aug 17 13:15 Thu Aug 17 13:41 s5n03 /space.0 snmpget failed 26m1s Thu Aug 17 13:16 Thu Aug 17 13:47 s5n03 Cfengine socket timed out 31m10s Thu Aug 17 13:18 Thu Aug 17 13:55 s5n03 dCache Client 4 instances offline 37m29s Thu Aug 17 18:37 Thu Aug 17 18:47 s5n01 dCache Client 3 instances offline 9m59s Thu Aug 17 19:05 Thu Aug 17 19:15 s5n01 dCache Client 4 instances offline 9m48s Fri Aug 18 02:08 Fri Aug 18 02:18 s5n01 dCache Client 3 instances offline 10m19s Fri Aug 18 08:07 Fri Aug 18 08:17 s5n01 dCache Client 4 instances offline 9m50s Fri Aug 18 09:35 Fri Aug 18 09:44 s5n01 dCache Client 3 instances offline 9m39s Fri Aug 18 10:22 Fri Aug 18 10:27 s5n01 dCache Client one instance offline 4m58s Fri Aug 18 10:34 Fri Aug 18 10:44 s5n01 dCache Client 3 instances offline 9m59s Sat Aug 19 00:46 Sat Aug 19 00:51 s5n01 dCache Client 2 instances offline 4m49s Sat Aug 19 01:18 Sat Aug 19 01:28 s5n01 dCache Client 3 instances offline 9m47s Sat Aug 19 20:31 Sun Aug 20 08:12 cumin RAID Status connect failed 11h40m Sun Aug 20 03:27 Sun Aug 20 03:42 s5n01 dCache Client 3 instances offline 14m58s Sun Aug 20 23:30 Mon Aug 21 08:09 anise AFS Volume Quotas lazaridis 100% 8h39m Mon Aug 21 00:21 Mon Aug 21 11:31 cilantro NSR Backups 0 GB, 0 saves 11h10m Mon Aug 21 15:39 Mon Aug 21 15:45 thyme AFS Fileserver Perf 57 blocked 5m19s Mon Aug 21 16:49 Mon Aug 21 16:54 s5n01 dCache Client 3 instances offline 4m59s Mon Aug 21 18:15 Tue Aug 22 11:06 thyme AFS Volume Quotas yencho 100% 16h50m Tue Aug 22 05:30 Tue Aug 22 05:54 UW-Web Ping 100% packet loss 24m17s Tue Aug 22 06:10 Tue Aug 22 06:19 UW-Web Ping 100% packet loss 8m56s Tue Aug 22 06:25 Tue Aug 22 06:34 UW-Web Ping 100% packet loss 8m45s Tue Aug 22 15:08 Tue Aug 22 16:14 CERN Peering hop is wiscnet.net 1h6m Tue Aug 22 15:09 Tue Aug 22 16:15 ESnet Peering hop is wiscnet.net 1h6m Tue Aug 22 15:13 Tue Aug 22 16:13 Fermi Peering hop is wiscnet.net 59m58s Tue Aug 22 15:13 Tue Aug 22 16:13 DESY Peering hop is wiscnet.net 59m57s Tue Aug 22 15:14 Tue Aug 22 16:14 SLAC Peering hop is wiscnet.net 59m50s Tue Aug 22 15:59 Tue Aug 22 16:04 thyme AFS Volume Quotas yencho 100% 4m59s Tue Aug 22 23:15 Tue Aug 22 23:25 cmsgrid02 RAID Status snmpget failed 9m59s Tue Aug 22 23:16 Tue Aug 22 23:26 cmsgrid02 Grid FTP socket timed out 9m58s Wed Aug 23 16:10 Wed Aug 23 16:40 cmsgrid02 RAID Status snmpget failed 30m0s Thu Aug 24 08:11 *Still Critical* anise AFS Volume Quotas bawa 100% 2h7m