{"id":95,"date":"2026-10-09T13:46:57","date_gmt":"2026-10-09T13:46:57","guid":{"rendered":"https:\/\/www.hep.wisc.edu\/cms-comp\/?page_id=95"},"modified":"2026-10-09T19:46:33","modified_gmt":"2026-10-09T19:46:33","slug":"monitoring","status":"publish","type":"page","link":"https:\/\/www.hep.wisc.edu\/cms-comp\/ops\/monitoring\/","title":{"rendered":"Wisconsin CMS T2 Monitoring"},"content":{"rendered":"<h1 style=\"margin-top:var(--wp--preset--spacing--50);margin-bottom:var(--wp--preset--spacing--20)\" class=\"is-style-mini-bar wp-block-post-title\">Wisconsin CMS T2 Monitoring<\/h1>\n\n\n<h2 class=\"wp-block-heading\">Local Monitoring<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Wisconsin CMS Condor Usage : [<a href=\"https:\/\/app.physics.wisc.edu\/condor-stats\/\">Pie Plots<\/a>] [<a href=\"https:\/\/app.physics.wisc.edu\/condor-stats\/?days=30&amp;pool=UWHEP&amp;resource=CPU&amp;timeseries=1\">Time Series<\/a>] [<a href=\"https:\/\/app.physics.wisc.edu\/condor-stats\/?days=30&amp;pool=UWHEP&amp;resource=CPU&amp;timeseries=1&amp;units=comp_hours_per_hour\">CPU-Hours\/Hour<\/a>]<\/li>\n\n\n\n<li>Condor Queues : [<a href=\"https:\/\/pages.hep.wisc.edu\/cron01\/cmsprod\/condorGlobalJobs.html\">Global<\/a>] [<a href=\"https:\/\/pages.hep.wisc.edu\/cron01\/cmsprod\/condorUsersMonitor\/\">Per User<\/a>]<\/li>\n\n\n\n<li>User Data Storage in T2: [<a href=\"https:\/\/pages.hep.wisc.edu\/cron01\/storeuser\/\">HDFS Usage<\/a>]<\/li>\n\n\n\n<li>Perfsonar: [<a href=\"http:\/\/perfsonar01.hep.wisc.edu\/\">perfsonar01<\/a>] [<a href=\"http:\/\/perfsonar02.hep.wisc.edu\/\">perfsonar02<\/a>] [<a href=\"https:\/\/maddash.aglt2.org\/d\/6848b523-d7a4-58d4-b6ad-1eb3d2dab91f\/us-cms?orgId=1&amp;from=now-24h&amp;to=now&amp;timezone=browser\">perfsonar matrix<\/a>]<\/li>\n\n\n\n<li>UW Network Activity : [<a href=\"http:\/\/stats.net.wisc.edu\/\">CampusNet<\/a>] [<a href=\"https:\/\/stats.net.wisc.edu\/wan-stats.html\">UW Research WAN<\/a>]<\/li>\n\n\n\n<li>Chamberlin RBN Network : [<a href=\"https:\/\/stats.net.wisc.edu\/cgi-bin\/genstatspage.fcgi?rrdlist=4f01fc6668b5f72d2aeba50e78c56354&amp;stack=3&amp;stack_by=Everything&amp;GO=GO\">RBN graphs<\/a>] [<a href=\"https:\/\/stats.net.wisc.edu\/cgi-bin\/gnmis.fcgi?interface_description==RBNCHMBRLN&amp;dataset=%3Dbytes\">RBN Ports<\/a>] [<a href=\"https:\/\/stats.net.wisc.edu\/cgi-bin\/gnmis.fcgi?dataset==bytes&amp;search_term=chmbrln&amp;speed=%3D100000000000\">All 100G Ports<\/a>]<\/li>\n\n\n\n<li>LHCOne Network Monitoring: [<a href=\"https:\/\/my.es.net\/lhcone\/view\/UWMADISON\">MyESNet<\/a>]<\/li>\n\n\n\n<li>Operating Conditions of Compute Rooms: [<a href=\"https:\/\/graphf.hep.wisc.edu\/d\/qc4dEAXGk\/grid-frontpage?orgId=1&amp;viewPanel=33\">Temp<\/a>] [<a href=\"https:\/\/graphf.hep.wisc.edu\/d\/qc4dEAXGk\/grid-frontpage?orgId=1&amp;viewPanel=35\">Flow<\/a>] [<a href=\"https:\/\/graphf.hep.wisc.edu\/d\/uBMXMYrMk\/pods?orgId=1&amp;viewPanel=5\">Power<\/a>]<\/li>\n\n\n\n<li>SSEC (not in Chamberlin; useful for comparison): <a href=\"http:\/\/dryice.ssec.wisc.edu\/\">Water Temp<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Rucio Data Xfers<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>CMS Rucio : [<a href=\"https:\/\/twiki.cern.ch\/twiki\/bin\/viewauth\/CMS\/Rucio\">Documentation<\/a>][<a href=\"https:\/\/twiki.cern.ch\/twiki\/bin\/view\/CMSPublic\/RucioGeneralQuestions\">FAQ<\/a>][<a href=\"https:\/\/twiki.cern.ch\/twiki\/bin\/view\/CMSPublic\/RucioUserDocsLocation\">UserDoc<\/a>]<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Central Monitoring<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>OSG GRACC : [<a href=\"https:\/\/gracc.opensciencegrid.org\/kibana\/app\/kibana#\/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-1w,mode:relative,to:now))&amp;_a=(columns:!(CoreHours),filters:!((%27$$hashKey%27:%27object:8098%27,%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:gracc.osg.summary,key:OIM_Site,negate:!f,value:GLOW),query:(match:(OIM_Site:(query:GLOW,type:phrase)))),(%27$$hashKey%27:%27object:10458%27,%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:gracc.osg.summary,key:VOName,negate:!f,value:cms),query:(match:(VOName:(query:cms,type:phrase))))),index:gracc.osg.summary,interval:d,query:(query_string:(analyze_wildcard:!f,query:%27*%27)),sort:!(EndTime,asc),uiState:(spy:(mode:(fill:!f,name:!n)),vis:(legendOpen:!f)))\">Kibana<\/a>] [<a href=\"https:\/\/gracc.opensciencegrid.org\/dashboard\/db\/probe-status?orgId=1\">Probe Status<\/a>] [<a href=\"https:\/\/gracc.opensciencegrid.org\/dashboard\/db\/pilot-jobs-summary?orgId=1&amp;var-VOName=All&amp;var-Facility=GLOW&amp;var-Facility=GLOW-CMS&amp;var-Facility=GLOW-CONDOR-CE&amp;var-ExitCode=All&amp;var-Probe=All&amp;var-interval=1d\">CE Usage<\/a>] [<a href=\"https:\/\/gracc.opensciencegrid.org\/dashboard\/db\/payload-jobs-summary?var-VOName=All&amp;var-Project=All&amp;var-Facility=All&amp;var-User=All&amp;var-ExitCode=All&amp;var-Probe=condor:login01.hep.wisc.edu&amp;var-Probe=condor:login02.hep.wisc.edu&amp;var-Probe=condor:login03.hep.wisc.edu&amp;var-Probe=condor:login04.hep.wisc.edu&amp;var-Probe=condor:login05.hep.wisc.edu&amp;var-Probe=condor:login06.hep.wisc.edu&amp;var-Probe=condor:uwlogin.cern.ch&amp;var-interval=1d\">login usage<\/a>)]<\/li>\n\n\n\n<li>Frontier\/SQuid : [<a href=\"http:\/\/wlcg-squid-monitor.cern.ch\/snmpstats\/indexcms.html\">Status<\/a>] [<a href=\"https:\/\/twiki.cern.ch\/twiki\/bin\/view\/CMS\/DatabaseAccess\">DatabaseAccess<\/a>]<br><\/li>\n\n\n\n<li>Site Readiness : [<a href=\"https:\/\/cmssst.web.cern.ch\/sitereadiness\/report.html#T2_US_Wisconsin\">det_report<\/a>] [<a href=\"https:\/\/cmssst.web.cern.ch\/sitereadiness\/sum_report.html\">sum_report<\/a>] [<a href=\"https:\/\/cmssst.web.cern.ch\/siteStatus\/summary.html\">sum_stauts<\/a>]<\/li>\n\n\n\n<li>HammerCloud Tests : [<a href=\"http:\/\/hammercloud.cern.ch\/hc\/app\/cms\/joberrors\/\">HC Job Errors<\/a>]<\/li>\n\n\n\n<li>USCMS T2 usage (for production) : [<a href=\"https:\/\/monit-grafana.cern.ch\/d\/fGtAZ-bmz\/cms-t2-facilities-use-cases?orgId=11&amp;var-Type=production&amp;var-Tier=All&amp;var-Country=All&amp;var-Site=T2_US_Caltech&amp;var-Site=T2_US_Florida&amp;var-Site=T2_US_MIT&amp;var-Site=T2_US_Nebraska&amp;var-Site=T2_US_Purdue&amp;var-Site=T2_US_UCSD&amp;var-Site=T2_US_Vanderbilt&amp;var-Site=T2_US_Wisconsin&amp;var-measurement=condor_1h&amp;var-retention_policy=long\">CPU hours<\/a>] [<a href=\"https:\/\/monit-grafana.cern.ch\/d\/mFDKvLPZz\/events-by-site?orgId=11\">Events Processed<\/a>]<\/li>\n\n\n\n<li>CMS Activities : [<a href=\"https:\/\/monit-grafana.cern.ch\/d\/000000628\/cms-job-monitoring-es-agg-data-official?orgId=11&amp;var-group_by=CMS_JobType&amp;var-Tier=T2&amp;var-CMS_WMTool=All&amp;var-CMS_SubmissionTool=All&amp;var-CMS_CampaignType=All&amp;var-Site=T2_US_Wisconsin&amp;var-Type=production&amp;var-CMS_JobType=All&amp;var-CMSPrimaryDataTier=All&amp;var-binning=1h\">Production<\/a>] [<a href=\"https:\/\/monit-grafana.cern.ch\/d\/000000628\/cms-job-monitoring-es-agg-data-official?orgId=11&amp;var-group_by=CMS_SubmissionTool&amp;var-Tier=T2&amp;var-CMS_WMTool=All&amp;var-CMS_SubmissionTool=All&amp;var-CMS_CampaignType=All&amp;var-Site=T2_US_Wisconsin&amp;var-Type=analysis&amp;var-CMS_JobType=Analysis&amp;var-CMSPrimaryDataTier=All&amp;var-binning=1h\">Analysis<\/a>]<\/li>\n\n\n\n<li>CMS Task Monitoring : [<a href=\"https:\/\/monit-grafana.cern.ch\/d\/cmsTMGlobal\/cms-tasks-monitoring-globalview?orgId=11&amp;var-user=All&amp;var-site=T2_US_Wisconsin&amp;var-current_url=%2Fd%2FcmsTMDetail%2Fcms_task_monitoring&amp;var-task=All&amp;from=now-2d&amp;to=now\">User Jobs<\/a>]<\/li>\n\n\n\n<li>WLCG [<a href=\"https:\/\/monit-grafana.cern.ch\/d\/000000425\/default?orgId=20\">Dashboard<\/a>]<\/li>\n\n\n\n<li>APEL Portal (for WLCG Accounting): [<a href=\"https:\/\/accounting.egi.eu\/tier2\/federation\/US-CMS-T2\/normelap_processors\/SITE\/DATE\/2024\/11\/2025\/3\/lhc\/onlyinfrajobs\/undefined\/\">Metrics<\/a>]<\/li>\n\n\n\n<li>T2 Resource Deployment : [<a href=\"https:\/\/twiki.cern.ch\/twiki\/bin\/view\/CMSPublic\/USCMSTier2Deployment\">Status<\/a>]<\/li>\n\n\n\n<li>Xrootd monitoring : [<a href=\"https:\/\/monit-grafana.cern.ch\/d\/000000422\/xrootd-transfers-30-days?orgId=20\">grafana<\/a>]<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Local Monitoring Rucio Data Xfers Central Monitoring<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":93,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_uw_migration_status":"in-progress","_uw_gutenberg_post_content_before_migration":"","_uw_seo_meta_title":"","_uw_seo_meta_description":"","_uw_seo_twitter_card_type":"summary_large_image","_uw_seo_meta_image":"","_uw_seo_meta_image_url":"","_uw_seo_meta_image_sizes":[],"_uw_seo_custom_meta_tags":[],"footnotes":""},"class_list":["post-95","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/95","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/comments?post=95"}],"version-history":[{"count":6,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/95\/revisions"}],"predecessor-version":[{"id":106,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/95\/revisions\/106"}],"up":[{"embeddable":true,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/93"}],"wp:attachment":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/media?parent=95"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}