Condor#

Introduction#

Condor provides the batch computing platform used in the HEP compute cluster.#

Installation#

Condor is installed using puppet (like everything else). Puppet module for condor :#

   puppet:/etc/puppet/modules/condor/

There is a version lock on every machine that controls what versions are installed :#

   puppet:/etc/puppet/modules/condor/install_condor.pp
   # look for condor_lock

Condor restarts itself automatically when it notices new binaries. On worker nodes, Condor’s configured MaxJobRetirementTime will be honored when restarting due to an upgrade. This means it will wait for CMS jobs to complete or for the time job time limit (2 days) to expire.#

Configuration#

Configuration files are in :#

   puppet:/etc/puppet/modules/condor/files

puppet copies these files to /etc/condor/ on each machine. All nodes read condor_config and host.local. Some machines, via a line in their local file, read glow_config and/or 00hep_wisc.config. Similarly, some read /cms/sw/job_router/etc/routing_config.host (directly read from AFS, unlike other config files).#

How are specific types of grid jobs directed to specific slots? Take cmssoft jobs as an example. In cmsgrid02.local, there is a line that adds requirements to all grid jobs so that if it is a job owned by osg_cmssoft, it will only run on slots advertising IsCMSSoftVM:#

APPEND_REQ_VANILLA = \
  ((($(APPEND_REQ_VANILLA)) && MY.Owner != "osg_cmssoft") || \
  ( MY.Owner == "osg_cmssoft" && (TARGET.IsCMSSoftVM =?= True)))

Then in the host.local files for g9n42 and g9n43, these hosts are set up to advertise IsCMSSoftVM and to require that only cmssoft jobs may run there:#

IsGeneralPurposeVM = False
IsGeneralPurposeSlot = False
IsCMSSoftVM = True
IsDedicated = True

STARTD_EXPRS = $(STARTD_EXPRS), IsGeneralPurposeVM, IsGeneralPurposeSlot, IsCMSSoftVM

START =(MY.IsCMSSoftVM =!= True || TARGET.Owner == "osg_cmssoft") && ($(START))

Therefore, to find cmssoft machines, you could grep for IsCMSSoft in /afs/hep/condor/etc or you could do this:#

condor_status -p condor.hep.wisc.edu -constraint 'IsCMSSoftVM'

Adjusting Priorities#

To adjust the relative priority of CMS users, edit the following file:#

puppet:/etc/puppet/modules/condor/files/conf/cms_common_group_prio_adjust.mult

That adjusts priorities in both GLOW and CHTC (after about an hour). To adjust priorities differently in one of these pools, edit the following files:#

puppet:/etc/puppet/modules/condor/files/conf/cms_glow_group_prio_adjust.mult
puppet:/etc/puppet/modules/condor/files/conf/cms_chtc_group_prio_adjust.mult

Pools#

Some machines are configured to be in the GLOW pool (condor.hep.wisc.edu) and some are configured to be in the HEP condor pool (condor.hep.wisc.edu). Here is how to query the status of the various pools we care about:#

# GLOW pool
condor_status -p glow.cs.wisc.edu
# HEP condor pool
condor_status -p condor.hep.wisc.edu
# CHTC
condor_status -p cm.chtc.wisc.edu

Monitoring#

Cron#

/afs/hep.wisc.edu/home/mbanderson/bin/updateCondorGlobalGraph.pl condor_q#

The following scripts are run by cron at regular intervals:#

  • /cms/cmsprod/condor/bin/glow_machine_hist_update_cron
  • /cms/cmsprod/condor/bin/glow_machine_hist_daily_cron
  • /cms/cmsprod/bin/condor_usage_rrd_cron

On nodes where CMS users submit jobs that we want to show up in gratia reporting, we are running the Gratia condor probe. It is installed by /cms/sw/gratia/bin/install_gratia, which is run by cfengine. A cron job /cms/sw/gratia/bin/update_vomap is run as cmsprod on cumin to periodically update the list of CMS users. Condor is configured to work with the gratia probe by setting:#

PER_JOB_HISTORY_DIR=/opt/vdt/gratia/var/data

Nagios#

Troubleshooting#

Symptom: Jobs run once but then got back to an idle state and do not run.#

One thing that can cause this is the updated ImageSize in the job’s ClassAd being so large that the job’s memory requirements are no longer met by any machines. The default memory requirements are TARGET.Memory * 1024 >= MY.ImageSize. This is problematic, since ImageSize (total virtual memory footprint) can be an overestimate of the job’s actual physical RAM requirements. A temporary solution is to use condor_qedit to change the ImageSize. A better solution is to submit the jobs with better memory requirements. Referencing TARGET.Memory in the job requirements will prevent the default requirements from being inserted.#

Managing#

How to turn off condor#

Useful Commands#

List T2 machines that have not not been running T2 jobs for over an hour.#

condor_status -const 'CurrentRank <= 0 && Cpus>0 && IsDedicated && CurrentTime-EnteredCurrentState>3600'

Show who has claimed the slot in the query above:#

condor_status -const 'CurrentRank <= 0 && Cpus>0 && IsDedicated && CurrentTime-EnteredCurrentState>3600' -af Name -af RemoteOwner

List T2 machines that are not advertising healthy CVMFS.#

condor_status -const '!CMS_CVMFS_Exists && IsDedicated'

Show mappings of account usernames to x509 subject names and VOMS attributes on the CE:#

condor_history -af owner -af x509UserProxySubject -af x509UserProxyFirstFQAN

Show names of users running jobs on the cluster:#

condor_ce_status -af GLIDEIN_SiteWMS_Queue -af GLIDEIN_SiteWMS_JobId -af GLIDEIN_SiteWMS_Slot -af x509userproxysubject | awk '{print $5, $6, $7}' | sort -u

Show user names in the PEM certificate files in the Condor job directories on the worker nodes:#

# $1 -- node list file, $2 -- Name to scan for
# Can use Subject: for $2 to get all users
ssh-pwrd-many --hosts="`cat $1`" --command="sudo find /var/condor/execute -maxdepth 5 \( -name 'x509up*' -o -regex '/.*/[0-9a-f]+' -o -name myproxy.pem \) | xargs -n1 --replace=XX sudo openssl x509 -text -noout -in XX " | grep $2

References#