CRAB#

CRAB at UW-HEP CMS#

This is a simple, bare-bones guide to CRAB.#

Quick Recipe for users#

We assume you have gone through getting a UW-HEP account, a grid certificate, etc…. More info on getting a grid certificate.#

1) Typical setup:
ssh weinberg@login01.hep.wisc.edu
mkdir /scratch/weinberg
cd /scratch/weinberg     # must be in /scratch!
scramv1 project CMSSW CMSSW_3_5_8_patch3
cd CMSSW_3_5_8_patch3/src/
cmsenv
kinit weinberg@CERN.CH
cvs co your/analyzer  # (or create your analyzer)
scramv1 b
cd your/analyzer

2) Create a crab.cfg here.  See example in section below.

4) Load CRAB at wisconsin:
source /cms/sw/CRAB_2_10_2p3/crab.sh

5) Create the jobs
crab -create

6) Submit the jobs
crab -submit

7) wait, then check on your jobs,
crab -status

8) If they're working, the output root files should be in your hdfs directory:
 /hdfs/store/user/weinberg/

9) Optional, if you want the log and err files do
crab -getoutput

10) Optional, if you want the integrated luminosity of the data you ran on
crab -report
# Crab returns something like:
# Luminosity section summary file: /scratch/mbanderson/.../res/lumiSummary.json
# Give that full path to that file to lumiCalc:
lumiCalc.py -c frontier://LumiProd/CMS_LUMI_PROD -i res/lumiSummary.json recorded

For information on creating an analyzer see Creating Analyzer for more information. Or mkedanlzr.#

Figure out what dataset you want to run over using DBS Data Discovery#

# Note: to continue submitting CRAB jobs for a particular WF to Condor. crab -submit 1 -continue crab_0_$DATE_$TIME#

We recommend using the remoteGlidein scheduler. Other options are to use the glidein server and glite.#

Additional information at Running CMSSW code on the Grid using CRAB from FermiLab.#

Example CRAB cfg file#

This is a typical crab.cfg file.#

[CRAB]
jobtype                = cmssw
scheduler              = remoteGlidein
use_server             = 0

[CMSSW]
datasetpath            = /MinBias/Summer09-STARTUP3X_V8K_900GeV-v1/GEN-SIM-RECO
#datasetpath            = /MinimumBias/BeamCommissioning09-Dec19thReReco_336p3_v2/RECO
pset                   = test/SinglePhotonAnalyzer_cfg.py
total_number_of_events = -1
events_per_job         = 40000
output_file            = SinglePhotonAnalyzer.root

[USER]
copy_data              = 1
return_data            = 0
storage_element        = T2_US_Wisconsin
user_remote_dir        = ./
check_user_remote_dir  = 0

[GRID]
rb                     = CERN
proxy_server           = myproxy.cern.ch
virtual_organization   = cms
retry_count            = 0


#se_white_list          = T2_US_Caltech, T2_US_UCSD, T2_US_Wisconsin

Advanced Set-up#

Put the following code anywhere into your .bash_profile:#

# CMSSW & CRAB Version
export CMSSW_VER=CMSSW_3_3_1
export CRAB_VER=CRAB_2_10_2p3

#Load CMSSW environment settings
cd ${CMSSW_VER}/src
cmsenv
cd ~

#Load CRAB environment
source /cms/sw/${CRAB_VER}/crab.sh

Now, whenver you login, you can just#

cd /path/to/your/analysis/workdir

# Check Jobs status in local Condor queue.
condor_q

# Check Job status in CRAB
crab -status

# Retrieve the output
crab -getoutput

For additional information on how to use CRAB, please read the Fermilab CRAB Instructions.#

CRAB at CERN#

# Login to CERN
ssh user@lxplus.cern.ch

# Load the CMS Environment
source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.csh

# Start your grid proxy
voms-proxy-init -voms cms

# Load up CRAB
source /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_1_1/crab.csh

If your grid-proxy isn’t installed at CERN but it is at Wisconsin, copy it this way:#

# Login to CERN
ssh user@lxplus.cern.ch

# Copy your grid proxy
scp -r user@login.hep.wisc.edu:~/.globus .globus

# Set the permissions on these files
chmod 700 ~/.globus/private/
fs setacl -dir ~/.globus/private -acl system:anyuser none
chmod 0600 ~/.globus/private/userkey.pem
chmod 640 ~/.globus/usercert.pem

Data Location#

If you would like to see where your data is located, use Data Aggregation System. A list of sites is kept too.#

Troubleshooting#

# To see a list of all your jobs
condor_q $USER

# To watch your jobs
jobWatch

# If some jobs go on hold
# See the hold reason
jobWhyHold

# Find out more info about a job
#  jobID = the number on the left when you type condor_q
grid-job-remote-info <jobID>

# Grab the stdout file from a job still running
#  run the above command to find the location of stdout
globus-job-run <site-you-submitted-to> /bin/cat /location/of/stdout

globus-job-run osg-gw-4.t2.ucsd.edu /bin/cat /osglocal/users/cms/uscms1370/.globus/job/osg-gw-4.t2.ucsd.edu/22350.1211409217/stdout

CRAB Installation/Update#

  1. Download the latest CRAB package from here#

  2. Take the tarball to /afs/hep.wisc.edu/cms/sw/ directory and untar it.#

  3. Go to the untar'ed CRAB directory and then run “./configure”. This will create the files “crab.©sh” there.#

  4. Run the standard tests (i.e. create/submit/verify jobs) with the new package.#

  5. Update the line (export CRAB_VER=CRAB_1_5_3) in the “Recipe for Crab …” section below with the new CRAB version.#

  6. Send an email to CMS users list announcing about the new package availability.#

References#