{"id":68,"date":"2026-10-08T22:35:32","date_gmt":"2026-10-08T22:35:32","guid":{"rendered":"https:\/\/www.hep.wisc.edu\/cms-comp\/?page_id=68"},"modified":"2026-10-08T22:58:59","modified_gmt":"2026-10-08T22:58:59","slug":"faq","status":"publish","type":"page","link":"https:\/\/www.hep.wisc.edu\/cms-comp\/userdoc\/faq\/","title":{"rendered":"CMS T2 Frequently Asked Questions"},"content":{"rendered":"<h1 style=\"margin-top:var(--wp--preset--spacing--50);margin-bottom:var(--wp--preset--spacing--20)\" class=\"is-style-mini-bar wp-block-post-title\">CMS T2 Frequently Asked Questions<\/h1>\n\n\n<div class=\"wp-block-group has-contrast-light-background-color has-background is-layout-flow wp-container-core-group-is-layout-7ddb81f3 wp-block-group-is-layout-flow\" style=\"border-radius:0.5rem;padding-top:var(--wp--preset--spacing--40);padding-right:var(--wp--preset--spacing--40);padding-bottom:var(--wp--preset--spacing--40);padding-left:var(--wp--preset--spacing--40);box-shadow:none\">\n<h2 class=\"wp-block-heading is-style-default has-large-font-size\">On this page<\/h2>\n\n\n<ul style=\"grid-template-columns:1fr 1fr\" class=\"group-of-links-spacing-condensed wp-block-uw-blocks-group-of-links\"  >\n\t\t\n\n<li class=\"wp-block-uw-blocks-uw-link\">\n\t<a target=\"_self\"\n\t href=\"#HDFS\">\n\t\t<span class=\"uw-link-text-wrapper\">\n\t\t\t\t\t<span class=\"uw-link-text\">HDFS Data Storage<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n<\/li>\n\n\n\n<li class=\"wp-block-uw-blocks-uw-link\">\n\t<a target=\"_self\"\n\t href=\"#farmout\">\n\t\t<span class=\"uw-link-text-wrapper\">\n\t\t\t\t\t<span class=\"uw-link-text\">Farmout Scripts<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n<\/li>\n\n\n\n<li class=\"wp-block-uw-blocks-uw-link\">\n\t<a target=\"_self\"\n\t href=\"#condor\">\n\t\t<span class=\"uw-link-text-wrapper\">\n\t\t\t\t\t<span class=\"uw-link-text\">Condor<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n<\/li>\n\n\n\n<li class=\"wp-block-uw-blocks-uw-link\">\n\t<a target=\"_self\"\n\t href=\"#misc\">\n\t\t<span class=\"uw-link-text-wrapper\">\n\t\t\t\t\t<span class=\"uw-link-text\">Miscelaneous<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n<\/li>\n\n<\/ul>\n<\/div>\n\n\n\n<h1 id=\"hdfs\" class=\"wp-block-heading\">HDFS Data Storage<\/h1>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How can I get a copy of the data files?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You must first find out the path to your files in UW-HEP HDFS.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you are trying to copy your files to another CMS site (e.g. Fermilab), then see <a href=\"#access-data-from-outside\">Can I access data files from outside the hep.wisc.edu domain?<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To do a bulk copy of all of the files in a dataset to a local disk, you can simply use a command such as the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cp -r \/hdfs\/store\/... destination-directory\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This should work on any machine with \/hdfs mounted (e.g. the login machines).<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 id=\"access-data-from-outside\" class=\"wp-block-heading\">Can I access data files from outside the hep.wisc.edu domain?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Yes. All data files are globally readable through several protocols: davs, xrootd, gsiftp.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to get a local copy of data files from UW-HEP, you just need a list of filenames. Once you have that, you may copy them.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Example using xrootd to copy files<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Copy data located at Wisconsin using local Xrootd redirector or manager (root:\/\/cmsxrootd.hep.wisc.edu)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>xrdcp root:\/\/cmsxrootd.hep.wisc.edu\/\/store\/... .\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Copy data located at Wisconsin using global(US) Xrootd redirector or manager (root:\/\/cmsxrootd.fnal.gov)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>xrdcp root:\/\/cmsxrootd.fnal.gov\/\/store\/... .\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Example using gfal-copy<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>gfal-copy -p -r davs:\/\/cmsxrootd.hep.wisc.edu:1094\/store\/user\/<em>name<\/em>\/src_file dest_filename\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 id=\"copy-to-hdfs\" class=\"wp-block-heading\">How can I copy files into HDFS from a local disk?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">One way to copy files into HDFS is with gfal-copy. First create a grid proxy with voms-proxy-init and then use gfal-copy to copy the file. Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gfal-copy -p \/path\/to\/src_file davs:\/\/cmsxrootd.hep.wisc.edu:1094\/store\/user\/path\/to\/dest_file\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How can I manage my files in HDFS?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">On machines with \/hdfs mounted (e.g. login machines), you should be able to manage files in HDFS using standard unix commands such as <code>rm, mv, mkdir, rmdir<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You should be able to cd to \/hdfs\/store\/<em>user<\/em> and manage your files. Be very careful when doing recursive rm operations! There is no way to recover if you remove files by mistake.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">File management can also be done via gfal-* commands (details below).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Using gfal commands (examples)<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">File Copy<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>gfal-copy davs:\/\/cmsxrootd.hep.wisc.edu:1094\/store\/user\/<em>name<\/em>\/src_file dest_filename\n gfal-copy -p -r \/path\/to\/src_file davs:\/\/cmsxrootd.hep.wisc.edu:1094\/store\/user\/path\/to\/dest_file\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">File list<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>gfal-ls -l davs:\/\/cmsxrootd.hep.wisc.edu:1094\/store\/user\/your_dir\/your_file\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">File remove<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>gfal-rm davs:\/\/cmsxrootd.hep.wisc.edu:1094\/store\/user\/your_dir\/your_file\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Using xrdfs commands<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">File management can also be done via xrdfs. This can be used as a command-line tool, like gfal-ls. It can also be used as an interactive shell.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">File list<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>xrdfs cmsxrootd.hep.wisc.edu ls \/store\/user\/example\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">File remove<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>xrdfs cmsxrootd.hep.wisc.edu rm \/store\/user\/example\/file\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Interactive shell<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>xrdfs cmsxrootd.hep.wisc.edu\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 id=\"open-root\" class=\"wp-block-heading\">How can I open HDFS files directly from root?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you know the \/store path to a file, you can have root read from the file directly by specifying <code>root:\/\/cmsxrootd.hep.wisc.edu\/\/store\/...<\/code> as the file name that you give to root.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How can I verify that our copy of a file is valid?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Calculate the CRC checksum of a file and compare it to that listed in DAS. For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cksum \/hdfs\/store\/data\/Run2010B\/Electron\/AOD\/Dec22ReReco_v1\/0002\/3861EBF4-2C0E-E011-A891-0018F3D0960E.root\n2443048860 2448344328 \/hdfs\/store\/data\/Run2010B\/Electron\/AOD\/Dec22ReReco_v1\/0002\/3861EBF4-2C0E-E011-A891-0018F3D0960E.root\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The first number is the CRC. Compare this result to that returned by DAS when you click on \u201cshow\u201d.<\/p>\n\n\n\n<h1 id=\"farmout\" class=\"wp-block-heading\">The \u201cfarmout\u201d Scripts<\/h1>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How can I use farmoutRandomSeedJobs to submit CMSSW MC jobs to Condor?<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>farmoutRandomSeedJobs jobName nEvents nEventsPerJob \/path\/to\/CMSSW \/path\/to\/configTemplate cmsRunArg1=value1 cmsRunArg2=value2 ...\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Use the -h option to see all of the options.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Either the configuration file or the cmsRun arguments must contain the macros <code>$randomNumber<\/code>, <code>$nEventsPerJob<\/code>, and <code>$outputFileName<\/code>. These macros are filled in for each job by the farmout script. Additional random numbers may be generated per job by using the macros <code>$randomNumber1<\/code>, <code>$randomNumber2<\/code>, and so on.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The following example assumes a configuration .py file that uses the standard <code>FWCore.ParameterSet.VarParsing<\/code> module. In addition to the standard options, it assumes an option named <code>randomSeed<\/code> is used. Since all required macros are specified in the command-line options, no macros are required in the .py file, so it can just be a standard .py file that you could use when running cmsRun interactively.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>farmoutRandomSeedJobs \\\n  jobName \\\n  nEvents \\\n  nEventsPerJob \\\n  \/path\/to\/CMSSW \\\n  \/path\/to\/config.py \\\n  'outputFile=$outputFileName' \\\n  'maxEvents=$nEventsPerJob' \\\n  'randomSeed=$randomNumber'\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Notice that the $ character must be protected from being interpreted specially by the shell, so in the above example, the options containing macros are surrounded by single quotes.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How can I use farmoutAnalysisJobs to submit CMSSW analysis jobs to Condor?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This script will run cmsRun root files in a directory or directory tree. By default, it runs on all root files in a directory in your \/hdfs area, using the jobName that you specify to find the files. However, you can direct it to an alternate path and tell it to exclude root files with names matching a pattern that you specify.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For full options to the script, use the <code>-h<\/code> option. Here is a brief synopsis:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>farmoutAnalysisJobs &#91;options] jobName \/path\/to\/CMSSW \/path\/to\/configTemplate cmsRunArg1=value1 cmsRunArg2=value2 ...\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Either the configuration file or the cmsRun arguments must contain the macros <code>$outputFileName<\/code> and <code>$inputFileNames<\/code>. These macros are filled in for each job by the farmout script.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The following example assumes a configuration .py file that uses the standard <code>FWCore.ParameterSet.VarParsing<\/code> module. Since all required macros are specified in the command-line options, no macros are required in the .py file, so it can just be a standard .py file that you could use when running cmsRun interactively.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>farmoutAnalysisJobs &#91;options] jobName \/path\/to\/CMSSW \/path\/to\/config.py 'outputFile=$outputFileName' 'inputFiles=$inputFileNames'\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Notice that the $ character must be protected from being interpreted specially by the shell, so in the above example, the options containing macros are surrounded by single quotes.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How can I use farmoutAnalysisJobs to merge together analysis output?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>--merge<\/code> option to farmoutAnalysisJobs may be used to merge a set of small files into one or more larger files. To use this option, invoke farmoutAnalysisJobs as though you were using the root files to be merged as input files. Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>farmoutAnalysisJobs \\\n   --merge \\\n   --input-files-per-job=50 \\\n   --input-dir=\/store\/user\/dan\/QCD-Trigger \\\n   example-merge-job-name \\\n   ~\/CMSSW_4_2_2_patch1\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Notice that <code>--input-files-per-job<\/code> must be specified. Choose a value that will produce reasonably sized merged output files. Aim for merge files of a few gigs. If they are larger than 10GB, your merge job may run out of space on the worker node. If there are more input files than the number you specify, then multiple merge jobs will be created, and you will end up with multiple merge files.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By default, mergeFiles.C is used to merge the files. If you wish to instead use the root hadd utility, specify <code>--use-hadd<\/code>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 id=\"fwklite\" class=\"wp-block-heading\">How can I submit framework-lite or other types of jobs using farmoutAnalysisJobs?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In addition to submitting standard CMSSW analysis jobs, farmoutAnalysisJobs can submit framework-lite or other jobs. The mechanism for submitting generic jobs (i.e. non CMSSW jobs) is to use the <code>--fwklite<\/code> option to farmoutAnalysisJobs. It\u2019s more generic than the name implies. Any sort of script can be run through this mechanism, not just framework-lite jobs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When you use that option, instead of supplying a cmsRun configuration template to farmoutAnalysisJobs, you supply an executable script. Note that farmout does <em>not<\/em> replace macros in your script or make any modifications to it whatsoever. Instead, the environment variables <code>INPUT<\/code> and <code>OUTPUT<\/code> are used to pass information to the script. <code>INPUT<\/code> gives the path to a file which contains the list of input files (one per line). <code>OUTPUT<\/code> specifies the path to the output file that the job is expected to produce. This output file will be stored in HDFS, just like output files from cmsRun jobs. In addition to those two environment variables, command-line options may be passed to the script. The same macro-substitutions are done in the command-line options as for CMSSW analysis jobs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As a convenience, if the \u201cscript\u201d is a .C or .cc file, it is executed like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root -q -b filename.C\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Otherwise, you can provide a shell script or python script or whatever is most convenient to start the job. All that matters is that it can be executed and that it exits with 0 status when all goes well. If the script relies on additional files, you can transfer those using <code>--extra-inputs<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How can I test my jobs before submitting them?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">It is important to test your jobs before submitting a large number of them to the cluster \u2013 if they fail, many hours of computation can be wasted in a short period of time. Very short and lightweight tests may be run on the <a href=\"#login\">login<\/a> machines. Longer or more intense tests can be run interactively in the cluster using interactive jobs. This allows you to test your job in an environment identical to our production Condor cluster. To submit a UI job, use the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>farmoutInteractiveJob\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 id=\"condor\" class=\"wp-block-heading\">Condor Batch System<\/h1>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How do I submit a generic job to Condor at the Wisconsin Tier-2?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">See <a href=\"\/cms-comp\/userdoc\/using-cmst2#condor\">UW-HEP Condor User Info<\/a>. Also see <a href=\"#fwklite\">farmout \u2013fwklite option<\/a>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Why are my jobs idle?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Jobs submitted to Condor at the Wisconsin Tier-2 may run on resources distributed across the campus grid. It can take a few minutes for the Condor negotiator to come around to your newly submitted job and try finding a machine to run it on. If no machines are immediately available, the job waits in the idle state (\u2018I\u2019 in the <code>condor_q<\/code> output).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To see how many machines could possibly run your job, you can use the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>condor_q -analyze &lt;jobid&gt;\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If your job requirements do not match very many machines, you can try to analyze the requirements:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>condor_q -better-analyze &lt;jobid&gt;\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It may happen that your urgent jobs have no problem matching the requirements of lots of machines, but they are still idle due to machines being busy with other jobs. In this case, let us know and we can see if a priority adjustment would help.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The above <code>condor_q<\/code> commands only analyze the resources available in the CMS Tier-2. When the Tier-2 resources are all busy, your jobs may run (via Condor flocking) in another pool of resources on campus. These pools can be analyzed using commands such as the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>condor_q -pool cm.chtc.wisc.edu -analyze &lt;jobid&gt;\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Why do my jobs get held?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Jobs may enter the \u2018held\u2019 state for a number of reasons. To find out why your jobs are held, check them with condor_q:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>condor_q -format '%s\\n' HoldReason &lt;jobid&gt;\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This will print a message explaining why your job was held. One common reason looks like the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>The job attribute PeriodicHold expression 'ImageSize \/ 1024 &gt; 4.000000 * 900 || DiskUsage \/ 1024 &gt; 10.000000 * 2000' evaluated to TRUE\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This message means that your job\u2019s memory or disk usage exceeded the allowable threshold. To see which attribute may have caused the job to become held, use condor_q again:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>condor_q -format 'Memory:\\t%s\\n' ImageSize -format 'Disk:\\t%s\\n' DiskUsage &lt;jobid&gt;\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then, compare the resulting numbers to the thresholds in the PeriodicHold expression above. If you\u2019re running CMSSW jobs and find that they are getting held because their ImageSize has exceeded the threshold, consider reducing the number of events per job or searching your code for possible memory leaks. If your jobs require more memory than is typically available on our cluster, you can explicitly set a higher memory requirement to limit your jobs to machines that have enough RAM (where N is the expected number of megabytes of RAM needed): requirements = (TARGET.Memory &gt;= N) If too few machines have enough memory to run your jobs, we may be able to specially configure some of them so that your jobs can run.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">What operating system will my job run in and how can I control that?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You can see which operating systems are installed on the cluster by using the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>condor_status -af OpSysAndVer | sort | uniq -c\ncondor_status -pool cm.chtc.wisc.edu -af OpSysAndVer | sort | uniq -c\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If your job runs on an operating system that is different from the one it was compiled for, it may experience problems such as missing libraries.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can control which OS your job runs on using the requirements expression in your condor submit file. Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>requirements = TARGET.OpSysAndVer == \"CentOS7\"\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">or<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>requirements = TARGET.OpSysAndVer == \"AlmaLinux9\" || TARGET.OpSysAndVer == \"CentOS9\"\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you are using the farmout scripts, the <code>--opsys<\/code> command-line option is used.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An alternative to restricting the OS is to run the job in a container, so that it sees the same OS environment no matter what the host computer is running.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you are using the farmout scripts, the <code>--use-singularity<\/code> command-line option is used.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you are writing your own condor submit file, you may specify a singularity image. Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>container_image = \/cvmfs\/singularity.opensciencegrid.org\/cmssw\/cms:rhel7\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">However, the above option may not work with jobs that use x509 proxy certificates. Therefore, you may need to invoke singularity (aka apptainer) directly in your job\u2019s script in order to control what gets mounted in the container. Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>singularity exec -B \/cvmfs -B `pwd` -B \/etc\/grid-security\/certificates --no-home \/cvmfs\/singularity.opensciencegrid.org\/cmssw\/cms:rhel7 COMMAND-TO-EXECUTE\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">There are some complications that can result in the absolute path to the proxy certificate still not being accessible within a container running with the above options. The following shell code can be used near the top of your job script to re-execute itself inside singularity. It mounts CVMFS, trusted CAs, and the working directory of the job. The reason both <code>top_dir<\/code> and <code>scratch_dir<\/code> are considered for mounting is to handle cases where one path contains a symlink to the other.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CONTAINER=\/cvmfs\/singularity.opensciencegrid.org\/cmssw\/cms:rhel7\n\nif &#91; \"$STARTED_SINGULARITY\" = \"\" ]; then\n  export STARTED_SINGULARITY=1\n  top_dir=$(pwd -P | sed 's|\\(\/&#91;^\/]*\\).*|\\1|')\n  MOUNTS=\"-B $top_dir\"\n  scratch_dir=$(echo $_CONDOR_SCRATCH_DIR | sed 's|\\(\/&#91;^\/]*\\).*|\\1|')\n  if &#91; \"$scratch_dir\" != \"\" ] &amp;&amp; &#91; \"$scratch_dir\" != \"$top_dir\" ]; then\n      MOUNTS=\"$MOUNTS -B $scratch_dir\"\n  fi\n  if &#91; -d \/hdfs ]; then\n    MOUNTS=\"$MOUNTS -B \/hdfs\"\n  fi\n  MOUNTS=\"$MOUNTS -B \/cvmfs\"\n  MOUNTS=\"$MOUNTS -B \/cvmfs\/grid.cern.ch\/etc\/grid-security:\/etc\/grid-security\"\n  this_script=.\/$(basename $0)\n  export SINGULARITY_CACHEDIR=\"$(pwd)\/singularity\"\n  exec singularity exec --no-home $MOUNTS $CONTAINER $this_script \"$@\"\n  echo \"Failed to execute singularity: $?\"\n  exit 1\nfi\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">When invoking singularity yourself rather than using the <code>container_image<\/code> submit file command, you should also add <code>HasSingularity<\/code> to the job requirements expression. Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>requirements = HasSingularity\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How can I debug my running jobs?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you need to interactively debug a running job, you can ssh to it using the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>condor_ssh_to_job jobid\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A message will inform you of the PID of the job. This is likely just a shell script, not the actual cmsRun process. To find the cmsRun process, you can use the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ps auwwx --forest | less\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Find the PID mentioned by condor in the process tree. You should then see cmsRun as a child of that process.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Another way to see your job\u2019s process tree is with the following command, substituting PID with the PID specified by Condor:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pstree -pa PID\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Once you know the PID of the process you wish to debug, you can attach with a debugger or use gstack to get a stack dump. Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gstack PID\ngdb -p PID\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">What does this WARNING (File \/afs\/blah\/blah.out is not writable by condor) mean?<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>WARNING: File \/afs\/hep.wisc.edu\/user\/blah\/blah.out is not writable by condor.\nWARNING: File \/afs\/hep.wisc.edu\/user\/blah\/blah.error is not writable by condor.\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The above indicates that your job is trying to access AFS. Remove all references to AFS files and directories from your job. AFS is not accessible to Condor jobs.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">How do I restrict my job to run on only the machines with \/hdfs mounted?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you are reading files via xrootd, your job can run anywhere and access the data. If instead you are reading files via \/hdfs, be aware that not all machines at UW have \/hdfs mounted.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The requirements expression for restricting your job to run on machines that have \/hdfs mounted is this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>requirements = TARGET.HAS_CMS_HDFS\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is automatically inserted into the job requirements for farmout jobs using the <code>--use-hdfs<\/code> option.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Should I use condor file transfer for reading from\/writing to hdfs ?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The simple answer is \u201cNO\u201d. We don\u2019t recommend using condor file transfer for reading\/writing to hdfs. The reason is that condor file transfer happens through the submit machine where the job was submitted. If many jobs are reading\/writing to hdfs at the same time, this creates a bottleneck. Since files can be read\/written to hdfs using highly scalable protocols such as xrootd, there is no need to be limited by this bottleneck.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So the best option is to use the script that runs your job copy all the output root files into hdfs using gfal-copy. This requires that you submit your job with a voms proxy so it can authenticate to hdfs. <a href=\"#copy-to-hdfs\">Example of how to use gfal-copy<\/a>.<\/p>\n\n\n\n<h1 id=\"misc\" class=\"wp-block-heading\">General<\/h1>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 id=\"login\" class=\"wp-block-heading\">What can I do on the <code>login.hep.wisc.edu<\/code> machines?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We provide a pool of machines for interactive use. See <a href=\"\/cms-comp\/userdoc\/using-cmst2\">Using CMS T2<\/a> and <a href=\"\/cms-comp\/userdoc\/cms-policies\">CMS User Policies<\/a>. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How can I protect files stored in AFS?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If your home directory is in AFS, then <code>echo $HOME<\/code> will look like <code>\/afs\/hep.wisc.edu\/home\/username<\/code>. AFS permissions are different from a normal Linux filesystem. Default permissions have been set up for you, but some understanding is good to have, since your home directory may contain several sensitive files, including your grid certificate private key or SSH private keys. These files must be protected from unauthorized access, though you may still allow unauthenticated users to view other files in your home directory. For each of the directories that contain sensitive files, create a private subdirectory and apply restrictive AFS ACLs to it. For example, to protect a Globus grid certificate private key:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cd ~\/.globus\nmkdir .private\nchmod 700 .private\nfs sa .private $USER rlidkwa -clear\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then, move each sensitive file into the <code>.private<\/code> directory and create a symlink back:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ mv userkey.pem .private\n$ ln -s .private\/userkey.pem userkey.pem\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Files that should be protected in this manner include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>~\/.globus\/userkey.pem<\/code><\/li>\n\n\n\n<li><code>~\/.ssh\/id_rsa<\/code><\/li>\n\n\n\n<li><code>~\/.ssh\/id_dsa<\/code><\/li>\n\n\n\n<li><code>~\/.ssh\/identity<\/code><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In addition, several directories may contain private files and should therefore be restricted in their entirety:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>~\/.mozilla<\/code><\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Contacts For Help<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Email: <a href=\"mailto:help@hep.wisc.edu\">help@hep.wisc.edu<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>On this page HDFS Data Storage How can I get a copy of the data files? You must first find out the path to your files in UW-HEP HDFS. If you are trying to copy your files to another CMS site (e.g. Fermilab), then see Can I access data files from outside the hep.wisc.edu domain?. &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":27,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_uw_migration_status":"in-progress","_uw_gutenberg_post_content_before_migration":"","_uw_seo_meta_title":"","_uw_seo_meta_description":"","_uw_seo_twitter_card_type":"summary_large_image","_uw_seo_meta_image":"","_uw_seo_meta_image_url":"","_uw_seo_meta_image_sizes":[],"_uw_seo_custom_meta_tags":[],"footnotes":""},"class_list":["post-68","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/68","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/comments?post=68"}],"version-history":[{"count":5,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/68\/revisions"}],"predecessor-version":[{"id":86,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/68\/revisions\/86"}],"up":[{"embeddable":true,"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/pages\/27"}],"wp:attachment":[{"href":"https:\/\/www.hep.wisc.edu\/cms-comp\/wp-json\/wp\/v2\/media?parent=68"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}