Using CMS Tier-2 Computing Center

On this page

Access

Access to the computing cluster is available through Jupyterhub, SSH, remote desktop, and vnc.

Jupyterhub

See the documentation for the Jupyterhub-based CMS Analysis Facility.

SSH

A pool of login computers is available with the address login.hep.wisc.edu. When you don’t care which one you connect to, use the name login.hep.wisc.edu.

ssh login.hep.wisc.edu

The processes you run and jobs you submit to condor are on a specific computer, so make note of what computer you were using and return to that specifically to manage the processes or jobs. Example:

ssh login01.hep.wisc.edu

xrdp

To use xrdp, a graphical Linux interface, you will need a Windows Remote Desktop application on your computer. On a Windows computer, it is already installed. On a mac, you can install Windows Remote Desktop from App Store. Any program that is compatible with the Windows Remote Desktop may be used. There are several options that may be used in Linux.

Before connecting to xrdp on the login.hep.wisc.edu computers, you will need to either connect to the wisc.edu network using WiscVPN (aka Global Protect) or use ssh port forwarding. WiscVPN access requires a UW NetID.

To use ssh port forwarding, you can use the following ssh command, replacing USER and login01 as appropriate:

ssh -L 10000:localhost:3389 USER@login01.hep.wisc.edu

This connects arbitrarily chosen port 10000 on your computer to the xrdp service port 3389 on login01 for the duration of the ssh connection.

Once either WiscVPN or ssh port forwarding to port 3389 has been connected, open Windows Remote Desktop or equivalent. The first time, you will need to add an entry for the target host. If you are using WiscVPN, enter the PC name as login01.hep.wisc.edu (replacing 01 with the number of the specific host). If you are using ssh port forwarding, enter the PC name as localhost:10000.

When you connect via Windows Remote Desktop, it may warn you that the identity of the host is not verified. This is expected, so you may tell it to continue. Enter your hep.wisc.edu username and password. Then you should see a graphical desktop where you can start terminals and run other programs.

You may disconnect and reconnect to the same session if you need to leave something running. However, beware that if your home directory is in AFS rather than CEPH, your AFS token may expire if you have a very long running session.

vnc

Another way to use graphical programs is to run a VNC server on a login.hep.wisc.edu computer and then connect to that via a VNC client. One advantage of this over X forwarding is that, like xrdp, you can disconnect your computer and later reconnect to resume where you left off.

To start the VNC server, run the following command on the login.hep.wisc.edu computer:

vncserver -geometry 1024x768

To connect to the VNC server using a VNC client running on the login computer, run the following command in an ssh session with X forwarding, replacing 1 with the display number assigned when vncserver was started:

vncviewer login01:1

To stop the VNC server, use the following command, replacing 1 with the display number assigned when vncserver was started:

vncserver -kill :1

Rather than running vncviewer on the login computer and viewing it over X forwarding, performance may be improved by running vncviewer on your computer and using ssh port forwarding to allow it to connect to the VNC server. To do so, you will need to know the port number being used by the server. One way to find it is to look in the log file for a line of the form Listening for VNC connections on TCP port N. The path to the log file is displayed when the server is started. It is in the .vnc directory.

Another way to find the port is to examine the VNC server process using the following command:

ps uwwwx | grep vnc

Look for the port number in a line like the following:

dan      1881388  0.1  0.3  50652 14284 pts/1    S    17:29   0:00 Xtightvnc :1 -desktop X -auth /afs/physics.wisc.edu/home/dan/.Xauthority -geometry 1024x768 -depth 24 -rfbwait 120000 -rfbauth /afs/physics.wisc.edu/home/dan/.vnc/passwd -rfbport 5901 -fp /usr/share/fonts/X11/misc/,/usr/share/fonts/X11/Type1/,/usr/share/fonts/X11/75dpi/,/usr/share/fonts/X11/100dpi/ -co /etc/X11/rgb

Once you know the port number, connect to the login computer with ssh using the following arguments, replacing 5901 with the port number, USER with your hep.wisc.edu username, and login01 with the specific name of the login computer:

ssh -L10000:localhost:5901 USER@login01.hep.wisc.edu

The above command forwards port 10000 (arbitrarily chosen) on your computer to port 5901 on the server. You can then run vncviewer on your computer and connect to port 10000, which will be forwarded by ssh to your VNC server port on the login computer.

vncviewer localhost:10000

Copying files to/from your computer

One way to copy files between your computer and login.hep.wisc.edu is to use scp or sftp command-line tools or an application such as Cyberduck that uses these protocols for you. Example:

sftp USER@login.hep.wisc.edu:

The rsync command is also useful, especially if you want to periodically synchronize data between two locations. Example of copying data to login.hep.wisc.edu with rsync:

rsync -rauv path/to/source login.hep.wisc.edu:path/to/destination

To copy from login.hep.wisc.edu, reverse the order of the arguments:
rsync -rauv login.hep.wisc.edu:path/to/source path/to/dest

If the path to the destination is not in your home directory, it should start with a '/'. Example:

rsync -rauv login.hep.wisc.edu:/nfs_scratch/user/source path/to/dest

For copying large data files, see the HDFS FAQ.

Batch Processing using Condor

HTCondor may be used directly for running CMS jobs on our cluster. Once submitted, the jobs flock all over the UW campus grid. In particular, the jobs run on hep.wisc.edu (CMS T2) or chtc.wisc.edu, Wisconsin’s Center for High Throughput Computing (CHTC) cluster.

Common Condor Commands

Submit jobs to run in the condor cluster:
condor_submit SUBMIT_FILE

A basic submit file:

executable = myscript.sh
output = output.txt
error = error.txt
userlog = condor_log.txt
queue

List your queued jobs that were submitted on the computer you are currently using:
condor_q

See the status of individual jobs:
condor_q -nobatch

See where jobs are running:
condor_q -r

Check the status of the local (CMS T2) compute cluster:
condor_status

Check the status of the CHTC compute cluster:
condor_status -pool cm.chtc.wisc.edu

You can ssh to a job to debug it interactively using:
condor_ssh_to_job JOBID

Accessing files from your HTCondor job

A job can read files in this way: Reading files.

A CMSSW program cannot write to HDFS directly. Instead, it must write to the local working directory. The script that executes the program must then copy the output file to HDFS.

“farmout” script for submitting a batch of CMSSW jobs

You may use our sample shell script called “farmout” that submits CMSSW jobs to fully process a dataset. This automatically submits the jobs from a your /nfs_scratch directory where logs will accumulate. The data files are copied to HDFS.

Example simulation submission:

farmoutRandomSeedJobs \
    dataset-name \
    total-events \
    events-per-job \
    /path/to/CMSSW \
    /path/to/configTemplate.py

Example analysis submission:

farmoutAnalysisJobs \
    jobName \
    /path/to/CMSSW \
    /path/to/configTemplate.py

Use the –help option to see the options used by these scripts.

Your data files will be stored in HDFS under /hdfs/store/user/username. See the FAQ for more information on how to manage your files, how to use farmout, and how to use Condor.