Dan Bradley

Email	dan@hep.wisc.edu
UW HEP	608-263-0545 Chamberlin 3241
UW CS	608-890-0032

Contents

• What I Do
• What I Have Done
• My Software

In November 2002, I became a system programmer for the High Energy Physics Department at University of Wisconsin, thanks to funding from the National Science Foundation.

I work on software related to CMS, an international physics experiment that will begin taking data sometime in 2008. Among other things, this experiment hopes to detect the Higgs particle, which is a predicted but so far undetected component of the current Standard Model in particle physics.

In order to prepare for analyzing real experimental data, physicists need to run monte carlo simulations of the detector. The cheapest way to run millions of these intensive simulations is not to buy supercomputers but instead to use a computing grid composed of thousands of commodity machines.

Here at UW, we are focusing on the use of shared resources, machines that are not solely owned or dedicated to one task. By using Condor, different groups of researches use the resources opportunistically, resulting in a high rate of resource utilization (fewer wasted CPU cycles). We have constructed a campus computing grid based on these principles. It is called GLOW, The Grid Laboratories of Wisconsin, which is now part of the UW's Center for High Throughput Computing (CHTC). GLOW is connected to OSG, the Open Science Grid, and we are active in DiSUN, the Data Intensive Science University Network.

• Condor (I can only take credit for some of it, of course)
  • Condor scalability
  • Condor Connection Broker (CCB)
  • Condor Shared Port
  • Condor Rooster
  • Condor Job Router
  • condor_ssh_to_job
• CMS Farmout scripts
• condor_cmsRun
• Useful Hacks for condor.pm in OSG
• condor_suspend
• condor_usage
• JugMaster, database-driven compute-job managent system
• CMSJug [download] [live], scripts for doing CMS production with JugMaster
• KMProd [live], scripts for doing CMS trigger knowledge management production with JugMaster
• UML wrapper, for running compute-jobs in user-mode-linux
• dCache installer (out of date)
• machine test utility
• PSafe, save and retrieve passwords

• For DiSUN, I worked in the Condor scalability group to focus on condor schedd scalability. We found and fixed several cases of temporary deadlock between the schedd and other Condor daemons. We also improved the performance of ClassAd functions and schedd scheduling algorithms. With 10k jobs in the queue and ~1600 jobs running, this resulted in well over an order of magnitude speedup.
• Started work (with other Condor folks) on a Condor job-router, which takes vanilla jobs submitted to the local Condor batchsystem, and channels them to remote grid sites. In the process, made improvements in Condor's support for Quill, a daemon which can unobtrusively follow the activity in the schedd's ClassAd collection by reading the job queue log.
• Fixed support for transferring stdout/stderr in Condor's vanilla universe to arbitrary paths. Generalized the feature to support other output files and to be more robust by avoiding intermediate copies of the output files.
• Wrote a recipe for running Globus jobs on the Condor batchsystem without having the GASS Cache on a shared filesystem.
• Added remote_initialdir feature to Condor vanilla universe. Fixes a problem running CMS grid jobs on the UW campus grid.
• Hosted Condor-C workshop for CMS, Atlas, and CDF.
• Took over responsibility for Condor Glidein. Added a number of additional features.
• Created a simple, extensible machine test utility, which can be used as a Condor Hawkeye module to advertise self-test results in the machine ClassAd.
• Began working on a Knowledge Management project for trigger validation.
• Created a suite of scripts for running CMS simulation jobs for local analysis at UW HEP: cmsjug.0.1.tgz.
• Created a pacman installation for dCache for possible use as a storage service on Grid3. Much work on little details so that the software can just drop in place and work.
• Implemented a new preemption mode in Condor. It allows the startd to publish a promise that it will not interrupt jobs for X amount of time, where X is some arbitrary ClassAd expression. When a resource claim is preempted, the startd will wait for the current job to finish or for the time limit to expire. This can provide much more efficient resource sharing when users have non-checkpointable jobs.
• Implemented greater control over job steering in the Condor negotiator. Two additional ClassAd expressions are used to sort available machines before and after the user's own ranking of resources is applied. A simple example of the use of this would be a negotiation policy that forces jobs to use idle resources before resorting to preemption. Much more complicated policies may also be expressed.
• Wrote a patch to the Condor jobmanager for Globus to solve problems with orphaned jobs on the grid.
• Wrote a user-mode-linux job wrapper to transparently run jobs inside of UML. This allows us to create a Red Hat 7.x environment for running CMS software, even when the computing nodes are being upgraded to Red Hat 9.
• Grid3 site admin for UW Madison. Worked with Shawn McKee to debug problems related to use of AFS to contain the GASS cache and other shared space used by grid applications. Here's a recipe to get things working.
• Created JugMaster for running CMS PCP production at UW HEP.
• Added Condor-G matchmaking to USCMS MOP. Wrote a technical report describing the details, since we immediately began getting requests from other groups for help on getting this sort of thing working.
• Working with Radhakrishna (Iyer) Gowrishankara, we added Sandbox Mode to USCMS MOP.
• Helped run the winter 2002 production of CMS simulation events, using a flock of computers from UW CS, UW HEP, INFN, and other machines scrounged from here and there (like our lone CERN office machine!).
• Submitted a COBRA 7.1.0 patch to CERN, implementing a new .orcarc datacard CARF:AutoResumeRun. This augments the existing CARF:ResumeRun mechanism so that the application will automatically restart itself from the last committed event.
• Working on self-checkpointing in the new generation of CMS software. Previously, we used Condor's standard universe, which does checkpointing, and remote I/O automatically, but the new software will be multi-threaded, which currently prevents it from being linked for standard universe execution. As the first step towards this end:
• Supporting remote I/O in ROOT when running in Condor's vanilla universe. This is done via a Chirp pluggin that I made for ROOT. (The patch for Root version v3.03.09 is here. To test it, see the intructions.)
• Investigated compilation of ROOT for Condor Standard Universe. This was easily achieved (but we had to abandon hopes of using Standard Universe because of multithreading in COBRA).