JUG Master Source Code Documentation
All state is maintained in a relational database.
The code in JugMaster is responsible for accessing and manipulating
the database. Workers never directly access the database.
Components
- CommitOutput
- Storage jobs report when files have been stored.
- CommitJob
- The final step in processing a job.
- DB
- Database connectivity. For multi-threaded applications
(like JugMasterRPC), this manages a pool of connections.
- GetJob
- Handle queries from workers who want work to do.
- JugMasterRPC
- The daemon that handles SOAP RPC calls from workers and
translates these into operations on the database. Any number
of rpc servers may be run on any number of machines, for
scalability and robustness.
- JugCGI
- This module provides a CGI interface for viewing job status
at an aggregate level and for closer inspection. It is possible
to wrap this inside of another python CGI script in order to provide
a customized interface (e.g. a portal for submitting jobs).
- JugConfig
- Access configuration information about the Jug installation.
- JugCommon
- A small amount of code that is used by both the master
and workers.
- JugSubmit
- Module for parsing acting upon jug "submit" files, where
jobs are defined. This is used by the jug_submit command-line
utility.
- PublishOutput
- Advertise the existence of output files that need to be stored.
- ReportStageIn
- Called by storage workers to report that a file has been
successfully retrieved.
- ReportStatus
- Called periodically by workers to report on the status of
jobs they are running.
- Utils
- AbortJob
- Module for aborting/restarting a job. This is used by jug_abort.
- BatchInfo
- Loads and summarizes information for JugCGI and jug_batch_status.
- JugException
- Exceptions used in JugMaster code.
- JobDescription
- Routines for manipulating job-related information, such as
environment substitution.
- Lookup
- Handles turning batch names into batch ids.
- RequestHandler
- Routines for doing common tasks related to RPC request handling,
such as recording failures and authorizing requests.
- TimeEstimation
- Routine for estimating transfer times. The accuracy is
not terribly important (fortunately).