jug_worker_setup

Manipulate worker class entries in the database. A "worker class" is a configuration of JugWorker, for use as a file storage, queue management, or job execution agent. For simplicity, you may derive classes from other "parent" classes. You may also specify that one type of worker should be created (e.g. submitted to a batch system) by another type of worker, called a "queue manager".

Options

--worker_class=X

This is the name you have chosen for this worker class.

--worker_type=Execution|Storage|Queue

This determines the type of jobs that this worker will run.

--parent_class=X

Attributes are inherited from the parent class (and grandparent, and so on).

--queue_class=Y

This specifies what type of queue manager will create workers of this type. The worker class defining the queue manager(s) should have a job_selector that matches whatever you specify for queue_class. If you are just running workers by hand, you don't need to specify a queue class.

--rank=N

Numerical weight when queue worker chooses a new child

--job_selector=X

This causes the worker to run jobs belonging to the named execution or storage class, depending on the type of worker. To match a class with an empty name (i.e. no execution_class specified by the user when creating a batch of jobs), you may assign this to an empty string. The "selector" may contain multiple class names in a comma separated list, and the class names may contain the wildcard character '*', so a selector can match more than one execution (or storage or queue) class.

--software=X

Newline separated list of software packages, if any.

--environment=X

Newline separated list of environment settings, if any.

--runtime_options=X

Newline separated list of worker options, if any. See below for possible values.

--queue_options=X

Newline separated list of queue environment settings, if any. See the documentation for the queue software package to see what options are supported. (For example, see the condor submission package.)

--base_output_path=X

For storage jobs, specifies where files are stored

--notes=X

Any comments.

--old_worker_class=X

For use with clone operation.

--new_worker_class=X

For use with clone operation.

--delete_children_too

For use with delete operation.

--show_inheritance

Show inherited attributes in view operation.

--quiet

Be silent when all goes well.

RUNTIME OPTIONS

Runtime options to JugWorker are specified via --runtime_options="option=value". You may specify multiple options in a newline separated list, or you may simply use --runtime_options multiple times on the same command line. The -- in front of the runtime option name is optional.

--max_stage_in=N

Maximum number of jobs to stage-in files at one time. The default is one.

--max_stage_out=N

Maximum number of jobs to stage-out files at one time. The default is one.

--max_running=N

Maximum number of jobs to run at one time. The default is one.

--max_pre_run=N

Maximum number of jobs that can be waiting to run at one time. The default is one for execution workers and no limit for storage workers. In the latter case, files will remain in a waiting state until all files from a given job have been fetched.

--max_polling=N

Maximum number of jobs running the polling command at the same time.

--max_post_running=N

Maximum number of jobs waiting to poll, polling, waiting to stage-out, or staging out. The default is one.

--quit_job_count=N

Stop after running N jobs.

--stay_alive

Normally, workers exit when they run out of work to do. This option causes the worker to keep running. This option is not necessary for storage workers, becuase they work this way by default.

--workdir=X

The worker needs some scratch space in which to run jobs. By default, it uses the current working directory. You may specify a different path if you need to.

--packup_interrupted_jobs

When shutting down interruptible jobs, this option causes the worker to produce a tar file containing the run directories of the interrupted jobs. This is useful, for example, when running under Condor in transfer-files mode (no shared filesystem), since Condor is not able (as of this writing) to transfer subdirectories. You can use Condor's setting when_to_transfer_output = ON_EXIT_OR_EVICT to save and restart from the interrupted state.

exit_status_on_idle=X

Worker exit status when no work found.

exit_status_on_job_abend=X

Worker exit status when job exits abnormally

Examples

To create a worker class that selects jobs from the default unnamed execution class:

To create a storage server that simply writes files to disk: