Installing Jug

The easiest way to install Jug is to use Pacman. Once you have Pacman installed and set up (which is extremely easy to do), you can install Jug like this:

mkdir install-dir
cd install-dir
pacman -get http://www.hep.wisc.edu/~dan/jug/pacman3:Jug-DB

If you are using pacman version 2 instead of version 3, substitute pacman2 for pacman3 in the above.

Most of the installation questions are answered automatically during this process. You will just need to pick a password and location for the MySQL database.

Once this finishes, you can go ahead and play around in a test database if you like. To use the main database, continue with the final setup.

Manual Installation

If you do not use Pacman to install Jug, you can do it by hand. The following software is required:

• JugMaster.tgz
• Python 2.2 or 2.3 (newer versions are untested with jug)
• MySQLdb Python Module
• MySQL 2.33 (newer versions are untested with jug)

Installation Recipe

There are many variations on the following steps. This is just one way that works.

Keep in mind that Jug workers do not need access to any other parts of the Jug software from wherever they run, since they interact with the master via RPC calls over the network. So the installation of Jug does not need to be on a filesystem that is accessible from the worker nodes.

1. Install MySQL and the Python module MySQLdb. You do not need to create any databases or start any daemons. It is convenient, but not mandatory, to install this software on a filesystem that is shared by all the machines that will be running RPC daemons or which will be used to run any of the Jug utilities that access the database, such as jug_make_worker. However, you can install it separately on each host instead of using a network filesystem.
2. If you wish to use SSL between master and workers, also install M2Crypto and a compatible version of Open SSL. Precise instructions for getting this to work are not available at this time, due to incompatibilities between different versions of M2Crypto and python's native urllib.
3. Once you get this far, you can create a simple test environment to play around in or go ahead with the real installation.
4. Choose a host for the database. Expect the database files to use 10s if not 100s of Megs, depending on the size of your batches. On the database host, run jug_init. It will ask you for various configuration information, such as the path to JUG_HOME, where the configuration information will be stored.
5. When jug_init asks if you want to create the database, choose yes. Otherwise, you can do this later by running jug_create_db.

1. If all goes well, the database should start up. You can shut it down and start it up as needed using jug_daemon. To have it start up automatically when the computer boots, symlink (or copy) $JUG_HOME/libexec/jug to /etc/init.d/jug and run 'chkconfig --add jug' or whatever you have to do on your version of linux to get the init.d script added to the appropriate run levels.
2. Now, whenever you will be using Jug commands (such as jug_daemon), source setup.sh to configure your environment first.
3. You should adjust the permissions on the directories for the various database role accounts in $JUG_HOME/etc. The directories admin, master, and reader each contain login information used to access the database for various purposes. (These are called "roles".) The stored password can be controlled by using jug_login_db and jug_logout_db. Information gathering commands, such as jug_batch_status require "reader" level access. Database modifying operations, such as jug_submit and the JugMasterRPC daemons, require "master" level access. After creation of the database, only the human administrator should require "admin" level access, so the saved password for the admin user is removed once the database is created.
4. Optional: when the database was created, an RPC server was also started up and configured to run via jug_daemon. You can augment or replace this with additional RPC servers on other machines. The daemons automatically register themselves in the database so that when you use jug_make_worker, the URL of the RPC server(s) will be automatically inserted into the worker, so that it knows how to phone home when it starts up.
5. If your jobs do their own stage-out transfers (or if your jobs do not generate output files that need to be stored), then you don't need any storage workers. Otherwise, you will want to set this up before you create any jobs (or use the "test" storage class to collect output in installation-dir/jug/var/storage). See Storage Workers.
6. In case you want to submit jobs to the Condor batch system, a Condor queue manager has been configured for you. If your jobs have no special requirements, you can simply create them with execution_class="condor" and they will use the generic condor worker class. Otherwise, you can create your own class of workers. See JugWorker.

   There is also a "test" execution class and a test worker running on the machine where you installed JugMaster (see it with jug_daemon list and jug_worker_setup view). If you run jobs with an execution class of "test", they will be run by this worker. So using the "test" execution class and the "test" storage class provides a simple way to experiment with running jobs and storing output directly on the machine where you installed JugMaster. It is obviously not intended for running large-scale work. To do that, you need to run workers on other machines, for example, by using the "condor" execution class or by running some other queue manager or by starting workers through any other means.

7. Now you are ready to create some jobs and run some workers. See jug_submit.
8. You can view general status of jobs being managed by JugMaster by installing a CGI script on your web server (in a directory that supports CGI execution). To do this, use the command $JUG_SYS/JugCGI.py --install /path/to/cgi-bin/JugMaster.cgi. If it is not convenient to install the CGI script in this way, you may access the same information by pointing your browser at the port used by the JugMasterRPC daemon(s).

This problem is due to a broken python installation. Installing a newer version of PyXML is the easiest way to fix the problem.