Phedex#

The Phedex service initiates data transfers to and from UW-HEP. For general information, see the website#

Installation Documentation : InstallDoc#

Installing and Upgrading Phedex#

The detailed documentation about PhEDEx is available here https://twiki.cern.ch/twiki/bin/view/CMS/PhedexDraftDocumentation#

At HEP, we follow the same procedure with only a few modifications. This document will discuss the installation of version 4_2_1.#

Phedex is installed in AFS and the volume is named corresponding to the actual phedex release i.e. /cms/sw/phedex_v4_2_1. The primary phedex server is “phedex02” and it runs both “Prod” and “Debug” instances. #

First, create a tree for the new Phedex installation in AFS :#

$ cd /cms/sw/phedex_v4_2_1

Next, install the software in the phedex_v4_2_1 dir following the insturctions here :#

https://twiki.cern.ch/twiki/bin/view/CMSPublic/PhedexAdminDocsInstallation#PhEDEx_Agent_Installation#

The “state” and “logs” directories (configured for all agents) are created (when the agents start) under /scratch/phedex dir.#

New releases often bring configuration changes as well. After installing the latest Phedex version, the SITECONF/T2_US_WISCONSIN dir/files need to be checked out.#

The SITECONF repository is currently maintained in CERN’s GitLab.#

https://gitlab.cern.ch/groups/SITECONF/#

The repository is web accessible to all CMS members. Detail information about SITECONF is available here :#

https://twiki.cern.ch/twiki/bin/view/CMSPublic/SiteConfInGitlab#

Checkout the T2_US_WISCONSIN dir/files from SITECONF Gitlab repo following instucations here : https://twiki.cern.ch/twiki/bin/view/CMSPublic/SiteConfInGitlab#Instructions_for_site_admins#

Since database credentials are stored in the DBParam file, it is not kept in the Git repo and must therefore be copied manually from one installation to the next.#

$ cp /cms/sw/phedex_v4_2_0/SITECONF/T2_US_Wisconsin/PhEDEx/DBParam \
    /cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx/.

After installing the software and migrating the configuration files, it’s important to check that the configuration works with the new version. To test the installation, run the Master script with the development config file:#

$ ./Master -config Config.Debug checkdb

It can be difficult to tell what’s failing (if anything) in the output from the Master script. In such a case, it’s often useful to compare our configuration with that of other sites; their configuration files can also be found in the SITECONF Gitlab repo.#

FTS3 Documentation#

Here is the documentation on how to configure your download agent to use REST interface: https://twiki.cern.ch/twiki/bin/view/CMS/PhedexAdminDocsUsingFTS3Backend#

Common Tasks#

Committing Configuration Changes to SITECONF/T2_US_WISCONSIN Gitlab#

Follow the instructions documented here :#

https://twiki.cern.ch/twiki/bin/view/CMSPublic/SiteConfInGitlab#Instructions_for_site_admins#

Renew a Proxy Certificate#

Since Phedex uses GridFTP and FTS to transfer data, it must authenticate in order to read from other CMS storage elements. Locally, Phedex must also authenticate to read and write from the HEP storage element. Also, the dataset healing script needs to be able to authenticate to download files.#

The credential used for file transfer is stored in the default cmsprod x509 user proxy file (/tmp/x509up_u282). This could be anybody’s credential, as long as they are a registered member of CMS. This needs to be a CMS VOMS proxy. We use the /cms/cmsprod/proxy_renewer script to create the proxy and change the ownership to user cmsprod. This service must be started manually so that the user can enter the needed passwords. To start it:#

/cms/cmsprod/proxy_renewer/proxy_renewer --rfc --voms=cms --hours=200:00 --chown=cmsprod

Note : The --rfc option is needed for the Spacemon script to work properly.#

The proxy renewer is needed on the Phedex node (phedex02) and the node running the dataset healing cron job (currently cmsprod04).#

How To…#

Start and Stop the Phedex Transfer Agents#

On phedex02 :#

$ sudo service phedex stop
$ sudo service phedex start

Both production and debug instances are controlled by the “phedex” service.#

Check the DB Parameters# 

[cmsprod@phedex02:/cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx]$ ./Master -config Config.Prod start
Testing your DB connection
2017-03-23 16:34:48: CheckDBConnection[24982]: (re)connecting to database
2017-03-23 16:34:49: CheckDBConnection[24982]: disconnected from database
Connection to TMDB with /cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx/DBParam:Prod/T2USWISCONSIN succeeded

If the above check (for Master) hangs or can’t get past the DB connection test, there’s little chance that the agents can start.#

Test Catalogue# 

[cmsprod@phedex02:/cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx]$ source ../../../PHEDEX/etc/env.sh
[cmsprod@phedex02:/cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx]$  ../../../PHEDEX/Utilities/TestCatalogue -c storage.xml -p srmv2 -L /store/Generators/2008/5/12/FastSim-Wjets-madgraph-1210606381/0019/0229F5DB-AD26-DD11-BB41-003048357A8C.root

... which results in the following output :

Testing file name mappings in storage.xml using protocol srmv2
LFN: /store/Generators/2008/5/12/FastSim-Wjets-madgraph-1210606381/0019/0229F5DB-AD26-DD11-BB41-003048357A8C.root
PFN: gsiftp://cms-lvs-gridftp.hep.wisc.edu//hdfs/store/Generators/2008/5/12/FastSim-Wjets-madgraph-1210606381/0019/0229F5DB-AD26-DD11-BB41-003048357A8C.root
TKN: 
Re-LFN: /store/Generators/2008/5/12/FastSim-Wjets-madgraph-1210606381/0019/0229F5DB-AD26-DD11-BB41-003048357A8C.root

Data/Block Consistency Check at Site# 

To inject:

[cmsprod@phedex02:/cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx]$ ../../../PHEDEX/Utilities/BlockDownloadVerify-injector.pl \ 
       --db ${PHEDEX_DBPARAM} \
       --block '%Summer12%' \
       --test size \
       --listonly \
       --node T2_US_Wisconsin

To read the reports:

[cmsprod@phedex02:/cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx]$ ../../../PHEDEX/Utilities/BlockDownloadVerify-report.pl \ 
        --db ${PHEDEX_DBPARAM} \
        --node T2_US_Wisconsin \
        --detail \
        --days 1


For help with Injection and Reporting options :

[cmsprod@phedex02:/cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx]$ ../../../PHEDEX/Utilities/BlockDownloadVerify-injector.pl \
       --help
[cmsprod@phedex02:/cms/sw/phedex_v4_2_1/SITECONF/T2_US_Wisconsin/PhEDEx]$ ../../../PHEDEX/Utilities/BlockDownloadVerify-report.pl \
       --help

Monitoring#

Logs are stored in /scratch/phedex/logs. The LoadTest/debug agent logs are prefixed with debug-, and the production logs with prod-. For example, to see logs from downloads in the prod instance, look in prod-download* files.#