Computing and Storage Policy: AFS, CephFS, NFS, and HDFS#

This document defines the computing and storage infrastructure and policies for users at the Wisconsin T2 site.#

Computing Infrastructure#

The site provides a High-Throughput Computing (HTC) cluster based on HTCondor, a powerful environment designed to efficiently handle massive computational workflows, parallel simulations, and large-scale data analysis.#

  • Compute Power: Approximately 17,500 CPU cores running Enterprise Linux 9 (EL9).#

  • Remote Access: Six interactive login servers that serve as gateway for remote login, managing files, writing code, and preparing jobs.#

  • Shared Resource Etiquette: Login servers are strictly for configuration and lightweight tasks; all heavy processing must be offloaded to the cluster.#

  • Workload Management: The HTCondor batch system automatically handles, balances, and distributes your tasks across thousands of available worker nodes.#

  • Express Execution: A specialized fast queue is available within HTCondor to fast-track and rapidly deploy short, time-sensitive jobs that require immediate processing.#

Storage Infrastructure#

The site utilizes four primary file systems: AFS, CephFS, NFS, and HDFS. Home Directories are managed via AFS and CephFS for standard user environments. NFS scracth space is provided for intermediate processing and short-term data. HDFS is primarily used for long term storage of CMS Monte Carlo (MC) simulations, official datasets, and user analysis ROOT files.#

  • AFS Allocation: Home dirs, Configuration files, grid certificates (~/.globus), SSH keys, and lightweight shell scripts.#

  • CephFS Allocation: Home dirs, Active CMSSW code releases, local software environments, and configuration repositories.#

  • NFS Scratch Allocation (/nfs_scratch): Local job execution, short-term staging, interactive test runs, and high-I/O analysis tasks.#

  • HDFS Allocation for Users (/store/user/$userid): Primary long-term storage of root files for analysis purpose. Each user is allocated 10TB of space (Quota) as a default. Request for Quota increase is subject to approval from the thesis supervisor.#

High-Throughput Grid and Local Scratch Usage#

  • No Grid Job I/O in Home: Grid jobs running on worker nodes must never perform I/O operations directly inside AFS or CephFS home directories.#

  • Scratch Utilization: Interactive and batch test jobs must use the /nfs_scratch path for operational read/write operations.#

  • Per-User Quotas: Individual user storage caps are enforced on /nfs_scratch. Exceeding this quota will cause jobs to fail with “disk quota exceeded” errors.#

  • Mandatory Storage Element (SE): Production scale job inputs and final simulation outputs must use the site’s official CMS Storage Element via xrootd or gfal protocols.#

Space Quotas and Purge Policy#

  • CephFS Quotas: An initial allocation of 100GB CephFS home space per CMS user is provided as defaul. This allocation can be further increased upto a max of 300GB.#

  • NFS Scratch Management ( /nfs_scratch ): This storage operates under a strict No-Backup policy. We strongly recommend using it only for active, temporary workloads rather than long-term archiving. Each account has a 200GB quota. If your usage exceeds this limit, an automated script will email you a warning, and the system will begin purging files older than 30 days. To avoid data loss, please review these warning emails promptly and reduce your storage usage.#

  • Dataset Prohibitions: Storage of official MiniAOD, NanoAOD, or RAW data files inside AFS or CephFS home directories is prohibited. #

HDFS & XRootD Data Management Rules#

  • HDFS File Size Guideline: To ensure optimal performance of the HDFS file/storage system, all user-generated ROOT files should be kept between 2 GB and 5 GB. Avoid writing files smaller than 1 GB, as a high volume of small files degrades HDFS I/O efficiency. Always merge smaller job outputs prior to HDFS storage.#

  • Data Transfer Protocol: Use xrootd for all data movement.#

  • HDFS Storage Quota Lockouts: Exceeding the allocated Quota limit will result in XRootD write failure (permission denied / quota exceeded). Please obtain approval from your thesis advisor before submitting a storage quota increase request.#

Issues with Small Parquet Files on HDFS#

Using Apache Parquet formats for CMS data analysis has become increasingly common. However, writing small Parquet files to an HDFS cluster causes severe performance degradation. Unlike ROOT files, Parquet files store heavy metadata footprints (schema definitions, row group metadata, and column statistics) directly inside the file footprint. Writing many small Parquet files triggers two critical issues: HDFS NameNode Memory Exhaustion and Metadata Overhead, which results in significant wastage of your storage allocation. Please adhere to the following Parquet Storage Rules:#

  • Minimum File Size: Do not write Parquet files smaller than 256 MB to the HDFS storage element.#

  • Target File Size: Aim for an optimal target size of 512 MB to 1 GB per file to align cleanly with HDFS block storage limits.#

  • Consolidate small files:If your parallel jobs or local processing steps generate thousands of tiny Parquet files, you must consolidate them before they are stored in /store/user/$userid.#

HDFS Storage User Exit Policy#

This document establishes a formal exit policy for researchers and students concluding their affiliation with the CMS group at UW-Madison. It enforces strict data management hygiene to protect costly Hadoop Distributed File System (HDFS) storage infrastructure.#

  • Data Auditing: Users must inventory all files stored under their personal HDFS paths (typically /store/user/$userid).#

  • Mandatory Deletion: All redundant, intermediate, scratch, and centrally reproducible CMS datasets must be permanently deleted. Any existing/valid/locked Rucio dataset rules created for T2_US_Wisconsin storage must be deleted/invalidated.#

  • Research Preservation: Critical, unique analysis outputs must be migrated to designated group spaces or transferred to the user’s new institution via Rucio.#

  • Ownership Handover: Remaining essential data must be assigned to a local custodian (such as a thesis advisor or an active local group colleague) and verified by site administrators.#

  • Enforcement Timeline: A 30-day grace period is provided post-association before account deactivation and automated, unrecoverable data purging occur.#