Impressive Data Rates
I'm rather certain we set a new record for egress traffic here at the University of Wisconsin. For a few minutes we peaked at 4.2 Gbps and we sustained 3.3 Gbps for a little over 30 minutes. The end-to-end (Fermi to UW HEP) applications (PHEDeX) data rate was around 2.7 Gbps (330 MBps.)
UW-HEP iperf Service
At the request of Dan Bradley, UW-HEP now has an iperf service available at iperf.hep.wisc.edu. The system (SL304) has reasonable (perhaps not the best?) kernel tuning, and the iperf daemon has a 16 MB TCP window. Internally, we're seeing around 942 Mbps of throughput. Enjoy.
Harvey Newman from CalTech quickly pointed out that this service is of dubious utility. And I certainly agree. But I also believe that our iperf service provides ball-park estimates that are useful, for example, as a "reality check". If you believe your site has 1 Gbps connectivity, ane you only realize, say, 155 Mbps, then you should probably revise your notion of reality!
CMS Tier2 Networking Workshop
I attended a US-CMS Tier2 Networking Workshop today. It was held at CalTech in (ah, rainy!) Pasadena California. The focus was on networking activities of the US-CMS Tier2 sites.
I gave a quick presentation on the UW-HEP Tier2 networking status. I'd love to show it to you, but, ah, I don't have a PowerPoint presentation yet. (I just winged-it.) For posterity, I have to create a presentation. So when it's done, you can find it here.
The 21st Century Network
We've about finished our work connecting the UW-HEP and Pheno offices to the 21st Century Network. DoIT's been itching to finish the migration, and we made a big push in that direction this week. For Pheno folks, the migration means moving into the HEP StaffNet network. HEP folks shouldn't notice much, but everyone should rest easy knowing their packets are being routed on shiny, new hardware.
FNAL Peering and 10gig Ethernet
The University's network engineers (at DoIT) turned up UW-HEP<->FERMI peering this morning. It's done using VRF (i.e. our own virtual router) and BGP (to exchange IP routes.)
At any rate, the good news is that we now have 10GE connections from our machine rooms all the way to/from Fermi. The bad news is that we don't have the ability to verify that we have lots of throughput.
AirPort Extreme Firmware Downgrade
Our AirPort Extremes have been locking up for almost a year now, and the problem has gotten really bad with the new 5.6 firmware. After some investigation, we believe the instabilities started after upgrading from the 5.5.1 firmware, so we've downgraded to the 5.5.1 firmware.
WiFi Network has been "Secured"
The UW-HEP WiFi network ("Chamberlin Airport") is now password protected (by a 40 bit WEP key.) If you need UW-HEP WiFi access or have any problems or questions, please don't hesitate to contact me.
Need... More... Bandwidth...
After bringing up our new "GridNet" network, we found that the trunk interconnecting our two machine rooms was completely saturated--running at 985-995 Mbps. So today we replaced it with a 4 Gbps "etherchannel" trunk. As luck would have it, our CMS and OSG computing network demands dropped below 1 Gbps at about the same time as when we brought up the link, so it took a while before we were convinced that we have a full 4 Gbps of bandwidth.
The Grid Network Is Born
Today was incredibly hectic! Not only did we bring up the 10 Gbps uplink to the Internet, but we also started using our new machine room ("mr2"), moved the CMS Tier2 Server Rack from the old machine room ("mr1") to mr2, reprogrammed three stacks of Cisco 3750 switches, brought up a gigabit interconnect from mr2 to mr1 for the GLOW racks, renumbered the CMS Tier2 Server Rack, and lastly, physically split our network in two. So we now have the "HEP Grid Network" with about 20 servers, 12 storage servers and 186 CPUs in 94 compute nodes, and the "HEP Staff Network" with, well, all the other servers and connections to offices.
Ten Gigabits
It's true: the single mode fiber from the MDF to our new machine room (3241 Chamberlin Hall) is in. We also have a Cisco Catalyst 3750G-16TD. Hopefully we'll get this stuff working soon.
