USF Information Technology, Research Computing

Login Node Performance Issues

Brian Smith — Wed, 16 May 2012 16:43:42 +0000

After having some issues with compute nodes in Tampa losing access to the /work filesystem, our team discovered some latency inconsistencies between Tampa and Winter Haven (network packets taking longer to get from one place to the next). After an exhaustive investigation, we discovered that one switch in the Winter Haven was experiencing a performance bug causing it to delay the forwarding of packets, introducing latencies of 30-90ms (miliseconds). This may not sound like a lot, but when you consider that any time you do an “ls” on your home directory, potentially hundreds of individual requests can be made to the NFS server. At 30-90ms per request, significant lag can occur between issuing the command and actually getting a response. Also, when you consider creating or accessing a tar archive, the thousands of operations involved will be slowed down by that additional latency, making a process that could take 5 seconds instead take 5 minutes. We’re now down to normal latencies which are around 200us (microseconds).

After fixing the configuration and reloading the switch, system performance across the board improved dramatically, resolving the file transfer performance issues we saw with NFS, the mount issues with the compute nodes, and a couple minor annoyances (tab completion, etc.).

Research Computing Users Group & Workshop

Brian Smith — Mon, 16 Apr 2012 18:11:49 +0000

For those of you that might be unaware, RC has been holding weekly user group and workshop meetings to answer questions and discuss issues in person.

The weekly Research Computing Workshop Sessions will be moving to the new Advanced Visualization Center beginning this week. The Visualization Center is in room PHY 147, which is across the hall from the physics auditorium. The sessions will still be held on Tuesdays from 2-3pm, and are open to anyone that would like to ask questions about or discuss issues with our systems and applications. Also, feel free to stop by to check out the Visualization Center and its brand new 3D-HD visualization wall.

Issue with /work 04/12/2012

John DeSantis — Thu, 12 Apr 2012 20:51:28 +0000

Today it was discovered that roughly 40 compute nodes within our cluster had dropped their /work mounts.

As a result of this issue, those nodes needed to be rebooted. Once the nodes resumed operations, /work was again available without any issue.

Because of the reboots, running user jobs would have been negatively affected. These jobs will need to be re-submitted given not only the interruption caused by the reboot, but also because when /work drops as a mount, output is no longer generated and the job is effectively stalled.

Please check your recently submitted jobs and make sure that they completed with appropriate output files, and if you notice that there is entropy within the output files or no output files at all, please re-submit the job(s).

John DeSantis
Research Computing

Maple 16 and Matlab R2012a News

John DeSantis — Thu, 12 Apr 2012 13:01:05 +0000

It is our pleasure to announce that Maple 16 is now available for use on CIRCE. Users of NX will find that it is listed under “Applications -> Education”.

Matlab release R2012a is now available for cluster use and can be downloaded by students and faculty engaging in research from the RC ISO’s site (NetID and Password required). See our documentation for more information about installing and running Matlab here: https://rc.usf.edu/trac/doc/wiki/MatlabUser

John DeSantis

InfiniBand Fabric Issue, 04/10/2012

Brian Smith — Tue, 10 Apr 2012 20:56:36 +0000

During work to install several more compute nodes to the wh.2012.01.q hardware pool, the power cable to one of the InfiniBand switches was briefly disconnected by accident. If any jobs fail with the below error, please resubmit them as the problem has been resolved. We apologize for any inconvenience.

…
WARNING: There is at least one OpenFabrics device found but there are
no active ports detected (or Open MPI was unable to use them). This
is most certainly not what you wanted. Check your cables, subnet
manager configuration, etc. The openib BTL will be ignored for this
job.
…

May 2012 Upgrades to /work Filesystem

Brian Smith — Tue, 10 Apr 2012 16:51:52 +0000

During the first week of May, we will be upgrading the current /work filesystem in order to provide significant performance enhancements and most importantly, long-awaited fixes related stability.

/work will be migrated from a home-grown GlusterFS volume of roughly 100TB to a vendor-supported Lustre filesystem of equivalent capacity, capable of 4GB/s of read/write throughput and a variety of options for supporting various applications as efficiently as possible. This upgrade will eliminate the issues we have had with /work since January providing greater stability for jobs and much improved performance for jobs that are I/O-hungry.

All current data on /work will be copied to the new /work filesystem. There will be a cut-off date, where the /work filesystem will be taken offline for several hours to do one last sync of the data before bringing the new system live. We will narrow down this date and let everyone know when the downtime period will be.

/work2 will be taken out-of-service following this upgrade as it will be made redundant by the new configuration. Data on /work2 will NOT be transfered and it will be up to you to ensure that data is copied either to /work or /home prior to taking the volume off-line.

This work is slated to occur sometime during the first week of May. As we narrow down the date, we will update you with our revised timeline.

Thank you for your patience and cooperation during this process. Please let us know if there are any questions related to this work.

New Compute Resources: Sandy Bridge and Fermi GPUs

Brian Smith — Tue, 10 Apr 2012 16:50:18 +0000

Despite the significant system expansion that occurred back in January, adding over 1500 CPU cores and several terabytes of memory, we can always use more resources. We are finalizing some small details to add at least 60 new systems to the cluster based on the E5-2630 Intel Sandy Bridge CPU. This will add an additional 720 CPU cores (minimum) of cutting-edge processing power to the system. We will be working to recompile applications and libraries to take advantage of the new CPUs and their capabilities when they come online during the month of June.

We are also working to integrate existing applications with our available GPU resources and would be very much interested in any early adopters/beta testers helping us to work out kinks in the new system.

Challanges to address include

1. GPU scheduling
2. GPU/RDMA/MPI integration
3. Application integration

If you have an application that will lend itself well to GPU computing and already has support for the CUDA architecture, please let us know if you are interested in testing the new resources.

CIRCE Research Users: UPDATE: /home issues — 3/2/2012

jfargen — Sun, 04 Mar 2012 15:54:44 +0000

The system should be back up and functioning normally. We had to do extensive checking and debugging of the filesystem which resulted in the loss of some files. Our timestamps indicate that many of these files are either

a) from today, which cannot be recovered
b) last accessed more than 2 weeks ago, which can be recovered

Please let us know, by filing a ticket at help@usf.edu, if you have any missing files so we can restore them.

We apologize for this inconvenience and are working with data center services to resolve the recurring issues with /home which are the result of faulty hardware supplied by our vendor.

My apologies for any inconvenience today’s outage may have caused.

Best Regards,
Brian Smith

Update: Storage Issue — 3/2/2012

jfargen — Fri, 02 Mar 2012 19:17:58 +0000

An attempt to replace a failed drive in the array has caused the storage system to go offline. We’re working to resolve the issue. Currently, /home, /apps, and /shares are unavailable.

-Brian

Storage Issue — 3/2/2012

jfargen — Fri, 02 Mar 2012 19:16:27 +0000

To all:

Due to a drive failure in one of the arrays which stores /home, I’m stopping all jobs that will run from within /home in order to reduce the load on the storage system so that the array will rebuild as quickly as possible. All new jobs submitted from within /home will be presented with an error message stating that they should not be run from that
location. Please use /work instead.

I will keep this list updated as the array repairs are completed.

Best Regards,
Brian Smith