Due to a problem that involved several compute nodes and our NFS file server which provides the /home directory, the system became slow and unstable during the day today. During an attempt to isolate the problem nodes/processes/users, we were forced to quickly delete all jobs in an effort to prevent the failure from affecting other systems. We found that a race condition existed on several nodes which caused our NFS server to crash several times today. Those 3 nodes were rebooted and are disabled pending further investigation.
The system should be back up and stable with no residual issues. If you do have any problems to report, please let us know. We apologize for any inconvenience this has caused.