vn.physics.ubc.ca: PIII Cluster News Archive

2006

JULY 10, 2006, 1600:

vnfe1:/home was completely filled earlier today.

Any users running jobs on that partition should check their job output for integrity and restart processes as necessary.

All users---but especially those relatively new to the cluster---are reminded to actively manage their disk space in order to keep all partitions at or below 80% usage. Refer to the summaries generated each morning at 0500 and posted via the cluster web page for a daily snapshot of your usage.

Also, for users in Steve Plotkin's group, note that all nodes are now back on-line. Report any remaining issues to Matt at your convenience.
JULY 7, 2006, 1230

THE FOLLOWING NODES ARE STILL UNAVAILABLE:

vn65, vn66 & vn67

Management is still working on the issue, and thanks you for your
continued patience.
JUNE 27, 2006, 1100

vnfe1 WAS ACCIDENTALLY REBOOTED THIS MORNING (SEVERAL TIMES) DUE TO MANAGEMENT ISSUES.

In the unlikely event that ANYONE was actually running a task performing I/O to a vnfe1 partition, the output from those jobs should be checked for integrity, with jobs restarted where necessary.

Management apologizes for the inconvenience.

Also vn36 is off-line, probably due to a power supply issue, and is awaiting attn from AJP (ahem!!)
JUNE 21, 2006 1800: THE FOLLOWING NODES ARE UNAVAILABLE

vn62, vn65, vn66 and vn67

Management is still working on the issue. Thanks for your patience.
JUNE 15, 2006 1600: IMPORTANT!!

vnfe1 is back on-line!

Management aplogizes again for the inconvenience and thanks Tony at Varsity for his usual prompt and professional service in replacing the power supply. Almost seven years on and the inside of the box is as clean as the proverbial whistle. Environmentals, children, that's what cluster stability and longevity is all about!

ROCK ON, and, most importantly, report and problems to Matt per usual.
JUNE 14, 2006 0430: THE FOLLOWING NODES ARE UNAVAILABLE

vn62, vn63, vn64, vn65, vn66 and vn67

Management is working on the issue. Thanks for your patience.
JUNE 14, 2006 0600: IMPORTANT!!

vnfe1 has died (most likely its power supply) and may be offline for much of the rest of the day

This will render this cluster useless to those of you that run off of partitions mounted from vnfe1.

Management apologizes for the convenience, and is doing what is necessary to get vnfe1 back up.

As a consequence, note that CVS@vnfe1 will be unavailable until further notice as well.
JUNE 5, 2006: THE FOLLOWING NODES ARE UNAVAILABLE

vn62, vn63, vn64 and vn65
JUNE 4, 2006, 1830: The cluster is back on line following the completion of the electrical work in Klinck. It appears as if this cluster escaped unscathed, but, as usual, report any problems to Matt.

JUNE 3, 2006, 0800: IMPORTANT!!

The power in the LS Klinck Bldg, which houses our cluster, will be turned off later today so some electrical work can be done, and thus our clusters are offline until further notice.

Management apologizes profusely for the non-existent notice of this event, and particularly for the fact that users did not have a chance to gracefully terminate their jobs. The work should be done by about 1000 tomorrow (Sunday), but management may be having dim sum at that time, so stay tuned to this page for status!

FEBRUARY 22, 5:30 PM: ATTENTION!! USERS IN STEVE PLOTKIN'S GROUP!

See HERE for up-to-date information concerning the incorporation of Steve's machines into this cluster.

2005

DECEMBER 12, 10:30 AM: OK, THIS IS BETTER. BUT ... THERE ARE STILL ABOUT 70 IDLE PROCESSORS.

PLEASE DO YOUR BEST TO KEEP OUR GOOD "OLD CLUSTER" HUMMING AS IT ENTERS ITS SEVENTH YEAR OF CONTINUOUS OPERATION!! (Note, the README-FRAG file has about 1,000 lines; the full README now weighs in at 23,842 lines, but it, and README.CRASH, contain a very high percentage of all of the keystrokes executed by yours truly in managing this beast!)

ALL THE BEST TO YOU AND YOURS FOR THE SEASON FROM MATT, MONIQUE AND PAUL!!
NOVEMBER 2: THE FOLLOWING NODES ARE UNAVAILABLE

vn62
OCTOBER 6, 3:00 PM: IMPORTANT! vn62 has changed identity, and is now vn41. Old vn41 has been taken off line for diagnosis. Users should execute '/bin/rm -f ~/.ssh/known_hosts' to avoid triggering confusing error messages from ssh.
AUGUST 29, 3:00 PM:

IMPORTANT: The vnfe3:/home partition was completed filled recently.

USERS running jobs off that partition should check program output for integrity and restart as necessary.

Management apologizes for the inconvenience but reminds users that ALL users are responsible for CLOSELY monitoring their own disk usage. Thanks as always for your cooperation on this matter.

NOTE THAT THE 80% FIGURE IS NOT ARBITRARY.
AUGUST 2, 5:45 PM:

Nodes vn22, vn29, vn32 and vn41 are back online. In actuality, only one power supply blew; the other nodes simply needed a "true" hardware reset (i.e. unplug AC cord and leave unplugged for 30 mins.)
JULY 30, 8:15 AM:

The following nodes all died---nearly simultaneously---about 19 hours ago:

vn22        down     19:00
vn29        down     18:59
vn32        down     18:57
vn41        down     19:22

Although this may seem oddly coincidental, it would not be a singular experience for us; we had, we believe, 6 power supplies fail in a single day on our new cluster shortly after its installation.

Users who were running on those nodes have the apologies of management and will have to, of course, restart their jobs.
JULY 12, 6:30 AM: vnfe1:/home2 was completely filled at about 5 PM yesterday (Tuesday) due to certain technical issues.

Users who were running off that partition should check job output for integrity and restart/resubmit as necessary.

Management apologizes for the inconvenience.
APRIL 29, 3:00 PM: THE FOLLOWING NODES ARE UNAVAILABLE UNTIL FURTHER NOTICE:

vn63, vn64
APRIL 25, 1:00 PM: THE FOLLOWING NODES ARE UNAVAILABLE UNTIL FURTHER NOTICE:

vn25, vn43, vn63, vn64
APRIL 10, 9:30 AM: vn25 crashed overnight and vn6 was also brought down by a technical management problem.

Management apologizes for any inconvenience caused.
MARCH 29, 10:30 PM:

UNAVAILABLE NODES

      vn31, vn63, vn64

Also note that vn6 and vn63 have now switched identities. In order to minimize the number of potentially confusing and/or annoying "host key has changed" messages, management has executed

root% /bin/rm -f /d/*/home*/.ssh/known_hosts

which, of course, removes everyone (but root's) ~/.ssh/known_hosts files.

This should be of no consequence to users, although you will see warning messages as you connect to various hosts in the cluster for the first time.

Send Matt a message in the unlikely case that you actually need the old copy of your ~/.ssh/known_hosts file.
MARCH 17 3:00 PM

vn6 HAS BEEN UNSTABLE LATELY AND SHOULD BE AVOIDED UNTIL FURTHER NOTICE

In conjunction with this instability, please do not start any jobs on vn6 until further notice, as management wishes to swap identities of vn6 and vn63 to maintain a contiguous range of stable nodes.
JANUARY 30: 4:30 PM: vn63 AND vn64 ARE UNAVAILABLE UNTIL FURTHER NOTICE.

JANUARY 30: 2:00 PM: vn20 IS UNAVAILABLE UNTIL FURTHER NOTICE.
With luck, in a few hours it will be vn64 that will be unavailable. Stay tuned.
JANUARY 27: 2:00 PM: The cluster has been partially repretoriusized, so there now only about 60 IDLE PROCESSORS; Get 'em while you still can!

JANUARY 20: 11:45 PM: vn25 IS UNAVAILABLE UNTIL FURTHER NOTICE.
JANUARY 16: 2:45 AM: The following nodes are unavailable until further notice:
```
   vn6  vn20  vn25  vn52
 
```
JANUARY 16: 2:45 AM
The cluster upgrade is now nearly complete.

Users may log in and submit jobs, but should note that various machines may still have to be rebooted on very short notice

Report any problems to Matt

2004

DECEMBER 21: 12 NOON: *** ANOTHER IMPORTANT NOTICE ***
Most of the necessary hardware maintenance on the cluster has been completed. Unfortunately, not only have we lost vnfe2's processors, we have also apparently lost vnfe2:/home2.

Until further notice, that partition (i.e. /d/vnfe2/home2) will now point to

/d/vnfe4/home/vnfe2-tmp/home2

The current state of /d/vnfe4/home/vnfe2-tmp/home2 reflects the state of vnfe2:/home2 as of approximately mid-November. Since this partition has seen very little usage lately, for the time being we will not attempt to FURTHER update/restore its contents.

Users with specific problems/questions can, as usual, contact management directly.