Thursday, October 16, 2008
Division of Information Technology
910.521.6260 | email@example.com
E-mail Outage Update:
Faculty, staff and students,
Restoring our old email system reminds me of herding cats. Just when you think you have one or two corralled they slip through your fingers. The same is true for problems we are having restoring the old e-mail system. There is a huge amount of data on the old system that must be restored and verified. Even though we actively purged old accounts from this server, there are literally tens of thousands of e-mail files that have been accumulating since early 2000 when this system was installed. This system is backed up nightly to tape.
While we were restoring this data from the tape backup, another hard drive out of the 16 drive array, began reporting some errors. Rather than risk further problems, we plugged in a replacement drive from our spares pool. This did not stop the restore process, but did seriously impact the speed of the operation as the new drive was initialized. This is the first time in the system’s eight year history that a full restore from backup has been necessary.
To those where I created the hope of having the system up Wednesday afternoon, I sincerely apologize. That was our best guess at the time. This drive replacement set us back so our data restore ended around 9:30 Wednesday evening. Then we began the process of indexing or verifying all the e-mail accounts and the associated e-mails for each account. We did not want to bring the system up and then have you experience potential issues with getting your e-mail archives. To give you a perception of e-mail volume, as of Wednesday afternoon we are holding on a temporary server over 75,000 messages that have been received over a 12 hour period. These messages will be delivered once the old system comes back online. Faculty and staff on the new Exchange system should be receiving e-mail notices that their messages are waiting to be delivered if sent to someone on the old system. Otherwise, these faculty and staff should see normal e-mail operation. Students who are testing our new student e-mail system are not impacted by this outage.
There are currently over 9,000 e-mail accounts on the old e-mail system. We hoped the verify process would go smoothly. However, starting at 10:30 p.m., the system processed about 2400 accounts in 4 1/2 hours. As of 3 a.m. Thursday morning, at the current rate this process will take approximately 15 hours. Again, just like herding cats we are searching for ways to speed up the verification. We also are looking for alternative ways to provide you e-mail services until the system is restored to full operation. This may mean rolling out the new BraveMail to all students. Our concern on BraveMail is that we have seen some very random login errors that have yet to be fully resolved to our satisfaction. These are so random in nature that we hope Microsoft’s infrastructure and our local login process are now working hand-in-hand.
Thank you for your understanding and patience. It is a difficult time for us all as we so depend upon our technology. We will get this fixed and we will, with our new Exchange system for faculty and staff and BraveMail for students, have a reliable and sustainable e-mail infrastructure.
For more information about the DoIT relocation, please contact the Help Desk at firstname.lastname@example.org or 521-6260.
© The University of North Carolina at Pembroke
PO Box 1510 Pembroke, NC 28372-1510 • 910.521.6000