Bungled Unified Messaging
We’ve done more than our share of firefighting over the last 30 days. While there are a lot of acceptable approaches to technology, straying from or outright ignoring best practices is just a disaster waiting to happen.
The call came in early Friday morning. The customer was in a panic, fed up with their current support, and we were referred by one of their suppliers. E-mail was down and it was costing the business plenty. We were able to do a remote session within minutes and access the Exchange server. The customer had explained the environment was HyperV and it was a virtual Exchange 2007 server. The most common cause of e-mail failures is low disk space or a large message hanging the queue. With virtualization, recovery is usually easier. Should be just a walk in the park, right?
No, wrong. Exchange was 2 service packs behind. We didn’t need them, but couldn’t get help from Microsoft anyway because unified messaging was installed – which flat isn’t supported in a virtual environment. It didn’t make sense either as the customer had a Cisco Unity system that performed the voice mail to e-mail function. Exchange had not been given enough RAM and had just one processor. There were two storage groups: one for executives and the other for the remaining staff. The exec database was only about 18GB, but the staff was over 200GB. Why hadn’t all 5 storage groups been used to make the staff database 4 manageable files? We mounted and started the executives storage group and they had communication, but the staff database was corrupt and everyone else had no e-mail.
When a system is down, you don’t want to waste valuable time removing unnecessary components or installing needed service packs because the likelihood is more problems will be introduced and recovery will be more complex. While the big dogs temporarily had e-mail, we had to take them offline to run a repair utility. The customer only had one virtual host and one Exchange server or we could have left executives working and run the repair on the other machine. The databases also were not on any storage and simply part of the virtual hard drive. Because of the poor configuration, file access was slow and the repair was calculated to take more than 72 hours.
The customer was also paying an outrageous fortune for a Zenith Infotel store and forward system billed as online backup, that just simply copied files to a NAS. In the best of scenarios with a properly functioning Exchange, many seasoned veterans don’t know how to restore an Exchange database and merge the recovery storage group. A copy of a raw Exchange database would be corrupt too and require repair, after taking an initial 4 hours to copy across the network from the NAS.
So 90 minutes after the initial call, the customer is informed that she has a costly and extended outage of e-mail until mid next week or switch to cloud computing and have e-mail flowing within minutes. All of her problems could have been avoided by:
1) Giving the virtual machine more memory and at least two processors.
2) Utilizing storage so data access is fast and cannot get corrupt by being part of a virtual hard drive. Also, snapshots can be restored in minutes rather than hours.
3) Having a second virtual host and virtual Exchange server so maintenance or backup/repair functions can be done on one machine while the other is servicing the rest of the organization that is functional.
4) Using a valid online backup solution that doesn’t simply copy raw files, but utilizes the Microsoft API to properly backup the databases.
5) Investing in managed services to keep current with maintenance by applying service packs and monitoring disk space.
