Virtualization Blunders
The CFO of one of our managed services customers was captivated with the owner from a new startup competitor. She also fell for the “we’re new and hip” line. The ironic part was we had proactively presented virtualization of their aging servers to buy less equipment less often along with storage, but the startup did the same and was selected like it was a great new idea. However, it wasn’t the same and the differences turned out to be striking.
To begin, there was zero planning or communication. Equipment showed up and sat idle for over a month before implementation was started. The customer wasn’t educated on options, involved in decisions, or informed of the conversion process. You knew it wasn’t going to end well.
In a power play, the startup was going to also replace the firewall, upgrade the domain, upgrade messaging, and implement online backup. This project definitely wasn’t about virtualization any more. The short story is that just about everything that could have gone wrong did: basic networking still remains flaky to this day, files were copied without previous permissions for access by all users, SMTP wasn’t restricted on the firewall causing the customer to be blacklisted, copiers and applications couldn’t send e-mail, everything was slow, and accounting and line of business application users were disconnected several times a day causing lost work and data corruption.
It turned out the startup had this nifty bundled package with virtualization and storage all monitored in India, using malware like agents on all machines. Data was replicated to a foreign-owned data center and the startup would get a call from India if there were any alerts. The customer of course never understood any of this scenario and it gets worse.
The first couple of mistakes were inadequate networking and storage. While the server was the right model with ample memory and processor, there were only two network cards. One card is required for the host and other cards are required for clustering, storage, and virtual servers with some high demand virtual servers requiring their own physical card. This configuration had 6 virtual servers with no Storage Area Network (SAN) and having high-capacity applications including Exchange and SQL that should have had their own physical network cards. It’s pretty easy to see why networking was slow as we usually specify 4 – 8 network ports on each virtualization server host.
The startup’s storage was only a hook for the monitoring and an on-site copy to store and forward data. It was a Network Attached Storage (NAS) device meaning all the data must be copied over the network, unlike a SAN which sounds similar but lets applications like Exchange and SQL run databases as if local drives. The virtual server hard drive files are then much smaller for quick copy or failover and snapshots on the SAN allow fast restore of databases in minutes rather than hours across the network. The startup combined data in the virtual machines so the Exchange virtual machine file was over 100GB because of the large Exchange databases. Copying a file that big over the network could take more than 2 hours on a fast network and a repair of an Exchange database in this configuration could take more than 24 hours.
The other major blunder the startup did was installing Hyper-V in core mode, which is designed for dense large data centers or specific purpose applications. Core mode is command line only, so it must be configured to allow a workstation with special software to manage the virtual host. While the startup thought this was cool, they didn’t configure the remote administration and you have to awkwardly RDP to another virtual machine to use the Hyper-V Manager to work with other virtual machines. However, in this configuration you are greatly hampered in identifying or rectifying a host machine that has numerous network and disk bottlenecks.
The customer did hold the startup’s feet to the fire, but our team still spent 5 times the support cases of similar environments for the next 6-9 months. For free, we migrated accounting and another line of business application back to old physical servers. In addition, we setup a virtual WSUS server, since updates were also forgotten by the startup.
The CFO was pushed out a few months after this debacle and our managed services contract was renewed. We’re in the process of moving e-mail and documents to the cloud for less cost than the startup’s gouging SPAM filtering. Plus, we’re reducing the client’s monthly cost for online backup and management with fewer servers and less data. Hopefully, the client has some budget next year, so a second virtual host can be properly setup for failover as you’re not really supposed to put all your eggs in one basket on one machine. Maybe at that time, we can then reinstall the first virtual host properly with a full OS and add some network cards. We hope this post gave you insight on avoiding common virtualization blunders.
