Being Diligent and Defending Yourself
I’m closing in on three years at my current job. My partner in crime, Chet, and I faced many IT battles together. It was he who brought me on board after working together at our previous company. He manages our Cisco network and EMC storage equipment. I manage our Symantec and Quantum backup systems and Exchange email servers. We both manage our VMware virtual environment. We have always been on the same side. In tight spots, I usually defer to him as his experience and skill set were broader than my own. That wasn’t the case yesterday.
As he tore down our old, obsolete, virtual desktop infrastructure, he shifted some virtual servers around and needed to reset their backup settings. He did a quick scan of the existing settings and noticed two production servers not being backed up. He gave me an earful. “Come on man, how could you miss these two?” In terms of backup settings, for the most part it is set it and forget it. My main task was to make sure backup jobs didn’t fail. The first miss was an underutilized SharePoint server. The second miss was a secondary domain controller.
He poured it on, “Do I have to audit your work and check your other backup settings?” I saw the error in my ways and quickly apologized. In my mind, he was making a bigger deal than needed. The SharePoint server was rarely used. The primary and third domain controller were being backed up. Their data is replicated amongst the three of them and they are designed to load balance for each other. I stepped out for some air. When I came back in, Chet tore into me again. “What about your Exchange email servers man?”
When we first designed and setup the Exchange email servers, I worked side by side with a highly skilled Exchange consultant. In a debate about backups, I was advised to use Symantec Backup Exec to capture the critical email data. In the event of a disaster, I would need to rebuild the Exchange email servers and restore the data. Although the Quantum snapshot solution would be able to recover quicker, because of the tight integration Exchange has with Active Directory, restoring old Exchange snapshots may introduce Active Directory corruption.
Chet kept up his attack. “You mean to tell me, we have to rebuild everything from scratch, patch them up, and then restore data? It would be much faster to restore at least the operating system from snapshots.” I shifted gears. “Look, you are getting worked up for nothing. On the front end, we have a server pair that is load balanced. On the back end, our mailbox database servers are in a active passive mode. We could lose two servers and still function. I’ve tested this a few times with no issues.”
Chet kept coming. “I know your design looks fool proof. But once again, if something goes wrong with our operating systems, it is much fast to revert to a good snapshot than to rebuild from scratch.” I countered, “I understand what you are saying, but the risk of longer term damage due to Active Directory corruption justifies the longer recover time.”
Having worked side by side with Chet for so long, I know how he operates and what drives his thinking. He is very diligent and was concerned about my lack of attention to detail. I added fuel to the fire when I jokingly said, “Looks like I’m turning into my predecessor.” He had a bad reputation for doing shoddy work. I have to admit I need to be careful of being too complacent. If I continue to slip, I will make myself irrelevant.
Chet’s second motivation is passion. We both marvel at vendors and consultants who are super passionate about their craft. That passion usually is a sign of mastery in their specific IT skill. Sadly, we have also encountered the opposite. People who seemed bored and disinterested usually reflect a low skill level and no desire to improve. Chet was purposely pushing my buttons to reignite my passion. I recall mentioning to close friends, “I may not be passionate about many things. But if you stray on the wrong side of something that I am passionate about, look out. I’m going to dig in and fight you on it.”
Thankfully we wrapped the debate in a calm civil manner. Chet concluded, “I hear what you are saying, I’m just bringing things up to review.” We compromised and activated a snapshot on one of the front end servers for testing purposes. Tomorrow I’m going to be more diligent. I’m going to audit my backup settings and take a long overdue look on how we can improve on our backup strategy. Ironically, if I had been more diligent, we wouldn’t have had this uncomfortable debate. However, now I have a great opportunity to learn and grow. Therefore the debate was healthy and successful.