Document Everything: Designing Operational Systems That Survive Handover

An exciting new role
I once received an offer to join a team working for a client evolving in an environment I was unfamiliar with. It would be the first time I’d work as a Technical Lead (TL) for this kind of project, so I was a little anxious about ramping up quickly and delivering valuable outcomes to show that I deserved the position.

It turned out that the team was extremely limited in size: along with a non-technical project manager, I would work with a new hire who had recently joined the team. That’s about it. As TL, I’d be responsible for taking all major technical decisions affecting the client today and, in the years, to come. As the only experienced engineer on the team, I’d also be responsible for, about everything else on the technical side.

10 days to learn everything…
Well, at least I’d be able to overlap for almost a month with the previous TL before he would move onto a new project himself. This should be enough to ramp up and become familiar with the main aspects of the client’s infrastructure.

Unfortunately, the month quickly turned into 3 weeks which turned out to be barely 10 days. 10 days to learn everything there is to know at a client that had been around for 10+ years, where countless engineers had contributed custom - now legacy - code, where the core infrastructure hadn’t been upgraded in years and had run out of official support. I’m sure you can imagine that task looked daunting.

Just figure it out!
Over the next several months, I painstakingly learnt to navigate the client’s environment, fixing bugs reported by end users on applications I had never heard of before, resolving network outages spanning multiple datacentres while trying to understand how everything connected together. I spent most of the remaining time upgrading the client’s system to slightly more recent technologies to make it more maintainable. It was an amazing learning experience: I came out with a deep understanding of pretty much everything that made up the client’s infrastructure, lots of that new knowledge would eventually be reused in my future career.

Everyone seemed happy
So, was it worth the effort? Well, it depends on who you ask: If you ask me, I actually had a great time learning about all these things. It was not super-efficient but I enjoyed it. I think it was worth it!

If you ask the previous TL, he was able to start working on his new project within days of me joining his old one, so I assume he’d see this as a win!

If you ask the client’s end users and main point of contact, they were very satisfied with my work, although it’s hard for them to imagine what could have happened in a different situation, so they’d probably say it was fine.

That’s it?
That’s it then. Let’s move on? Well, not yet. As much as I enjoyed the experience, I was unable to quickly resolve emergency situations the first few months after joining the team. I had to find out about convoluted aspects of the infrastructure that didn’t really provide me with much reusable experience. I ended up spending a lot more time discovering and learning about the client’s infrastructure than actually improving it. A lot of time was spent trying to figure stuff out that I could have spent upgrading infrastructure, fixing bugs or creating helpful workflows for users.

Documentation to the rescue!
So, what could we do about this situation to avoid it reproducing?

Obviously, the previous TL could have stuck around longer, but then we run into staffing issues, and this doesn’t really get rid of the bus factor which is a major concern for most of these small projects where knowledge can get lost very quickly when key contributors are lost. I also had no control over the staffing choices being made, so no point in digging on that angle further.

Instead, I decided to apply one of the major learnings I had acquired in a previous project: DOCUMENT EVERYTHING! Indeed, if I had had access to adequate documentation, I would have been able to ramp up on my own much faster, and I would have been able to resolve problems much faster as well.

Throughout the 18 months I worked on this project, every time I learnt something new that was non-obvious and potentially tricky to discover, I would document it the best I could. The aim being that people would later be able to find out about these explanations and hopefully be able to resolve issues faster than I did.