"All models are wrong; some are useful."
We have all heard this. If there's ever a leaderboard for most-mentioned data science quotes, this one probably takes the silver medal, after "Correlation does not imply causation."
There's a parallel in software development:
Both scientific modeling and software development are blends of extremely technical yet creative endeavors. They can be very sophisticated and require a lot of craftsmanship and attention to details. They both take years of experience and learning to master.
However, none of this matters if the end product is not useful. Utility is the ultimate fitness function.
Often, to be useful, we have to solve complex problems. Real-life problems are almost always complicated: there are several factors at play, and webs of interdependencies. There are variable scenarios, edge cases, and exceptions.
So we need to think very hard to devise a complex solution suitable for a complex problem. But is that enough?
The Cost of Complexity
The capability to build complex solution should not be taken for granted. It's not something you can pick up overnight. In addition to learning a broad scope of knowledge (algorithms, data structures, infrastructure, systems engineering...), you'd also need lots of practice and experience. A functional, complex solution is a reflection of a lot of thinking and ingenuity. One should be proud.
But months after launch, one issue will arise. A complex solution that took a lot of ingenuity to build will also require a lot of ingenuity to maintain.
That's the cost of complexity. It's cognitively expensive to build. And the cognitive overhead will never go away.
Functional and Maintainable: The Two Pillars
Go is boring and that's fantastic. I hope this book teaches you how to build exciting projects with boring code.
– Jon Bodner, Learning Go
There are many tenets we have as software developers. Clean code. Modularity. Separation of concerns. Test-driven development. Reproducibility. Reusability. "Don't Repeat Yourself (DRY)". But then also "Write Everything Twice (WET)". Different people have different principles held firmly.
But I think we can boil it down to two pillars: functional and maintainable.
- Does it work?
- Can it be maintained?
These two are the most critical requirements. Good software should solve a problem (functional). It should work. And it should continue to work and be expanded. So the second pillar, maintainability, kicks in.
Complex Solution is a Misallocation of Cognitive Overhead
Since complex solution is expensive to build and expensive to maintain, it's "a gift that keeps on costing". This is very unideal. We want our software to be useful. But if it is useful, it'll get used, a lot, which will forever incur high maintenance overhead.
Instead of shipping a complex solution, which already requires thinking very hard, we actually should think even harder, to a point where we can ship a simple solution.
Making a complex solution look simple, now, that's the ultimate ingenuity. But it's a different kind of ingenuity. By definition it's not flashy. It'll look straightforward. Everyone will be able to understand it. It's almost... boring.
The Paradox of Simplicity
And because it takes extra effort to make a complex solution look so simple, and because its simplicity means such effort will almost certainly be overlooked, there's little incentive to walk this path. Instead, present a complex solution that's visibly clever, and be admired, appropriately, for the cleverness.
This is very similar to the "Prisoner's Dilemma of Communication" I had described before. In both cases, the individual incentive works against the collective good. Just as withholding information might benefit an individual but harm the organization, building visibly complex solutions might benefit the engineer's reputation but burden the entire team's future.
The Art of Invisible Engineering (and the Art of Recognizing It)
Perhaps the highest form of software craftsmanship is making the complex appear simple. It's a thankless art – the better you do it, the more invisible your effort becomes. But to well-trained eyes, the simplicity on the surface belies the deep thinking underneath. And I firmly believe within every high functioning, highly technical software companies, there are mechanisms to ensure the mastery of turning complexity into simplicity is recognized and rewarded.
After all, the goal isn't to think hard forever. It's to think very, very hard now, so that everyone (including our future selves) can think very little later. Because we will have other new problems to think very hard about.
