Building infrastructures where the systems are completely hidden from the people building applications is a very powerful thing. Consider what it takes to create an artificial intelligence and machine learning platform. It isn’t necessary to build the actual machine learning platforms, learn how to build models or learn neural networking computational methods. Hidden complexity makes it possible to simply apply data to training sets and get results.
Hiding complexity enables faster innovation because these abstractions hide incredibly complex thinking. This pattern is most apparent in the public cloud industry, where cloud and SaaS providers serve easy-to-consume services and APIs to their customers. This hidden complexity– or cheap complexity problem– makes me think of Jevons Paradox.
Jevons Paradox
Jevons Paradox states that as the per-cost unit of consuming a resource become less expensive, the consumption of that resource actually increases. The economics of reducing the per-cost actually increase demand and ultimately increase the utility of that resource.
For example, every time we make cars more efficient, people drive more and ultimately use more gas. I believe the same thing is happening in cloud technologies with complexity. It’s a Jevons Complexity Paradox.
Jevons Complexity Paradox
We all know that complex infrastructure systems are fragile and unsustainable. They are difficult to change, hard to build on top of, and risky. Yet, by consuming services offered by others (the Jevons Complexity Paradox) we’re able to build very complex systems without the apparent cost of adding complexity. This is because the abstractions hide that complexity.
Hidden complexity has become inexpensive, leading some to consider it low risk. The way we think about building systems in general has changed. This belief has evolved because sometimes we don’t understand or don’t see the complexity behind the scenes. In many cases we expect the service providers on which we rely to bear the cost of complexity, for which we willingly pay a marginal cost. We’re happily outsourcing the management of the complexity to somebody else with the expectation that that complexity will not come back to bite us.
Unexpected Impacts of Outsourced Complexity
We have seen examples where outsourced complexity impacts our systems in unexpected ways. Consider the recent AWS Cloud failure, where one system failure cascaded into other system failures across their network.
It started from a simple IT challenge: extending server capacity past the threshold. However, this event caused outages across the internet. Primary services were impacted by cascading system failures. As subsequent services were impacted, it showed just how interconnected all of our technology stacks have become.
Ultimately, this failure proved that we are not aware of and cannot anticipate the ways in which our technology stacks are connected. Most of the time, those interconnections are completely invisible to our day to day environment. The risks posed by these interconnections only surface during failures or security violations.
What is the Breaking Point of Hidden Complexity?
The purpose of this post is not to throw stones at complexity. In many ways, the services and the cloud models have enabled us to build fantastic systems.
But hidden complexity prompts us to consider the following:
- Have we created a positive feedback cycle for complexity? Do we accumulate complexity because it’s hidden or inexpensive? What is the breaking point?
- WIth hidden or outsourced complexity, when does that debt come due?
It’s possible that there is no breaking point. Perhaps you the systems you build are isolated and abstracted. Maybe you build amazing protections against the types of faults that we inherit through our dependence on an external service.
Personally, I believe there is a breaking point. One of the ways this breaking point surfaces is the dependency on ever faster innovation and release cycles. As an industry, we are becoming incredibly dependent on maintaining and keeping up with the latest patches, changes, and updates. The window in which we can expect technology to work has steadily decreased from years to months to weeks.
Hidden Complexity and the Shrinking Systems Decay Time
Producing repeatable results requires understanding the huge list of interconnected tangle of APIs, libraries, modules and operating systems. If you don’t constantly update, patch, and keep up with your software in a very narrow system window, then your software decays exponentially. It doesn’t take long for the software to decay to the point where starting up an application that hasn’t run for six months is expected to fail.
In the Jevons complexity paradox model we might expect that decay time to continue to shrink. To keep these systems working together they must be maintained within a very limited window so the system and the API work as expected. Without this constant maintenance synchronization, the systems can’t continue to respond to the hidden complexity. If this happens, they won’t innovate.
Real Talk
The challenge isn’t having complex systems that we don’t have to worry about maintaining. The challenge is about the dependency that we create across all of those systems. If we depend on something that depends on something that depends on something, the complexity as a system view still accumulates. The bill for hidden complexity will still come due.
The Jevons complexity paradox questions if this complexity is a system issue, or if it can be isolated into just a single organization’s scope of concern. History tells me that it’s a systems problem. It tells me we can push systems out of balance very quickly by not factoring in, hiding, or ignoring complexity. But history has also shown me that how things surface in cloud and cloud economics is very difficult to predict.
We have a real enigma on our hands. What happens to all of that complexity? Are we creating systems that build complexity ever faster instead of systems that limit complexity over time? What are your thoughts on the Jevons complexity paradox?
