How To: Software Architecture Thoughts (Backend)

My last post was a bit of rant on very high level concepts about how I think software should be architected. It didn't really do much to help with how to get there or how to design for it.

In a past career I wrote a very (VERY) abstract system which ended up being used FAR beyond the original intended scope and to a not so limited degree of success at that.

How and why that system was so successful, I think encapsulates a lot of good architecture. I was also, at best, an intermediate dev at the time, so there was undoubtedly a lot of silly mistakes as well. The core architecture on the other hand was quite solid.

As I'd later come to learn I had been (without realizing it) implementing the Rule Engine pattern. Here is a good article, and another one.

To avoid going too deep into any details, it was a system which performed an evaluation of a questionnaire of sorts and then acted upon the results at different stages in the lifecycle of that document.

Document here is a very loose concept. It had a very particular initial application. Fundamentally though, if you could scrape the data from an external system, generate some questions around it, and then either have a user or another system supply the answers, then anything you could code based on that was possible with this system. I definitely won't say that it would be the best solution to all problems of this nature. I just want to frame why the system was so powerful and so easy to take beyond the confines of its original design.

What allowed this system to work so effectively was a series of what I called "engines". These engines processed a series of contracts which could be provided as scripts or precompiled code.

The engines themselves were relatively lightweight. And, anything which could be evaluated as a part of an existing contract provided a rather trivial extension point. Want to enable a new type of question? There was a contract for that. Need to calculate the output the questionnaire with a different weighting? There is a contract for that.

Now, given the nature of this specific application it is easy to see why this architecture works and makes sense. Each document will have different questions, types of questions, criteria, weighting, etc... And so extensibility is a fundamental requirement. Even the system's scraping logic was just another engine. So, something like adding a new type of data source was also fairly trivial.

Changes to the engine itself could be a bit more involved. At the end of the day though, not overly so. It would generally be about figuring out how to define the new contract, which engine to put it in and where/when. The actual heavy lifting goes on inside of the implementation of the contract. To boot, the code is generally quite simple in each contract because the plug and play nature within the engine meant you could simply create unique implementations for each use case rather relying on branching inside of the code.

I've gone fairly deep into this. And based on the last article you can probably see why. This sort of "engine based architecture" makes the division of code rather simple in terms of determining where and how to draw lines in the application. On top of that, the ability to essentially "inject" the implementation from elsewhere means that it can be possible to move the code closer to the point of execution, potentially eliminating a lot of network hops if you end up with a Microservices or Hybrid architecture.

However, even in a monolith, this sort of design will help draw those lines between teams and internal services. 

Two things to note:

1. Not every part of an application is a good fit for this pattern
2. This is not the only solution

The point I'm trying to make is that there are design patterns out there which intrinsically make it easier to deal with future upgrades or incremental rewrites. Down to even the DB layer (which while I mention it a lot, it is probably not a great idea to wind up in a place where you need to change your DB layer).

For frontend, the equivalent would be building a shell which is able to load pages dynamically and even load pages from other frameworks. I think the UI side is harder. Though, I'm also much more versed in backend code, so I could be wrong.

As a developer, I want to be able to focus on code. I also want to be able to innovate and evolve the solutions I work on. Making decisions that tie you to an architecture stifle that. Making decisions which abstract away the real code stifle that. There are probably approaches which are better (using other metrics). For me, the two objectives of any architecture are that it should allow for evolution and innovation and that it should, at the end of the day, be developers working with real-world code.