Software Engineering vs Engineering.

 I've spent 15 years writing code. And in that time I've grown increasingly aware of some issues with the field. We have come a long ways in 15 years. Standards and practices have evolved. And a lot of engineering concepts have been applied to computer science and software engineering in general.

But, I find these practices are very front heavy. By which I mean, we've gotten good at creating the foundation; an initial version of a product. And that is where it goes downhill. We, frankly, do a shit job from an engineering perspective over the longer term.

I lay the blame on 2 complementary issues; a lack of planning for change and a lack of accepting how much things have changed.

You see, most other engineers either design something for a specific use, or design it to be adaptable within a range of uses. When it reaches a point where it exceeds the original specifications, it is often accepted that the engineering process needs to begin anew. That doesn't happen often enough in software engineering.

This is a big reason behind Apple's initial success. Planned obsolescence allows engineers to restart the engineering cycle. Sure, if you're forcing it every product cycle you're also perhaps taking things a bit to the extreme. This however is what made the iPhone possible. It is also, incidentally why I hate Apple products. Sure, they replace hardware, but you don't buy the phone for the hardware. You're really buying it for iOS and the App Store. And those have not been obsoleted from the beginning. And it shows. 

It is the same problem most software faces as well. Maybe they iterate through products well in the beginning. But, eventually you get a hit. And then you get invested in it. And then the business is afraid to innovate. So, they start pushing the envelope. And the frequency of bugs and vulnerabilities grow disproportionately with the amount of new code. Every cycle the amount of technical debt grows. Which has the opposite effect to what it should have. Instead of seeing the ballooning ball of bloat for what it is, the metric that gets focused on is how it becomes increasingly harder to throw it away and start fresh.

Every cycle adds more features, more users and thus more risk in retiring it. Every cycle, the effort to replace it becomes bigger and the amount at stake grows as well.

In my experience, we also have a tendency to over estimate the effort of rewrites. We discard how much of the original process was learning vs raw effort. And ignore how new technologies, frameworks and languages make the problems easier.

I think, in the ideal world, you throw away every 2-5 years. Maybe you keep a skeleton crew on maintenance mode if it is an important product, and that is it.

My background on this? I wrote software for my wife's business and still write software for my brother's. Every few years I start fresh with a new language or framework or something that seems interesting and like it would get the job done. And every time it has gone faster and gotten better. I can rewrite the backend of my brother's app in a weekend now. It isn't a big app. But, it took several weekends the first time. And the code was MUCH worse.

With every iteration is also gets increasingly modular and extensible. As I said above, I've learned where it is common to spend time. Which areas are most likely to change. I can now anticipate where to put hooks in my code to support the future. So, more and more of my changes are natural extensions of the existing code rather than things being hacked in which are causing more downstream bugs.

It is a mix of things really. On the one hand, the back end has remained a .Net application and my skills have grown a lot there. I make extensive use of reflection and other meta-programming features. Attributes for example. And I use them in intelligent places where I only pay the price for reflection once or very infrequently. I use code generation in other areas so I can get something compiled. And these improvements weren't feasible before I had mastered the domain I was working in. And you don't really get there if you never start over.

Companies can mitigate the pains by leaning on smaller units of deliverable work (Functions or Microservices as examples). But ultimately, I think companies need to understand what made them successful in the first place. And it wasn't riding out the same software or platform for decades.