Software Happens – A Challenge for the 21st Century

“My toaster says it’s “downloading a firmware update. Please wait…” What is a firmware update and why does a toaster need one? And, why is my toaster talking to me in the first place? All I want is a lousy piece of toast!!!”

We were startled years ago to find that every day appliances like washing machines, refrigerators, and even toasters contained microprocessors. For those of us old enough to remember when our appliances were just what they appeared to be, nothing more, nothing less, it came as a pretty rude surprise when a new generation of appliances, each containing enough embedded computing power to replace one of the flight computers on the space shuttle, started showing up at Sears. For those of you too young to remember this transition, no, I am not some old curmudgeon who hates the advances of technology in everyday life. In fact, I’m amazed at what appliance manufacturers have been able to make our everyday appliances do, by employing software. I’ve been in the industry for about 30 years now, so you’d think there wouldn’t be many surprises left for me, but it seems that the opposite is the case, more often than not. I never cease to be amazed at what creative, self-disciplined people can make using software.

Not all curmudgeons are technophobes – A scenario

OK, I guess I am a curmudgeon, but being one doesn’t mean you have to hate technology. In fact, that’s actually a pretty outdated stereotype. Not sure if you believe me? Fine. Go to your nearest Starbucks and sit there for an hour or so. In that span of time, you are nearly certain to see at least one person who is clearly well past retirement age wearing a Bluetooth headset.  This person might even be conversing with two or three others in the same age group, also wearing headsets. One or more of those headsets will probably be paired with a high-end smartphone and it wouldn’t be much of a stretch to imagine that at least one individual in this group also has a net-book or a tablet. Case closed, but since you’re already there, might as well keep watching…

As you observe said retirees, each with a microprocessor sticking out of his or her head as if it were a fashion statement, consider this: Possibly, even probably, at least one of those folks is also sporting the latest in advanced pacemaker technology implanted beneath their skin. That person is “wearing” an embedded device (gives new meaning to “embedded,” doesn’t it?) containing several microprocessors running tens of thousands of lines of code. Unlike the headsets, smartphones, and tablets, this device is not a fashion statement or a convenience. It’s essential to that individual’s survival.

But, wait… There’s more!

You may have to use a little imagination for the rest of this scenario, but I hope you’ll agree it’s not much of a test of our imaginative abilities…

The same individual may leave the store in a few minutes and get into a late-model, high-end automobile. For the sake of our scenario, we’ll say it’s a Mercedes S-class, running nearly 100 million lines of software code distributed across a real-time interconnected network of more than 70 microprocessor-based electronic control units (ECUs).*  In just a minute, some of that software, namely the code running in the car’s radar-based automated collision avoidance system*, will decide to tell the software that controls the anti-lock braking system to execute a controlled, straight-line skid – without any input whatsoever from the driver – in order to avoid a collision with the tractor-trailer rig slowing down just ahead.

Why didn’t our driver react in time? Because he was distracted by taking a call through the car’s hands-free integrated phone system that automatically connected to the cell phone in his shirt pocket right after he got into the car. True, he probably shouldn’t have taken that call in the first place, but the software-based voice synthesis component in the car announced that the call was from his cardiologist and he knew better than to let it wait. Probably for the best, since it turned out that his internet-connected pacemaker had just signaled a dangerous arrhythmia it detected in his heart, thanks to the sophisticated software embedded in it. Part of that software was adapted and reused from software originally developed for the ECG monitor device made by the same company for hospital intensive care units.

Caveat emptor

The only part of the preceding scenario that was a bit of a stretch was the part about the pacemaker connecting to the internet and possibly the reuse of code between pacemaker and ECG products. I threw those in to make the scenario more interesting and to illustrate a few points:

1. Pacemakers will, in fact, be able to signal the wearer and his or her cardiologist about dangerous heart conditions as they develop in real time in the very near future*; and
2. Medical device manufacturers will be reusing code across product lines in the near future, if they haven’t already figured out how, despite the increasingly-restrictive regulatory environment.

Why reuse the code? Because it costs a small fortune to design it, code it, test it, and maintain it, and manufacturers of safety-critical products are increasingly finding that they can’t afford to reinvent the wheel in every new product they develop.

I tried but couldn’t figure out how to work an airplane, a piece of heavy agricultural machinery, or an internet router into my scenario without loosing all credibility, but software is all over those devices too. It’s become ubiquitous. So, here’s the $64,000 question (if you don’t know what I’m referring to here or why the question is only worth $64,000, I dare you to ask one of those curmudgeons at Starbucks)…

How reliable is all of this software we depend on every day?

You might be tempted to respond with something like: ‘I have to reboot {insert name of your least favorite desktop operating system here} every stinking day! Oh, that’s just perfect! Why are product manufacturers cramming software into everything from my toaster to my car to the airplane I’m about to board? What happens if the pilots get the blue screen of death on their displays when we’re taking off?’

Thankfully, little, if any, embedded software in these devices is running on {insert name of same desktop operating system here} today. People who develop software-intensive safety-critical products are a special breed and are not given to throwing in every feature they can imagine whenever the mood strikes them while racing to get the product to market, relying on customers to do the beta testing. The software engineers among them are also not interested in developing their software to run on commercial operating systems that were developed using the same philosophy. Instead, most of these folks act and think like engineers, whether they have engineering degrees or not. The public should be thankful that this breed of person is predominant in product engineering, and more to the point of this blog entry, in software engineering. It is these people who deserve the credit for the amazing level of reliability we enjoy in our software-dependent lives today.

‘But,’ you say, ‘I hear about software-related failures in critical products and systems nearly every day!’  While ‘every day’ may be a bit of an exaggeration, this is essentially true. The important thing to remember about this observation is that you only hear about one failure every day or so all over the world. Given the pervasiveness of software in our lives along with it’s unimaginable complexity, it is nothing short of stunning that we don’t hear about 10 or even 100 catastrophic failures every day, and we have our engineers and software developers to thank for that.

Oh, by the way, some of those engineers and developers are also curmudgeons, a quality that should further endear them to us since their curmudgeonly tendencies drive them relentlessly to ask ‘Why?,’ and ‘What if?’ and to make irritating statements like ‘Oh really? You’re wrong and I can prove it.’ If you work or live with one of these people, I offer my sincerest condolences. I also encourage you to talk to my wife for some pointers since she’s developed some very effective coping skills over the past 26 years. While this behavior can lead to products that are late to market and cost more than we’d like, we all know of examples of products that did not benefit from this kind of thinking and can probably all agree that, given the choice, we’d choose the more expensive and late to market product over the other one when our lives will depend on it.

That’s all well and good, but it’s not enough

Despite our best efforts, software development is a monstrously complex undertaking and software-driven complexity doesn’t appear to be abating at all. Embedded software developers, with their real-time operating system kernels, specialized development environments, rigorous development processes, and increasingly rigorous quality standards, are still prone to make errors which sometimes slip into the final product. And while we may all agree that we’d wait longer and pay more for a safety-critical product in a hypothetical scenario, the real world of the 21st century is one in which product manufacturers must:

1. Understand market needs better and faster;
2. Create ever more innovative products to meet those needs;
3. Bring products to market faster;
4. Design, manufacture, and service products at lower cost;
5. Ensure products comply with a growing body of regulatory requirements and industry standards; and
6. Deliver increasing value to shareholders (a requirement that is not always in perfect alignment with the previous ones).

Collectively, these requirements are driving manufacturers to dramatically increase their use of software, both embedded in the product and in the product support infrastructure, to enable rapid innovation and, at the same time, help meet the remaining requirements which can be summed up as better, faster, cheaper, and more reliable. It’s a daunting task to say the least.

So, there it is:

The challenge of the 21st century for product manufacturers:

To drive innovative products to the marketplace better, faster, and cheaper through increasing use of software while ensuring reliability in the face of dramatically increasing product complexity that is the result of greater reliance on software. 

Of all the challenges associated with the increasing role of software in products,  safety and reliability will be the most public aspects of them for the rest of the century and will thus be the primary driver of changes in the way we engineer complex, software-intensive systems and products.

Is this an intractable problem?

Since the challenges are interdependent, they cannot be addressed independently. Instead, we will have to approach developing solutions to these challenges in a holistic manner. That’s where an engineering discipline called systems engineering comes into play. Systems engineering actually began as a formally recognized engineering practice area decades ago, in the days when we were regularly solving intractable problems, like putting a man or two on the moon, bringing the crew back to earth alive, and doing it several times over to prove we weren’t just lucky the first time.

Those challenges were far too large and intractable to approach using conventional engineering methods.  They required a holistic approach that could divide the problem into smaller chunks each of which could be engineered separately while not losing sight of the system as a whole. That’s what systems engineering is all about.  The practice of systems engineering has matured over the years to meet even greater systems challenges. However, some aspects of it have not matured at the same rate as others, leaving engineers with some gaps in the methods and tools they require to do the job right.  And, that’s where I’ll quit on this already too long posting.  Look for future posts where I and others will write about these gaps and the need to advance the practice of systems engineering to a new level of maturity in order to meet the seemingly intractable problems we will face this century.

What do you think? Will software safety and reliability be the driving factors in product engineering in the 21st century?