If you’ve ever tested software and had to report an unexpected problem back to the developer, then you’ve probably heard this frustrated exclamation at least once:
“But it works on my machine!”
What does this mean? Why is it a problem? More importantly, how can we help avoid future repeats of this frustrating phrase? In this article we’ll explore these questions and more. So, hold on to your mouse and keyboard, and let’s explore the world of personalized machines!
Where are we and how did we get here?
As recently as 50 years ago, when someone talked about a computer, they meant a whole room dedicated to diodes, transistors, capacitors, and a whole variety of rainbow-colored cables. Most companies couldn’t afford one, let alone multiple. These machines only had one user at a time, and had to be shared between dozens, sometimes hundreds of users, and programming on them was slow and tedious (raise your hand if you remember using punch cards). Users and coders alike never had to worry about something ‘only’ working on their machine, because for all practical intents and purposes, ‘their machine’ was the only one they would ever see.
Fast forward to 1979, when two important developments first coincided: the Apple II, a small portable computer, had been available for a few years, and a new software program called ‘VisiCalc’ caught the eyes of businessmen. In fact, VisiCalc proved so invaluable for managing the often-tedious job of manually writing and updating spreadsheets, that many companies used it to justify completely dropping all their old technologies and buying an Apple II, just to be able to access the spreadsheeting software. Within just a few years, somewhere between a few hundred thousand to up to a million PCs were on the market and running VisiCalc. The entire business ecosystem had completely changed.
Keep going to the present, and the situation, while more complex, is still fundamentally the same. Sure, our computers can run more than just a simple spreadsheet program, and our graphics are a lot more powerful (take a look at any recently developed video game if you doubt me), but for the most part people expect a one-to-one mapping between machines and users. A user for every machine, and a machine for every user. I don’t have to stop working on my latest blog article if my coworker down the row wants to send an email, and vice versa. The system is affordable, powerful, and convenient. What’s not to love?
Well, the introduction of personal computers led to an unexpected problem, one which continues to cost countless hours and dollars of lost productivity, product defects, and delayed timelines: the “it works on my machine” dilemma.
Before, programs only ever had to run on a single computer. Now, they have to run on dozens, hundreds, thousands, even millions. And the situation isn’t even as simple as designing code that’s cross-platform compatible with different hardware. My environment consists of various stacks of software which, in theory, are isolated. But in practice, a totally different program running on the same box can break registry values, overwrite environment variables, corrupt memory, and wreak all sorts of other havoc. Guaranteeing software works consistently (let alone correctly) can become a nightmare, and except for some of the most rigorously developed and tested code, is effectively off the table.
A Personal Example
In my role as an application engineer at BTC, I’ve had to deal with the “works on my machine” problem more than most. I’ve worked with countless people who had to deal with failing CI pipelines, broken model simulations, and inconsistent test results that simply went away as soon as someone else followed the exact same steps they had taken.
Perhaps the most dramatic example I’ve worked with, and one that will make my point clear, was related to a CI pipeline at one of my clients. They were testing a series of models, and (seemingly at random), their pipeline would throw a Java error and crash on a different model almost every time they ran it.
I started the case by making sure to clean up anything in his environment that might have been causing issues. We went through the registry, and also deleted any temp folders on his computer, but the issue was still popping up. I tried downloading the exact same versions of all the tools he was using, and even ran his pipeline script on my computer, but I still couldn’t reproduce the issue. After many hours of technical debugging, we decided that, besides his computer being cursed, the issue was simply worth too much hassle to try to track down, and we developed a workaround solution that quickly resolved all the issues and brought the pipeline back online.
Was there anything that this client could have done to prevent these issues? Hindsight is always 20/20, and maybe he simply got unlucky with his machine. Still, there was one thing he could have done that would have made something like this far less likely to occur in the first place…
Introducing Docker
You might have heard of Docker. It’s been all the rage amongst DevOps ever since it was first publicly released in 2013. But what is Docker? What makes it so useful? And how does it help solve the “it works on my machine” problem?
Docker is an OS-level virtualization engine that lets you package, deploy, and run isolated containers. That’s a lot of jargon! Let’s break it down a little before we continue:
- OS-level: In older computers (think really old, like “this-computer-takes-up-a-whole-room” old), you would only ever run one program at a time. This meant that the program was fully responsible for everything the hardware did. Mouse and keyboard? The program needs to implement device-specific IO keylogging, buffer, and translate the results into an action that corresponds to what you actually want to do. Graphics display? Any output had to be written directly by the program to the screen (if there even was one), or somehow printed out and made available. This was before the era of drivers, graphics engines, or even dedicated GPUs. The result was that even a very simple program had to implement a bunch of tedious and hard-to-maintain code that ended up being really similar to what other programs were implementing, because they were controlling the same device. Plus, the code wasn’t very efficient or optimized, since it wasn’t related to the program’s ‘particular’ functionality. And, if the user wanted to run a different program – even for just one minute to do something on the side – they would have to completely shut down the computer and swap out the program floppy disk for a different one. To get around these difficulties, computer engineers quickly realized the need to develop an ‘operating system,’ or OS. This was code that ran directly on the hardware (just like before), but instead of ‘doing’ anything, it provided a common environment that could be shared by multiple programs. This way, rather than an excel spreadsheet needing to read, buffer, process, and write keystrokes to a display, it could simply ‘receive’ input events, code only the relevant spreadsheet functionality, and ‘send’ output events to display results. Furthermore, because the OS was handling any common functionality to all programs, it became the ‘main’ program that the computer would run, and then it would ‘spawn’ multiple processes (programs), so that the user could switch between them at-will.
- Virtualization: OSes were powerful. But a problem quickly began to rear its head. How could the user make sure Program A wasn’t trying to maliciously tamper with the data that Program B was running? Clearly, even though all the programs were co-existing on the computer at the same time, some kind of cage had to be put into place around each program to make sure that it couldn’t just break a different, unrelated program. From the program’s perspective, it has to think it has sole, exclusive control of the machine when it’s running, as if it had all the memory and CPU for itself. The solution was for the OS to create ‘virtual’ hardware. The program, when reading and writing memory, or executing instructions, would run ‘as if’ it were accessing the actual memory and CPU. But the OS would intercept these actions and translate them into a unique section of memory, and make sure that any CPU results weren’t overwritten by a computation running in a different program. This virtualization, where the program talks to the OS as if it were talking directly to real hardware instead, isolates code sets and prevents accidental or malicious cross-talk. Many advanced security exploits today try to break this virtualization, attempting to gain access to data that they shouldn’t and run commands that take over the computer.
- Engine: No, I’m not talking about your car engine, although the notion is similar. In this case, an engine refers to a complex piece of software that ‘runs’ something. In Docker’s case, it’s a program that runs other programs. And just like how an engine requires a certain type of fuel, Docker has a special syntax, called a Dockerfile, that lets you specify how to ‘drive’ it, or how to get it to execute the programs you want in the way you want them to. Putting all of this together, we can talk more concretely about what Docker does. As an OS-level virtualization engine, Docker lets you specify and run programs (it calls these ‘containers’) by taking advantage of the virtualization software already present in the OS. In other words, Docker acts like a mini-OS on top of the local OS, to let you run programs without having them interfere (or be interfered with) by other programs.
What? Doesn’t the OS already do this? What’s the point in running Docker?
Well, in theory the OS separates out programs. In practice, people quickly realized that it’s really useful to let programs talk to each other. Modern day OSes are complex ecosystems that, for practical purposes, enable quite a bit of cross-talk between programs. These are well-defined (a program will never be able to overwrite another program’s memory, for instance!), but there’s a lot of different ways one program could affect another program’s functionality. Docker purifies your system by isolating your ‘normal’ workspace, including all of the overhead chatter, from your ‘Docker’ workspace, which dramatically minimizes the number of interactions you have to worry about. A typical computer will be running anywhere from 200 to upwards of 500+ background processes at a time, most of which are irrelevant to what you want to do at any given moment. Have you ever heard of crss.exe, for instance, and could you explain how that would impact some code you want to test? I couldn’t. By enforcing strict isolation between these hundreds of processes and the handful of things you want to actually run, you cut down the number of possible interactions (and interaction-related bugs!) by 90% or more. This is the value-statement of Docker.
What is Docker useful for?
So, now that we’ve talked about what makes Docker useful… what is it useful for?
Obviously, you don’t want to run everything in Docker. Either you’ll end up just duplicating your OS (and then what’s the point?), or you’ll break the ability of programs to talk to each other and lose out on critical functionality. Plus, you have to tell Docker exactly what you want it to run and how, and that can be time consuming and tedious. What’s the sweet-spot that makes it worth the investment?
I started this article by talking about the ‘it works on my machine’ problem. Oftentimes, it’s fine if something only works on my machine. Collaboration is often more about sharing results than the processes that got you there, and I’m not going to be heartbroken if the script I made to process Excel data only works on my computer because, well, I’m the only one who’s ever going to run it.
But sometimes we don’t just care about results. We care about consistent results and consistent processes, because we’re expecting many people to interact with them, and want to make sure there’s as few misunderstandings as possible. Today in Automotive, we often see that it takes teams of dozens, sometimes hundreds of people, just to create a single component of a vehicle. If we didn’t have a way to ensure that information flowed smoothly, all the gears of this gloriously complex machine would immediately come to a halt. This is where Docker steps in.
Consider one of the most common development paradigms. I’ve edited some existing code (or a Simulink model!), and I want to make sure that my changes haven’t broken anything. My team has been hard at working testing our software as we develop it for months now, and we have a comprehensive test suite of hundreds (or maybe thousands) of tests that we can run to validate that my changes are good to merge into our active release branch. Rather than clogging up my machine with having to sit and execute these tests for hours (or even days), I push my changes to another computer, which spins up all my software and runs my tests on the cloud, so that I can continue working, getting the test results and any feedback as soon as they’re ready. This is an extremely common use-case, and it often goes under the name of “Continuous Integration” or CI Automation.
Now, it would be awful if my CI pipeline either broke (oftentimes dozens of people are relying on the machine!) or gave inconsistent results (how can I debug something that I can’t reproduce?). Generally, my development stack only consists of a handful of programs – a dev environment like MATLAB/Simulink, test tool like BTC EmbeddedPlatform, and maybe some virtual simulators like dSPACE VEOS. I want these programs to communicate with each other, but I don’t want anything else to get in the way! This is the perfect use case for Docker. By defining a simple Dockerfile, I can create a container which runs these programs (and only these programs), ensuring I have a consistent, reliable environment for my testing. Had my client from earlier been running Docker, perhaps he never would have had to deal with any of the inconsistent, impossible-to-reproduce results that necessitated an alternative solution in the first place. Docker takes you from “it works on my machine” to “it works on all our machines!”
Conclusion
Modern tech stacks are incredibly complicated, and require interactions between hundreds of different programs. Oftentimes, however, we’re only interested in a handful of programs at a time, and want to minimize any possible interference from the other hundreds of programs in order to produce consistent, reproducible results. Docker helps us all but guarantee this by curating a minimal environment containing only the code we’re interested in, along with its dependencies. In Automotive, this ensures robust CI pipelines and pain-free test executions, making the “but it works on my machine!” problem a thing of the past!
