Some Questions Driving Empirical Software Development
empirical (adj.)
Pertaining to or based on experience, as opposed to theory.
Pertaining to, derived from, or testable by observations made using the physical senses or using instruments which extend the senses.
Verifiable by means of scientific experimentation.
Sometimes I do test-driven development, and sometimes I don’t. But no matter what process I’m using to write code, I want to stay grounded in reality, by getting frequent, low-latency feedback on my work. TDD is one way to bring empiricism into software development, and I think it’s the best way in many cases, but it’s not the only way.
Here are six questions I ask myself as I’m working — on code or anything else. These are the questions that ground an empirical development process.
What problem are we solving?
What is the baseline? What are we comparing our solution to?
How do we know the problem is actually a problem?
How will we know when the problem is solved?
Is there a cheaper solution to this problem, and only this problem?
How much do we trust our techniques for answering these questions?
I don’t always have perfectly articulated answers to these questions. But by asking them, and always looking out for better answers, I keep myself from going too far off the rails.
What problem are we solving?
In other words, what’s the goal? Why are we working on the system at all?
This seems like it should be a simple question to answer, but often, software gets written even though no one can clearly articulate what problem it’s meant to address.
Step 1 is to say, “I am solving this problem. The problem is X and therefore we’re going to do Y.” I have seen so much software made where no one ever said that. No one ever wrote that down. And then we have a whole system, and no one ever said what problem it’s supposed to solve. If we’re not solving problems, I have no idea why we’re in this room.
Be honest in your answer to this question. Any attempt to prevaricate will get shot down by the next question in this sequence, so just be real.
Honest answers might include:
“This project idea is burning a hole in my brain and I can’t think about anything else.”
“I just want to learn Kotlin, and this is one way to do that.”
“We need to migrate to Next.js because it might solve all our infrastructure problems, and devs are complaining about legacy code, and oh god I just don’t know what else to try. At least if we switch we’ll be able to say we’re using a modern JS framework.”
These are not necessarily bad answers. In the right context, they might represent problems worth solving!
More typical answers might include:
“We’re fixing this bug — here are the repro steps.”
“We’re implementing password reset for users with email addresses.”
“We’re improving performance.”
“We’re tidying up this messy function.”
What is the baseline?
In other words, what are we comparing our solution to?
In many cases, the answer to this question is self-evident: the baseline for our work is the current state of the system.
In other cases, especially when starting a new project, the baseline isn’t obvious and needs to be stated. For example, if you say “I want to write a high-performance statistics library,” the next question you should be prepared to answer is “high-performance compared to what?” What’s the fastest library out there right now? What are you trying to compete against?
Answering this question can change how you look at the problem. Maybe it turns out that there is a very fast library already available, but it doesn’t have all the features you need, or it’s poorly documented, or the API sucks. In that case, performance might not be the right focus! Maybe you can wrap the existing library in a better API, and thus solve the real problem.
How do we know the problem is actually a problem?
This is the step that many programmers skip — and TDD stops you from skipping it. It’s all too easy to see a “bug” in code you just typed and swat it immediately, before you verify that it is a bug, and not just a piece of lint.
If you think there’s a case where the code won’t behave correctly, consider proving so with a test before you change it. If you don’t do this, at best the code will be more complicated than it needs to be. At worst, you might introduce new bugs that weren’t there in the simple code!
How will we know when the problem is solved?
This is another question that TDD answers for us. If we pinned down the problem with a test, then the problem is solved when the test passes.
In the absence of TDD, we should always have some way to get feedback on our work and verify that it had the desired effect. Maybe that feedback comes from other programmers, or QA, or even users — but it has to come somehow.
Is there a cheaper solution to this problem, and only this problem?
This question steers us away from writing unnecessary code, or doing any other unnecessary work. I do not mean to imply that less code is always better. Rather, easier-to-write code is better, other things being equal. Don’t reject the quick and dirty solution until you know it isn’t good enough.
The pedantic way to answer this question is to write the quick and dirty code first, and then to identify everything that’s wrong with it and to fix those problems one at a time. Quick and dirty solutions often have many problems: they might be hard to understand, or might have performance or security issues. The point is, the new problems are separate from our original one. By taking one problem at a time, we ensure we’re solving real problems, and not imaginary ones.
However, the pedantic way is not the only way. Sometimes, you can foresee in advance of coding that the cheap solution will be irreparably bad in some way, and will need to be ripped out and rewritten. In that case, there’s no reason to write the cheap code. Just fix the problem “in the design phase.” Whether to do this is a judgment call — as is everything.
How much do we trust our techniques for answering these questions?
If we’re relying on tests to tell us if our software works, our confidence in the code is limited by our confidence in the tests. That can be problematic, because tests aren’t perfect. Tests can be invalidated by mistakes in setup or assertions. And no matter how many tests we write, there will always be gaps where bugs can sneak through.
Every software verification technique has limits. This is true of static types and code review as well as tests. We can try to compensate for the weaknesses of each technique by using them in combination, and that often works very well, but we’ll never reach perfection.
I don’t worry about this too much. That’s because there are no wrong answers to the questions I’ve listed here. There’s no need to reach for some unattainable, perfect process. The important thing is to keep asking yourself the questions, and to trust that with enough repetition, your process for answering them will improve.