Why video game coders don't use TDD, and why it matters
Whilst working on Sol Trader, I’ve written many unit tests for my code. Many of these tests have been written before the code itself, using a practice called Test-driven Development (TDD).
Test-driven development is the practice of writing a failing test in order to specify the behaviour of a piece of code, then writing the code to satisfy the tests afterwards. We then refactor and improve our code from there.
In most programming environments, people are talking about TDD and trying to practice it. It’s even become an essential bullet point on job adverts, as if not practicing TDD makes us fundamentally worse programmers (which isn’t true, by the way.) TDD seems to be everywhere.
Everywhere, that is, except the games industry.
Why is this? Is it because TDD is flawed in some way, or simply not applicable here, or that practices have grown up to counter the need for TDD?
The benefits of TDD
Let’s begin to answer this by looking at the specific advantages TDD gives us:
-
It forces usage-first coding. TDD represents another client for our code, independent from our production code. It asks specific questions of the codebase to ensure that it’s correct. It forces us to think about our code from the point of view of ‘what it does’ first, rather than ‘how it works’. This can often lead to surprising realisations about the code we actually need, and prevents us from writing spurious code we might think we need, but actually represents wasted effort.
-
It helps us minimise code size and complexity. If we adhere strictly to the principle of only writing enough code to satisfy the test, then our tests should capture every possible path through our code. Additionally we only have enough code to satisfy the exact problem we’ve used tests to define - this is important because code is a liability, not an asset. The same is true of code complexity. If we find ourselves writing reams of tests to satisfy a particular piece of code, that code is too complicated or very risky and a prime source of bugs.
-
It provides design feedback. Well designed code is easy to test. Therefore, when initial tests become harder to write, that might be because our code isn’t well designed, or well understood. Typically, pure functions are easier to test and reason about as they have no side effects: testing these functions is very easy and therefore our code tends to gravitate towards them.
-
It allows for verification over time. I’ve listed this last, as I don’t see it as the most important benefit of TDD. As I refactor my code, tests become simple enough to be self evident and the tests can be safely deleted. At the system level, ensuring that a complicated system continues to work after significant change is useful, but considerable effort is required to write tests that check an entire system safely. Poorly written tests give a sense of false confidence to those new to the practice, which can be highly dangerous - our tests can start lying to us. In practice, a few end-to-end tests to verify basic functionality of a module are worth the effort, but many more can slow development and provide false confidence.
How games industry experts verify their code
Let’s look at expert coders in the games industry and discover what they do to gain these advantages. Here are some observations:
-
They write the production usage code first. Casey Muratori on Handmade Hero writes his code from the point of view of the usage of the code first, by writing exactly the calling code before defining structures and basic methods. This gains many of the advantages of TDD, using production code rather like tests to discover what the code should do. By implementing a client for the code first, we discover what the code should look like before we write it.
-
They fail fast using assertions. Assertions are conditions in the code that are often only present in developmental builds, causing an artificial exception or crash when the condition is not true. They ensure that the running program is in a good set of known states: code running in the wrong state is a hugely common cause of bugs. As unit tests only check a certain set of known states from the outside, assertions are useful in catching unexpected behaviour that wasn’t initially thought of. As the code tends toward purer functions with less possible states, the usefulness of assertions within that section of code diminishes. They also do not provide design feedback on the code in the way that TDD does.
-
They rely on static compilation to catch type errors. Static compilation is a form of testing. If we are thinking carefully about the distinct types we are using, avoiding Primitive Obsession, then this distinction between types will help ensure that we aren’t passing the wrong things to the wrong functions, or confusing distinct concepts in our code.
-
They use automated testing where the code is risky. John Carmack recently wrote about the value of testing in his essay on functional code:
What are games developers missing?
Games developers have a number of techniques that give them similar benefits to TDD. We see that by writing usage code first, developers get good feedback on their design as they go. Code verification over time is taken care of through judicious use of assertions and using automated tests with risky code.
The area that games developers miss out by not using TDD is in the reduction of code size and complexity. However, in high performance computing, the size of the compiled system and the branching complexity are constant concerns. There’s a real performance penalty through having too much code, breaking branch prediction and accessing memory too often by jumping the execution path all over the place. The fastest and most efficient code boils down to data transformation as functionally as is possible within the obvious constraints of the gaming environment.
If all of this is taken into account, games developers have side-stepped the need for TDD.
However, there are bad reasons to dismiss TDD in games. There’s a perception that games are too ‘emergent’ and complex to apply TDD to. This is false. Games are more deterministic than people think, especially in the inner workings of the code. Moving to a more functional programming style makes this explicit, although often enough so much risk is removed from the code that TDD’s design feedback is less useful.
There are clearly areas in games development where TDD is the wrong approach - games are about ‘feel’ and the ‘experience’, and we can’t test-drive ‘fun’, or test the output of complex interactions of hundreds of entities. Sometimes however TDD is dismissed because we cannot imagine how we might begin to test our code: this says more about the quality of our code than the merits of TDD as a practice.
Summary
Video games devs don’t do TDD for two reasons:
-
The good reason: the best practices in the industry deliver many of the same benefits as TDD.
-
The bad reason: an insufficient knowledge of TDD and good code design can lead people to believe it’s just not relevant to games. The smokescreen of “we cannot TDD fun” can mask a poor understanding of good coding architecture.
In practice, I attempt to TDD much of my low level code, especially my functional core code which is simply transforming data from one type to another. I use TDD where I’m weakest as a programmer: reasoning about pointer and bitfield arithmetic aren’t my strong points and therefore I like to test-drive it!
I don’t use TDD at all for UI testing and where the ‘feel’ of something is important, and for self-evident code.
TDD has helped to teach me about good code design, side effects, the perils of state, architecture, programming in a functional style and the evils of prevalent inheritance-based object-oriented approaches. Perhaps the real value is not in the continued practice, but in the lessons that it teaches?
Share
More articles
The Job Is Not To Build
Startup CTOs or founding developers are the first technical people in the business. It is natural to think your job is to write code and build software. This is backwards.
Your first job is not to build software. Your role is to use your technical expertise to help the startup figure out fast if you have a valid solution to a compelling problem, and then a valid product for a big enough market.
You might do this through building software, but you might not need to.
Here is a story of how I did this wrong, and how you can do it right.
Read moreEaldorlight: A Kickstarter retrospective
It’s now been over three months since the end of the Ealdorlight Kickstarter campaign. I’ve deliberately been taking some time to think and learn from the fact that it didn’t reach the target, and to work out what to do next. Frankly, I was pretty upset that the campaign didn’t make it, and it’s taken a while to get over it.
It’s also taken a while to think through the campaign properly. Some things are obvious in hindsight, and others less so. A lot of post-Kickstarter analysis feels like a stab in the dark. Nevertheless I’ve given it a lot of thought, and these are my best guesses for why I think Ealdorlight’s Kickstarter failed:
Read moreEaldorlight's Kickstarter is live at 4pm today
The sixth of June is a significant day for me personally. In 2004, I spent the entire of the day in hospital. I remember the 60th anniversary commemorations of D-day on the TV in the background, as I sat beside my wife, in labour with our first child. I became a father an hour after midnight on the 7th June; my son becomes a teenager tomorrow.
Twelve years later, in 2016, I spent the entire of 6th June glued to Steam watching and waiting whilst my first game Sol Trader was released to the world. This was a career dream come true: since I started programming at six years old I’d always wanted to create and ship my own games. Sol Trader’s release was ultimately a painfully formative experience for me, which I wrote about at the time and was interviewed about recently in GamesIndustry.biz.
Over the last year, I’ve been keeping busy doing two things. One is to support Sol Trader as much as I can with countless updates and patches. I’ve also been very busy working on a new game, Ealdorlight, a medieval RPG-style take on Sol Trader’s mechanics, with turn-based combat, realistic damage and great graphics. I announced Ealdorlight in March and demonstrated it at Rezzed, strengthening my hope that the idea was a good one.
I decided fairly early on that I wanted to take Ealdorlight to Kickstarter. Sol Trader’s successful Kickstarter was a brilliant experience. The Kickstarter community is one of the kindest, most positive on the Internet. I also needed funding for this game: Sol Trader was self-funded through many long evenings and contracting work, and for Ealdorlight I need a bigger team to realise the vision. It’s built in Unreal Engine 4, which simultaneously saves me loads of development time and means I need a bigger team to pull off the realistic art style I’ve gone for.
As time came near to launch, the first anniversary of Sol Trader’s released seemed an appropriate day to launch the campaign. So today, 6th June 2017, I will spend the entire day glued to Kickstarter as my campaign goes live at 4pm today.
Visit Ealdorlight’s Kickstarter Campaign
There’s plenty more about Ealdorlight on the campaign - head over there and read all about it! A huge amount of work has gone into it, and I’m very grateful for all the support and help I’ve received from the team I’ve put together, and for friends and family who have given me endless encouragements and feedback.
This post is a little earlier than 4pm so that you can watch it go live if you want. Earlier backers get lower edition numberings on some of the rewards, so you might want to be there from the start!
Read moreHow Ealdorlight's story stands out
As we head towards the Kickstarter campaign launch on June 6th, I want to talk a little about the story behind Ealdorlight works.
The basic story stays the same for each game: you are discovered wandering through a remote village at a young age, and realise your destiny is to overthrow the King. However, like in Sol Trader, every person you meet is randomly generated. This means that your real identity will be different every time, and you’ll have to discover it all over again every time you generate a new game.
Handcrafted story in a random world
The trick is layering a great story on top of a generated world with random characters. Building empathy with the main character and his family when all characters are generated is hard, and hinges around being able to hook the story in at the right moments.
My plan is to write plenty of tightly connected story arcs that are triggered on events that happen during history generation. These will in turn trigger future quests the player can undertake. Not all story-arcs will appear in every game: it will depend on how the history generation goes. I will constrain things such that there is always a route through the game, and that players always have a way to overthrow the King, even if that might be easier or harder depending on the starting setup. These story-arcs then should interact with each other, hopefully producing a unique path through the game.
Identity
Ealdorlight is set within a low fantasy world, and there’s no traditional magic. The player gets more powerful through discovering key pieces of knowledge about their past. These insights into of your real past feed directly into your character’s stats, skills and abilities.
I’ve long been fascinated with identity: knowledge of who we truly are affects many areas of our lives for the better. In Ealdorlight I wanted to tell a story which takes this to an almost supernatural level. By removing the player from their birth family, they start as an entirely normal person within the world. It’s only after their early game encounter with the Ealdorlight and the discovery of their past that things begin to change.
Much more on this to come, but in the meantime, here’s a glimpse of our story’s beginning.
Ealdorlight: backstory teaser (updated)
Read more
Ealdorlight Kickstarter on 6th June, Sol Trader 1.3 released
I am now back from some extended time away after Rezzed, both on holiday with the family and training some clients away from home. I’ve released Sol Trader 1.3 today, and set the Kickstarter for Ealdorlight to 6th June.
Rezzed was fantastic: it was great to see lots and lots of people wearing our Ealdorlight crowns. We ran out of crowns on both days, with some creative head displays on offer:
View post on imgur.com
Ealdorlight Kickstarter launch date: 6th June
Yes, I know I said May :) I’ve decided to go for a 31-day campaign, starting on the 6th June, for a few reasons:
- It doesn’t clash with any major US holidays, like Memorial Day. The 48-hour reminder email should go out on the day after 4th July.
- I want to give myself the best chance of success by polishing the combat demo hard. It was great to get such good feedback at Rezzed and I think it’ll be a great hook. I need more time to do that well.
- 6th June is the first anniversary of Sol Trader’s launch, so it ties in nicely with the ongoing Revelation Games story.
I’m excited and nervous about this Kickstarter campaign: my third one to date. After succeeding last time I’m really trying to take my time and get it right.
Sol Trader 1.3 released!
Now that I’m back, I’m able to support a new release of Sol Trader: 1.3 is now finally released after a length beta period.
Here are the highlights:
- You can now chat to characters directly on the right if they’re in the same location as you
- Pirate Chief and criminals are now more likely to try to destroy you
- Fix a bug where you’re not paid enough for a mission
- Dignitaries now fly around a little less than before to make it easier to pin them down
- Inter-faction missions now pay slightly less
- Business trips now pay slightly less
- Taxi missions now pay slightly more
- Talking to your criminal parents will no longer cause them to forget who you are
- Fix crash where a character develops an opinion of the player mid-conversation
- Fix crash when showing GUI for a ship the AI is driving
- Fix crash where a character would attempt to sell a good on a ship they’ve lent to the player
- Can now initiate conversations when paused - will restart the game but at realtime speed
- Fixed the Tiger II achievement
Your steam copy should automatically update. I’ll be releasing an updated downloadable version to SendOwl in the next few days.
Read more