We've been rethinking our testing process for Kerbal Space Program and we wanted to share these changes. So we've written a nice, chunky blog about it.
Software developers might already understand KSP uses an incremental waterfall methodology - Incremental Development Model. This means, we go through planning, development and testing phases for each update, which also includes an internal Alpha and Beta to make sure our game is as playable as possible with each update. It's an important part of our process and helps us make sure our players can always enjoy KSP, whether it's digging into new content for existing players or a new player joining in for the first time.
Branch Testing allows us to take this model and apply it to each feature and treat each feature as though it is the update itself. There's two key elements to discuss. First is "Branch Testing," which is where we have the ability to test each feature individually before they're all combined into a full-featured build. The second is our extra QA phase, which is going to take place before we integrate all new features into the build. Branch Testing is the largest and most important improvement and changes a fundamental part of our process. In the past, we'd implement all features onto a central build. This was a good method of testing but isolating each feature allows us to know exact changes and how certain issues are related to the changes. Testing becomes much more controlled, which is always a valuable addition to the process.
The image below is a simplified overview of our prior testing process.
Development takes a large chunk of time before we proceed to integrate everything together, while testing this 'QA Branch.' Once the features are implemented and major bugs squished, we pass it through to Experimental Testing and proceed to test the 'Experimental Branch'. We went into this in further detail in a previous blog: All About QA and Experimental Testing!.
Branch Testing, however, is more compact and optimized. You can see an illustration of these improvements below.
Adding the extra QA phase before the features are fully integrated might not seem like much but it's going to save a lot of time for the team.
Let's look at a specific application of branch testing. We can use the Procedural Craters HarvesteR developed for 0.21. In the past, the timeline for that feature's development and testing would have looked very similar to the first illustration shown. Now, Branch Testing changes that a fair bit, we have the following lifecycle up until the QA Phase, starting from the Development and Implementation:
- Main Development and Implementation of Craters.
- Preliminary Testing by Developer (very minor).
- Craters Branch Build is created.
- Craters QA Testing and Feedback.
- Bugfixing and Balancing.
- Repeat Steps 4-6 for the scheduled Branch Testing Period (usually 2-3 days).
Now, as soon as a new feature is ready for QA, we begin testing it. We grow familiar with the feature and have most major issues dealt with by the time the QA phase arrives and the new feature is integrated into the central QA branch. Additionally, we can test one new feature while our developers work on something else. We expect this to help streamline our process, particularly in testing. Now we can test as soon as a feature is ready and will have more knowledge and more control over what is changing between each build and branch.
This is still in the testing phases itself, so we're planning to continue with the normal QA and experimental phases. We don't want to rush this change in our process and hope to avoid any negative consequences so we remain as sure-footed as possible during all stages of development and testing.
Once we complete this change, it should be a major improvement to the testing process and its efficiency and organization, as well as development overall. Overall, I hope that this was an interesting insight into what we've been doing, specifically, to make everything run smoothly over here. I'll hopefully have more to write about in the future, especially more insights like this!
I leave you with the following video from 0.21 Update QA Testing:
Feel free to ask any questions that you have in the comments.