What we learned from the Battleborn Closed Technical Test
What Did We Learn from Battleborn’s Closed Technical Tests?
We asked you to help stress-test Battleborn servers and you guys totally nailed it! Over the course of a week, you hammered our test servers – and each other. We learned a ton and want to share that with everyone. After all, you made this possible.
We locked Gearbox Software’s Chris Brock (Producer), Randy Varnell (Creative Director) and Jimmy Sieben (Lead Programmer) in a room and forced them to answer some of your questions:
A big part of the Closed Technical Test is to stress the servers. How’d that go? What did you find out?
Randy Varnell: As a “Technical Test”, our first goal was to be sure that the servers and technology worked. We had a couple of bumps, and found some great problems that we couldn’t normally find at the user count of our daily internal tests. So, super-useful test in that regard.
Chris Brock: Exactly what Randy said. Thanks to the tests, we found matchmaking issues, which we’ve been able to address. We never would have found those without testing at such a large scale. It was awesome to see that most of our internal services held up great, though. And those that didn’t, we were able to quickly fix.
Jimmy Sieben: It was great being able to simulate some launch-day levels of traffic. In general, our systems performed quite well, but a few surprising issues came up and were addressed in real-time during the event, while others will receive attention now that the Closed Technical Test is over.
Part-way during the CTT, you adjusted Phoebe’s abilities. What did you learn about the available heroes and how they’re balanced? Will you consider adjusting characters’ abilities similarly when the Battleborn ships?
Chris Brock: We found out that there was at least a perception (and there’s data to back this up) that Phoebe was too powerful. This mostly taught us that we need to make sure we don’t isolate characters from their ‘counter-characters.’ We also learned that we can adjust these things pretty quickly without having to put out a full patch in a lot of situations.
Randy Varnell: We’ve long had running solid theories of how our balance worked, but we simply lacked the volume of play to prove them out substantially. For example, we designed the game to create excellent play synergies between characters. Looking at the CTT data, we’ve been able to statistically demonstrate that a team’s ability to win is greater than the sum of the individual characters chance to create wins. It’s partly a co-op competitive game, and it’s awesome to see the data prove out that our designs are on track!
We also know that you changed the frequency of gear drops during the test. What did you learn about the gear system and how players were using it?
Randy Varnell: The QuickPack drop was an experiment to test the boundaries of our technology, as well as how much we could use the pack-dropping strategy as a delivery for our gear. The technology behaved quite well under stress, but the testers let us know very clearly that it wasn’t fun opening up dozens of QuickPacks. That’s an easy thing to change, so we alleviated some of that during the CTT, and will be altering the loot drop design for the shipping game to balance getting regular rewards with being overwhelmed by tedious pack opening.
Chris Brock: Just adding to what Randy’s saying, this was strictly a part of the test. There’s a particular backend service that’s hit every time a loot pack is opened and we wanted to hammer that hard. Mission accomplished!
The CTT threw players straight into a game mission or competitive multiplayer. How did people generally react to that? Will there be a tutorial or introduction on how to play the game?
Randy Varnell: This was interesting. Our goal with Battleborn is to do something differently with familiar parts – that means while the game moves and plays as a shooter, leveling up and learning the intricacies of a layered sports-like game mode can be a bit daunting. Players showed up on our Closed Technical Test forums asking for tutorials. What was awesome was watching other players coach and help new players learn the game mode. That’s a good start for a community!
Chris Brock: One thing we realized is that we didn’t have enough training in place for Meltdown mode for this test. It makes sense, because it’s a mode that’s pretty unique to Battleborn. Players tended to pick it up after a few matches, but they need to understand quicker, so we’ll be working on addressing that in the final game.
Randy Varnell: Well, we’ve always planned introductory missions in the full game that teach game concepts. Now, we’ll be looking at other ways to help train other aspects of play.
So, Randy, who were you playing as during the CTT?
Randy Varnell: I played all 15 at least one match. It’s been a while since I’ve been able to play against a huge pool of talented players with different approaches to play, and it was really great to get the chance to see how characters fared against the unknown. Later in the second week, I played regularly with a full pre-made team, and settled into two primary roles. Caldarius is one of my loves, but we had a great Caldarius player on our team. So, I played Montana.
Jimmy Sieben: I am sad that the Closed Technical Test is over. I really enjoyed coming home from a long day of work keeping the event running, and playing some of the game myself! I really enjoyed playing Orendi, Oscar Mike, and Caldarius. I still have a lot of exploration to do with finding the perfect Helix choices for those characters, and of course finding the right Gear to complement my play style. I can’t wait to play with the larger community again in the future!
OK, so you’re out there, playing alongside (and against) everyone online. Did you learn anything about the game from playing directly with the public?
Randy Varnell: Yes, related to my Montana games! There were some threads on the Closed Technical Test forum that suggested Montana was way underpowered, and I wanted to experience that. I won 7 out of 8 of the Montana games. Defenders are interesting – they make the team better, but if your team isn’t cooperating, they can be very challenging to play. I had a good team, and typically went 7-0 as Montana within a match. Data-wise, we were seeing Montana picked less frequently than other characters, but that he was present in many of the most-winning team combinations. My experience seemed to match that very well. Montana can make a team super-powered, especially if you have a support character that can help you through some of those early rough levels.
By the way, killing an opponent with Montana’s ultimate ability (Mansformation) by stomping them to death…very satisfying! I did this a lot. I laughed. It was great fun.”
Now that the Closed Technical Test is done, we’ve got eyes on the Open Beta coming up. What kind of changes can we expect for that – and the final game?
Randy Varnell: Many! With the characters, we will continue to analyze data and make balance changes to characters. We actually recorded all of the Helix choices too, and will be taking a deep look at the augmentations which were less chosen, and see if we can make them more interesting choices.
Gear will be getting some additional tweaks, especially in regards to the activation costs and the way in which they stack. That should help gear to be more useful and more balanceable, as well.
Finally, the Meltdown map, Paradise, is getting some tweaks to make the mode a little less “snowball-y.” We watched and agreed with the community that it was too easy for a good team to take advantage of early success, and wanted to give more of a chance for later game comeback. We’ll have more details on this!”
Speaking of things you learned from the tests, were there any happy accident discoveries that came out of all this?
Chris Brock: It wasn’t really a surprise, but I was personally really happy to see how fast a meta developed and how the forums were lighting up with discussions on how to counter various strategies.
Also, we found out early on that the Story Mode, particularly Single Player, was very popular in an event that we thought would be more PVP-focused. People, apparently, really want to dig into a single player experience as well.
Randy Varnell: From a design standpoint, we’re learning how to better evaluate, create, and test a balance patch. We did this (for Phoebe) live, during the week of the CTT and are now working on ways to move even quicker. This should give us a tremendous amount of agility at launch.
How is feedback and data from the CTT helping Battleborn development at this point?
Chris Brock: Feedback is being used to confirm suspicions we have and to prioritize the order in which we tackle issues. The loudest problems tend to get the most attention.
Randy Varnell: What it comes down to is that not only has playing with everyone been great, we loved hearing what you had to say – all the feedback has been amazingly helpful, for every department! I spent a lot of time listening to players on the forums, and we’ve had some real super-heroes who have written some very well-thought-out responses to their time playing. We take all of those suggestions to heart, and will do what we can to incorporate all that feedback. As mentioned above, all of our development departments are looking at the feedback and the data, and are polishing the game with user experience in mind.
And for all that, we can’t thank you enough!
What can I say? You guys are badass.