Game development is notoriously stressful, and crunch has become one of the hallmarks of the industry. We often hear horror stories about developers being forced to work long hours, resulting in burnout and high turnover for many companies. In his talk for this week’s virtual GDC, Butterscotch Shenanigans CEO Seth Coster said that it doesn’t need to be this way. He outlined the steps his team have taken to eliminate crunch entirely and make the development process more efficient.
“Our industry is notorious for chewing people up and spitting them out as burnt-out husks,” Coster said in the presentation. “So much so that those of us that have been around for a while are called ‘veterans,’ as if we’ve been at war. Do you know what they call experienced people in other industries? Experts.”
Coster said that seasoned devs are experts, not just in programming and design, but also in “working 80 hours a week…dropping a huge day-one patch to fix the thousands of bugs missed during testing, [and] losing sleep for a week.”
But Coster doesn’t want to be an expert in those things. He just wants to be an expert in making games. That’s why a few years ago he and the team at Butterscotch stepped back to reexamine from a macro perspective the way things are done. The result has been a streamlined dev process that does not physically and mentally tax the workers.
It was the concept of DevOps that did the trick. It’s a set of best practices aimed squarely at making software development flow more efficiently. Throughout the development of Crashlands, which was released in 2016, the team at Butterscotch Shenanigans was forced to engage in crunch in order to address the many game-breaking bugs that were born of a messy development cycle. For Levelhead, the team’s follow-up game, Butterscotch employed DevOps.
DevOps revolves around the automation of many of the processes involved in making a game. As a whole, DevOps can be broken down into three main pillars, referred to as “the three ways” — managing the flow of work, amplifying the feedback loop, and continuous learning. By utilizing DevOps, Coster said that he was able to get Butterscotch running like a clockwork machine.
“Each ‘way’ is essentially a different lens for you to use to get a better perspective on your work,” Coster said.
The first “way,” managing the workflow, is all about eliminating what Coster called “waste.” Waste refers to anything that the team is doing that is not delivering value to the customer. Things like team members constantly switching tasks, sending defective projects to someone further down the production stream, or engaging in unnecessary processes are forms of waste.
To help cut down on waste, Coster said that his team built tools that helped automate specific parts of the workflow that had traditionally been done by hand. As an example, Coster explained GamePipe, a tool developed by a software engineer at Butterscotch that made the deployment of the most recent Levelhead builds to the QA testers an automatic process. In the past, this was a manual process that Coster himself oversaw, one that took a week to complete. With GamePipe, it became doable in just a day. Not only that, but the automatic nature of GamePipe all but eliminated human error from this process.
The key, Coster said, is to identify the bottlenecks in the development process and search for ways to work around them.
“A bottleneck is a part of a process that’s receiving more work than it can handle. Understanding where your production bottlenecks are is super important,” he stressed.
In this case, his inability to quickly compile a test build of Levelhead was holding up the entire project, so they built a tool to address it. The result was a massive increase in productivity without demanding additional work hours from the team. In addition to compiling new Levelhead builds, Butterscotch engineers also automated the process of importing art assets and writing patch notes, among other things.
Visualizing the workflow is also a prominent step in the DevOps process. Butterscotch integrated Trello’s task management system into the development of Levelhead to great effect, and it wasn’t just the software side of things that received a boon. Coster said they applied the DevOps method to the business administration aspect of the studio.
“We increased our capacity in places where it mattered, we hired for roles that were specifically meant to relieve pressure on our production bottlenecks, and it worked,” Coster explained.
The second “way” of the DevOps process is to amplify the feedback loop. What that means, Coster explained, is that feedback from the customer flows backwards directly opposite to the workflow visualization.
“The second ‘way’ says that, while the work is flowing in one direction, we want feedback to flow in the other direction,” Coster said. “Feedback allows us to catch problems, improve processes, and build quality into the production pipeline. To get that feedback, we need to develop tools and systems to allow the people downstream from us to report problems.”
To that end, Butterscotch sought a way to automate the reporting procedure. The method already involved creating Trello task cards in order to visualize the ongoing tasks, but doing so was still a manual step. So, once again a new tool was built, this one tasked with automatically creating a Trello card with the feedback received from the QA team.
“Although this only took a few minutes per card, this was a manual process, which meant that it created tons of task switching, motion, and defects,” Coster said. “Because of all that waste, a few minutes per card can easily turn into five or even 10 minutes per card as the volume of cards increases because fatigue sets in and I would become increasingly distractible trying to process 60 cards in one go.”
To fix this, Butterscotch employed a Trello API (application programming interface), so that anyone reviewing an issue can just click a button to auto-convert it into a Trello card.
The third “way” of DevOps revolves around continuous learning and ongoing iteration. Coster said that Butterscotch encourages a culture of “psychological safety” where everyone feels free to experiment with new efficiency methods.
“Your studio is always changing, and the world is always changing around it,” he said. “If you want to truly reach your potential and get the most out of your work, you need to be able to adapt, improve, and grow. And you need to build that philosophy into everything that you do.”
Coster said that adopting the DevOps practices has allowed Butterscotch Shenanigans to reboot itself into a studio built around efficiency. Today, it’s deploying 2 to 3 content patches a month on Levelhead without engaging in crunch or other malpractices. Stress and high turnover in game development is the result of bad practices, according to Coster. The goal, he said, isn’t perfection, but improvement. If only the rest of the industry would adhere to DevOps.
Sam, the Editor-in-Chief of GameDaily.biz, is a former freelance game reporter. He's been seen at IGN, PCGamesN, PCGamer, Unwinnable, and many more. When not writing about games, he is most likely taking care of his two dogs or pretending to know a lot about artisan coffee. Get in touch with Sam by emailing him at sdesatoff@rektglobal.com or follow him on Twitter.
GameDaily.biz © 2024 | All Rights Reserved.
