Insider: Mike Acton

We submitted a few proposals for next year’s GDC here at Insomniac. I’d like to share those with you. We’d love to get any feedback you might have so that we could make them as interesting and useful as possible! (If they are accepted by the GDC committee.)

Culture Shock: User-Research on Resistance 3

Speakers: Bill Fulton (Founder, Ronin User Experience); Drew Murray (Creative Director, Insomniac Games)
Track: Game Design
Format: 60-Minute Lecture

After participating in formal usability tests near the end of Resistance 2 production, Insomniac Games began a fundamental shift in how we make games. While Insomniac has always been focused on making great games – and has a long history of doing so – we’ve relied largely on our gut-feelings of what’s going to be fun, utilizing limited playtesting near the end of production to find and fix major problems. Formal usability testing on Resistance 2 showed us that there was a surprising and often painful gap between how we, the developers, played and understood the game and how typical gamers did. Based on the difficulties we saw playing having in our Resistance 2 usability tests, Insomniac as a company decided to move beyond paying lip-service to user-research and fully committed to systematically integrating it throughout the Resistance 3 development cycle, not just near the end.

What Testing Methods Worked? We used many types of user-research over the course of Resistance 3 production – external RITE usability tests, weekly company-wide playtests, informal one-on-one internal playtests, large-scale external playtests, and other methods. We’ll discuss the pros and cons of the different methods we used, the different information we were able to get from different kinds of testing, and the realities of financial and time costs of each.

Overcoming Culture Shock. There was more than a small amount of opposition on the team to the amount of user-research that we did on Resistance 3. User-research can put a tremendous burden on all members of the team, whether it’s running the tests, gathering and analyzing data, quickly iterating to respond to feedback, or just carving out an hour or two a week to actually play the game! We’ll discuss our experiences on Resistance 3, ways to get your team excited about doing user-research on your game, and how to handle common team complaints.

Did It Work? As we ship Resistance 3 (on the day this talk is being submitted!), we’ll examine whether we accomplished our goal with use-research for the game and some of the specific – and sometimes very significant – changes that we made to the game based on the findings. We’ll also talk about some of the things that didn’t work as well as we’d hoped with our testing methods for Resistance 3, and what we plan to do differently in the future.

Attendees of this session will get an insider’s view to the sometimes-bumpy road that Insomniac Games has traveled over the past three-and-a-half years as we’ve revamped our game-development process to focus on frequent and consistent user-research. Different user-research methods will be illustrated by specific examples from Resistance 3, such as videos of the RITE testing we did for core controls and aim-assist, and graphs created to track how the rating-scores assigned by playtesters for different levels and features trended over production. Attendees will also learn about common pitfalls and how to overcome them, and how to convince a team that user-research is an invaluable tool for improving your game.

Using Middleware With a Custom Job Manager

Speaker: Chris Kosanovich (Engine Programmer, Insomniac Games)
Track: Programming
Format: 60-Minute Lecture

A year and a half ago we started work on a brand new engine at Insomniac Games. We decided to start from scratch on both tech and tools. Because we have aggressive timelines for games using this engine we made choices to use middleware where we had previously developed custom solutions. Among the middleware vendors we are using is Havok, however one of the first issues we found with integrating their systems into our engine is that on the ps3 we don’t use spurs and instead have rolled our own job manager in house that is more tailored to what we need. This has caused all sorts of issues from simple things like having to port the havok spu elfs to our custom spu modules, to dealing with cloth’s use of overlay tables and finding a solution for that.

My talk will focus on the challenges (like those listed) that we faced and how we dealt with them. I will detail what benefits we are seeing and what we expect to see on future titles.

This will be valuable to others thinking of using middleware in conjunction with a custom core system, or something like Sony’s WWS job manager.

Developing Imperfect Software, how to prepare for development pipeline failure

Speakers: Ronald Pieket (Senior Engine Programmer, Insomniac Games)
Track: Programming
Format: 60-Minute Lecture

Computer game software development, more than any other kind of software development, relies on the usage of software that is itself under development.
In this talk I will assume a common scenario: the game content is developed in parallel with the game technology itself. Game technology includes the game executable as well as a custom content production pipeline.
Development team members rely on up-to-date versions of the game technology. And because this technology is itself in development, it will fail regularly.
In a medium to large development team, day to day breakage of the game technology can lead to productivity loss. The larger the team is, the greater the potential loss.

In this talk I will assume that development teams have already taken steps to minimize breakage. I will assert that it is not practical to prevent all breakage. The talk is about measures that you can take in your code and pipeline, to reduce the impact of breakage, and minimize productivity loss.

I will briefly discuss my experience with previous attempts to address this issue, including unit testing, code vetting, peer review, and controlled version distribution. They are valuable but have drawbacks.

I will propose a three way approach: resilience, reversion, and reporting.

(1) Resilience: avoid “perfect or nothing” implementation. Don’t abort runtime execution for errors that can be fixed up. (Use automated reports to avoid issues going stale – more under “reporting”) Verify asset data upon loading, and be tolerant of data format differences. (I will explain options) In tools, prevent loss of work by adopting a client-server architecture. (I will explain what we have done at Insomniac)

(2) Reversion: offer a simple mechanism to revert to earlier versions of everything that can break, including executables and asset data. Revision control systems and package installers have shortcomings. (I will explain what they are) Code/data version dependencies make reversion harder. (I will explain how to avoid them)

(3) Reporting: an assertion dialog and stack trace on the screen of an artist has a long way to travel. It interrupts their work flow, and if it is “skippable”, it is likely to get ignored. I will propose an automated remote error reporting system and database. (I have a lot of detail available)

In order to develop the next hit game, we must ensure that during development, the game and other development software not only works well, but also fails well.

Designs: Ideas with Intent. Realizing What You Want to Solve and then Applying the Best Solution.

Speakers: Mike Ellis (Design Director, Insomniac Games)
Track: Design
Format: 60-Minute Lecture

In the world of game design the coolest idea is king. However, as an industry, game development has very few processes for creating and developing ideas that can be used to spawn the next smash hit. All too often lots of ‘loose’ ideas are smashed together and fail to make a complimentary and well rounded design.

This session aims to help change that by demonstrating that great designs aren’t simply ideas, but ‘ideas with intent’. That is, design solutions that not only are cool, but are crafted to solve very particular problems that have been identified and communicated.

The session will show the audience the four steps to designing with intent (Defining, Creating, Evaluating and Choosing) and demonstrate several techniques (Intent Statements, Goal Summaries, Defining Conditions and Parameters, Conversation Framing and Keeping Each Other Honest) that can be used by designers in order to ensure that they are communicating their intent. It will also discuss why communicating the intent of your ideas is an important and effective weapon for leveraging the power of the team and generating increased buy-in.

Attendees will walk away with an appreciation of why designs are more important than ideas and be in possession of several techniques that they can instantly apply to their work upon returning to the studio.

Math and Physics for Game Programmers

Speakers: Erin Catto (Principal Software Engineer, Blizzard Entertainment); Erwin Coumans (Principal Physics Engineer, Advanced Micro Devices); Gino van den Bergen (Owner, DTECTA); Glenn Fiedler (Senior Staff Online Game Programmer, Sony Computer Entertainment); Graham Rhodes (Senior Software Engineer, Applied Research Assoc., Inc.); Jim Van Verth (Engine Programmer, Insomniac Games); John O’Brien (Senior Gameplay Programmer, Insomniac Games); Mike Acton (Engine Director, Insomniac Games); Richard Tonge (Senior software engineer, NVIDIA Corporation); Squirrel Eiserloh (Director, TrueThought); Takahiro Harada (Researcher, AMD);
Track: Programming
Format: Two-Day Tutorial

As the complexity of games as increased, so has the knowledge needed to create them. Creating the latest code for graphics, animation, physical simulation, even some extent artificial intelligence, requires greater knowledge of the necessary mathematical underpinnings than ever before. And of the fields described above, one of the most rapidly growing is physical simulation, as shown by discussion boards, the latest games, and a recent burst of articles and papers. Creating such a simulation may appear to be a daunting task, but it is possible with the right background.

This two-day tutorial continues and expands the tradition of both the “Math for Programmers” and “Physics for Programmers” tutorials by bringing together some of the best presenters in both gaming math and physics. Over the course of two days they will get programmers up to speed in general 3D math, and then deepen their knowledge by focusing in on the topic of physical simulation.

Day One of the tutorial (Math for Programmers) will concentrate on the core mathematics necessary for sophisticated 3D graphics and interactive physical simulations. The day will focus on the issues of 3D game development important to programmers and includes programming guidance throughout. Topics include affine transformations, quaternion algebra, computational geometry, curves and interpolation, math for game AI, and optimized data-oriented design for mathematical operations.

Day Two of the tutorial (Physics for Programmers) will provide a toolbox of techniques for programmers interested in creating physics engines, with references and links for those looking for more information. The focus of the course is to study various pieces of the simulation pipeline and show how problems along the way can be solved and optimized using standard 3D mathematical concepts and engineering know-how. Topics include integration of a physics engine into a game, rigid body solvers, collision detection and contact resolution, physics on mixed CPU-GPU systems, rigid body destruction, and networking for physics programmers. Sample code libraries and examples are provided.

Game Tools as a WebApp: Lessons from Insomniac’s Next-Gen Tools and Engine

Speaker: Mike Acton (Engine Director, Insomniac Games)
Track: Programming
Format: 60-Minute Lecture

Insomniac, like many developers, has traditionally treated development tools as a second-class development effort. While we have had dedicated Tools programmers for many years, very little effort was put into the overall user experience and runtime performance was always valued over development effort. This resulted in a systematic increase in development cost and an entire “Engine Economy” based on brute-force effort. Two years ago, we decided to tackle this problem head-on. The result is an entirely new suite of tools, written from the ground up to fundamentally change the way we work as a studio.

It was also clear that the world was changing. The explosion of web-based enterprise applications, fully-web enabled mobile devices (like the iPhone and iPad) and browser-based gaming sent a clear signal that the era of native applications living in a sandbox was over. Not just for games, but also for the tools we use to develop games. We felt this was an opportunity to make a strategic change that would position us much better and net us valuable experience for the inevitable changes to operating systems, development tools and user expectations.

This is the story of our experience so far.The first game Insomniac will ship on these tools is Overstrike. We will share our most significant choices and the costs of those choices. We will suggest alternatives where we feel that better results can still be achieved. And we will share details of the technical architecture of the new tools suite.

Details such as:

Usability is not random. We needed first-class information to develop first-class tools. We set up a formal usability practice without a lab or additional budget using a single PC. We will also share specific, novel tools features such as our “neighbor” and “tile” mode which allowed us to increase production time on common tasks by as much as 20x.

The choice to go webapp. The initial version of the tools was not web-based. It was a native application which used Flash for the front-end. This model was not significantly improved over the traditional native model and introduced the same data complexities. Webapp tools allowed us to not only take advantage of many available development tools, it forced a clean separation of UI and back-end data.

Client/Local-server model. One of the concerns of browser-based tools is adding latency. We largely solved this issue by running a mongoose-based web server local to each machine.

Additional details. We will also share our experience with allowing custom tools and UI development by non-tools programmers. Our take on the Flash vs. HTML5 debate. Details of integrating a native engine. And browser and security workarounds.

We’ll share enough about how we made our tools and why so that you could begin to duplicate the process. We’ll share why we made the choices we made so that you can justify the effort and search for reasonable alternatives. And we’ll share where we went wrong so that you can avoid the same costly mistakes.

Developing Animation Tools Using Web-Based Technologies

Speaker: Giacomino Veltri (Engine Programmer, Insomniac Games)
Track: Programming
Format: 60-Minute Lecture

The goal of any game toolset is to provide content creators with the ability to access and tweak a wide variety of in-game parameters. Providing high-level access to base features often leads to an almost unmaintainable mish-mash of low-level systems code and high-level user interface code. Fortunately, the separation of user interfaces from implementations is a problem that has already been largely addressed by the Web.

Browsers are becoming more and more ubiquitous and are showing up on more and more hardware devices. Thus it seems only natural to make browser-based game tools. This presentation will break down how the task of creating an animation set for an in-game character can be done using a browser-based solution.

The first step is to place a lightweight web server on top of our engine code. Functions and variables can be exposed to the user interface via URLs; to call a function or access a variable, all one needs to do is create the appropriate URL and type it into a web browser. Communication between the web server and the user interface is done via JavaScript Object Notation (JSON). All our asset data is stored in JSON and sent to the user interface in that manner. From there, any user interface can be created using the rich set of web-based libraries available.

The result is that by accessing a web page, content creators are able to add, remove, and preview animations; they can blend animations using data from the game; and they can place and trigger in-game events on a timeline. All of this is done through a combination of Flash and ActionScript, JavaScript, and the HTML5 canvas.

However, the devil is in the details, and this presentation will discuss some of these details that need to be taken into consideration when deciding to make browser-based game tools. Some of the advantages include:

• Separates high-level user interface code from low-level code
• Portable to a wide variety of platforms
• Lots of custom controls already created and available
• Can easily and quickly prototype user interfaces (no compile step; just reload the web page)
• It is easy to create custom controls with custom graphics
• JavaScript has built-in support for dynamic objects, which makes complex UI logic easier
• Stability; crash-inducing bugs in the user interface do not necessarily cause data loss

Other things to consider are:

• Learning curve – often requires a use of multiple languages/tools (HTML/CSS/JavaScript/ActionScript)
• When to chose the right tool (HTML5 canvas vs. Flash, and so on)
• UI design challenges
• Debugging (and there are some silent failures)
• When should one use ActionScript/JavaScript/some other web technology?
• What interface should one use to communicate with the low-level code?
• What will content creators think of this style of tool?

In conclusion, we hope that by creating browser-based tools, we provide our users with a richer and flexible user interface to create the games they want to make.

It Stinks and I Don’t Like It: Making a Better Engine Experience At Insomniac Games

Speaker: Sean Ahern (Information Architect, Insomniac Games)
Track: Production
Format: 60-Minute Lecture

Video game development has progressed significantly over the years – from punch cards and user made tools to enterprise level applications. While the technology that we use on a daily basis has changed significantly, can we really say that the experience of creating games has improved as well?

With each jump in console technology, we push the limits of both a studio’s budget, along with the amazing content that we can create utilizing the tools at our disposal. Studios are constantly on the look-out for the next best tech, middleware, or ideas to help make their next title a reality (and success). With licensed tech and often-times even proprietary software, the User Experience is frequently viewed as a second class-citizen when efficiency and deadlines are in the fore-front.

At Insomniac Games, we’ve decided to make the User Experience of our tools a priority. By leveraging traditional engine programming with web application technology, Insomniac Games has created their new engine from the ground up with a focus on the User Experience.

In this lecture, Sean Ahern, Information Architect for Insomniac Games, addresses how User Experience has helped lead the development of their new engine and its’ features.

Always Online: Network Systems in Insomniac Games’ Overstrike

Speakers: Peter Kao (Senior Gameplay Programmer, Insomniac Games)
Track: Programming
Format: 60-Minute Lecture

Overstrike is an upcoming four-player, cross-platform action title from Insomniac Games, supporting online play. Getting Overstrike online required a fundamental shift in the studio’s development approach, making networking an integral part of the development process. This session will establish our ‘always online’ programming philosophy and provide a comprehensive overview of the network technologies used to support that goal.

On Overstrike, we are all network programmers. Each team member is responsible for implementing and debugging the network sync implementation of his or her feature. Adopting this development approach has reduced encounters with “it works offline” bugs, the online gaming equivalent of the classic “it works on my machine”.

High-level network APIs are always available in Overstrike—even when running without a network connection—and game state changes use the same interface in all modes. State changes are deferred by default; this programming model presents the programmer with a consistent interface and looser timing guarantees, reducing the inevitable differences between online and offline behavior.

We will present Insomniac’s high-level synchronization system, called SyncHost. This system provides ‘synced field’ feature which may be familiar to attendees. SyncHost has a number of other novel features which will be of interest, including a synced object allocation system, and the ability to send updates on an individual field. SyncHost is very loosely coupled to the network, and we will explain how this loose coupling enables the common interface between online and offline modes.

We will also present Insomniac’s synced components, which leverage synced fields to provide a network-centric complement to the popular component-based update systems appearing lately. As an example, we will show how synced components are used to implement state machines that sync across the network. Our approach uses synced fields to share transition data among clients, but allows each client to apply that data in a way that results in the best player experience.

The discussion of synced state machines will introduce our last syncing primitive, called the MessageInstance, which is a flexible message passing system used to provide non-deterministic updates to long running states. This interface transmits data defined by key-value pairs, and supports compression and querying if a value is present. The query technique leads to code-sharing opportunities that are not available with more traditional network models of sending an entire pre-defined packet of data, usually expressed as a C-style structure.

Overall, this presentation will give attendees a clear and complete picture of how robust networking technologies form a fundamental part of the development of Overstrike.

[Updated: 13 Dec 2011 – Additional exercise examples from 2011.]

Each week in our Core team meeting, I give the team an exercise. I roughly target them to take between 5 and 15 minutes. I’ve found the process to be beneficial on the whole and a good bit of fun. Basically, at the end of our meeitng, I explain the exercise, then people group up (usually around 2-4 people), write down their answers and give them to me on their way out. I write up all the answers afterward and send them all out to the whole team. It’s a pretty simple process.

I think doing these kinds of exercises helps:

  • To get different people to work together, if even just for a few minutes. People have a tendency to stick with the people they work with in their project groups. This gives everyone a small chance to interact with someone different and share thoughts and ideas.
  • To exercise the brain and think about something different. A bit of something random helps break people out of whatever they’re stuck in and “cleanse the palette” a bit.
  • And often, the exercises help to remind us that ultimately we are all Game Developers. We may focus on Engines or Tools or UI or Usability, but ultimately we make games. And it’s easy to narrow your focus over time and forget that.

Here are some of the exercises we’ve done recently:

  • Name three ways that our model of the world is out-of-date and how is that reflected in what you do?
  • Sketch a design of a 3D adventure-style game. Constraint: You have no collision intersection information.
  • Sketch a design of a game that unique identifies each person in your group. i.e. We should be able to infer who the game is about from the description of the game (without mentioning them explicitly.)
  • Sketch a design of a game based on making new friends on the internet. Constraints: Game happens in only one room and is playable in four combinations of: Same place, same time; Same place, different time; Different place, same time; Different place, different time.
  • Sketch a design of a multiplayer game. Number of players is however many people in your group. Constraints: Each player must have opposing, but overlapping goals. The only (direct) inter-player communication allowed is a color change. Outline a possible winning strategy for each player.
  • Sketch a design of a game specifically for the Core team to play. The catch? It has to be something that Mike Day and Al Hastings *specifically* would want play. [ed. Both members of the Core team.]
  • Sketch a design of a game based on dividing the game industry into factions. (e.g. Casual and AAA; Artists, Programmers and Designers) Use a metaphor that represents the industry (but cannot literally be game industry.)
  • Define three things that you would want to tax/challenge internally for these different departments: Design, Art, Gameplay and Animation. How could you reward that specifically?
  • Sketch a design of a game that applies location-based games/entertainment to virtual spaces. Both inter-game and intra-game. E.g. Foursquare between games. Or Color within a game.

Here are some additional examples from 2011:

  • Get into pairs. Talk about what you do. What would you do if that thing became commodity? (i.e. there was a cheap, ubiquitous solution available for it) How would you then raise the bar? Then what if *that* thing became commodity? Repeat until out of ideas.
  • Pick something that people don’t understand well or that you find difficult. Sketch a game that teaches the concept. Examples: Discrete Fourier Transform or Architectural Composition.
  • Game sketch. You have all the data from Wikipedia. But you cannot use any of the text from it in the game.
  • Consider issues which you have hit in the last week or two. What metrics, if available on some kind of dashboard, would have significantly reduced the time to resolve them.
  • Apply the lesson of Kindle highlighting to games and tools.
  • No meetings, no email. (1) What problems do they solve? (2) What would be (practical) alternatives?
  • Pre-mortem: (Imagine yourself a few months from now, what do you think could possibly happen…) Top Five – What went right? What went wrong?
  • “What I learned this year that makes me better at my job every day…”

You may also be interested in checking out these books:

  • Challenges for Game Designers by Brenda Brathwaite
  • Gamestorming: A Playbook for Innovators, Rulebreakers, and Changemakers by Dave Gray, Sunni Brown, and James Macanufo

Every exercise doesn’t work for everyone. Some people like the more open-ended ones. Some people prefer puzzles with well-defined constraints. I don’t expect to find the “perfect set” of exercises. I’ll just keep adapting, taking feedback and have fun with it. But I don’t think a week has gone by where I haven’t learned or been surprised by something that came out of this.

Mike.

Today I introduced the Limited Edition Al Hastings* Award in our Core team weekly meeting.

Of course Al was the first recipient:

It will be awarded for one of:
• Technical excellence
• Silently waiting… then NINJA ATTACK!
• Passing the Turing test (Note: The difference between Al and AI is so small in most fonts that it has lead to more than one hilarious, confusing conclusion about Al.)

History

I started working on this last year in December. It takes a while for something this good! ;)

To start with, I needed good images of Al for the sculptor to work with. I recruited Ryan Schneider for help! (Thanks, dude!)

Ryan convinced Al that we needed new promo shots for the website. He set up the fake photo shoot and got the shoots I needed:

[Actually, now that I’m posting these on the website as part of this update, I suppose it was technically true!]

At which point I could send them off and get the sculptor started. Interesting side note: Al is officially ‘Pepper and Salt’ and not ‘Salt and Pepper’ (I’ve never made the distinction, so news to me.)

About a month later the sculpt came in for review:

After approval, it went to paint. A test model was created so that I could review the coloration, etc. To make sure that it was sufficiently ‘Al-like’

Another round of approvals and it went to press.

Fast forward to last week, and we had our final models in-hand.

I see that Al’s Bobblehead doppelgangers (Bobblegangers?) have a lot of drinking ahead of them there…

Mike.

[*For those of you that don’t know Al Hastings. He’s one of the owners of Insomniac, along with Ted Price and Brian Hastings. He’s also a member of the Core team. And he’s the main man responsible for the awesome rendering tech that changed the world in Spyro the Dragon on the PSX among many other acheivements since.]

Every week here on the Core team we have a 5 minute show and tell. Where someone shares something interesting to them.

I’ve shared some of these on our R&D pages before as well:
* Carl Glave spoke about Minecraft
* Giac Veltri shared tips on being a better bowler.
* Scott Michalek talked about rotoscoping.
* Jonny Garrett shared how Insomniac had to hack our own game to make a patch.

And recently John Fiorito (COO of Insomniac, who can often be found sitting in on our Core weekly meetings) shared his memories of his very first game:

Previously I shared with you some of the Review Questions I used to interview the team.

Today I’ll just quickly share some of the questions I’m asking as part of one-on-one meetings coming into this near year. These questions are really just used to guide the discussion and build the conversation.

* What needs to happen for you personally to feel really productive every day/week?
* What are some of the things you’d like to be able to accomplish?
* What would you like to be able to say (about yourself, progress, etc.) at the end of this year?
* Is there anything you feel is wasting your time?
* Are there any trouble spots I should know about?
* Do you feel like you’re on-target?
* How do you think you could make your communication with the rest of the team even better?
* Do you think there’s anything that our team should be doing more of?
* Do you think there’s anything that our team should be doing less of?
* How do you think we could apply more lessons from games into the tasks you’re working on?
* How do you think we could test what you’re working on a larger scale?

As always, I’m interested in what questions *you’ve* find helpful. Either asking or answering.

Mike.

[20 January 2011] I thought I’d add a couple of great suggestions here:

* “If you could change any one thing right now, what would it be?” (It’s a great reflective question. Suggested by my uncle on Facebook, actually!) :)

* “How are you?” (Definitely don’t want to forget this one! @maximilianburke reminded me to add it on Twitter. He also suggested you might want to check out Rands in Repose on 1-on-1 meetings: http://bit.ly/gmj11M)

Actually, AAA Game Development *can* be a dream job. It’s certainly mine.

By some measure, I am one of “them” when we talk about management decisions affecting the lives of developers in our industry. As the Director of our Core (Tech and Tools) team, in no small measure whether people’s jobs are miserable or joyful is influenced by my choices as a boss.

It’s a responsibility I take very seriously.

Lately there has been a lot of renewed talk about the toxic environments that sometimes occur in professional game development. I can tell you from first-hand, personal experience at previous companies, that *absolutely* those things do happen. There are places that without a doubt, make lives miserable (both in and out of the office), stifle the creativity and threaten the integrity of some of the best minds in the business. But it doesn’t have to be that way. And it certainly isn’t always that way.

So today, I’d like to share with you some of the things I try to work on and think about every single day so that I can do my part to ensure that we here at Insomniac have earned (and continue to earn) our title of one of the “Top Small Companes to Work For”

* Work with good people: It all starts with having great people that can rely on each other. I believe that every single person on the team is a world-class talent, or on their way to becoming one. Everyone needs to know they can rely on each other and that no one is going to be left picking up the pieces of some slacker time after time.

* Passion for making things better: Making a great work environment doesn’t mean making a *perfect* environment. Perfection is a fool’s errand anyway. But what is important is that we’re always looking to figure out what is and what’s not working and improve things, every day. Making things better is hard work. It’s never finished. To do it right, it must be a habit.

* Fix mistakes: When you make mistakes, you have to fix them. Things change quickly. And to compete we need to change quickly. It’s too easy to stay stuck with our past choices and not re-evaluate them even in response to real evidence. There’s a lot of cultural momentum, no matter where you are to keep things the same. Sometimes things that were the right choices at the time, become the wrong choices as development changes. As an example, the usability of our tools and engine was not a major priority in the past. We relied on a lot of brute force to get things done. But as things have changed, our goals have changed. These days, we have a full-time UX designer on our team and everything is usability tested constantly. We’ve even been experimenting with how to usability test code for programmers. Because the code is really just a (text) UI for controlling the underlying data transformations – exactly like tools GUIs. It’s no small effort to make these kinds of changes, but as a team we understand the importance of making fundamental changes when we know they’re needed.

* Freedom to raise the bar: This is one of the main principles here at Insomniac. But this is where I need to give Ted and Al and Brian mad props. Because rather than just paying lip-service to the idea of raising the bar, they built a company that recognizes that to be better, you must fail. A lot. And it’s exactly that freedom that allows me to experiment with what will make our team better. And it’s the same freedom to push harder and try new things that I try to pass on to the individuals on the team.

* Share ideas: None of us got where we are without learning from others. You simply can’t be a great developer without a healthy exchange of ideas throughout your career. Both internal to our company and externally, sharing our ideas and experience to help make everyone better, makes us better. Along with our R&D pages, inside the studio I try to ensure that we have every opportunity to share what we’ve learned with each other. Whether it’s through presentations or informal one-on-one conversations, creating an environment where people can learn from each other is a pillar of any great team. It’s not just everyone has a voice, it’s that we *expect* everyone to have a voice and find to express it.

* We participate: As a company, we participate with our fans. Our community team is dedicated to being out there and listening to what people are saying and working with us to make sure we understand the relevance of those conversations. For example, Ryan Schneider, our Director of Brand Management, is known for giving our team presentations on the state of our community so that we can discuss how that might affect our choices. But I also think it’s important for us to be out there, not only talking to our fans, but also our peers. Whether it’s on Twitter or Facebook or blogs or whatever, there’s a lot of conversations and information out there that will make us not only better developers, but also keep us honest about our own work. The important thing is that we’re not *afraid* to participate or try to micro-manage every single thing every person in the company says. Some things are secret, obviously, but beyond those specific details we know there’s a lot to be gained from getting out there and just talking to others.

* Push people outside their comfort zone: In order to be better, you need to grow. In order to grow, you need to do something different. I try to work hard to make sure that everyone has the opportunity to try something they simply aren’t good at (yet.) Even when there is someone more expert available, they wouldn’t be an expert if they didn’t have practice. So you have to let others practice those same skills and not just rely on the same people to solve the same problems every time. Plus, having new people experienced in those skills will also give the experts some extra flexibility to expand in to other areas as well.

* One-on-ones: This is probably one of the most important parts of my job. I do my best to meet with twenty-something people for at least a half-hour, at least once every two weeks. For sure, that’s a lot of time. But no time is better spent than this. It’s an opportunity to listen. Every person is different. There is no “format” for these that works for everyone. Sometimes it’s about technical advice and feedback. Sometimes it’s about personal issues. Sometimes it’s about the day-to-day struggles of development. But whatever it is, it’s always about letting each person simply tell me what’s on their mind and doing my best to help them, as best I can, grow as professionals.

* Define the problem and the constraints, not the solution: This one is obvious, but it’s deviously hard to do in practice consistently. My job is to guide and direct. To give suggestions and feedback. To review and to set limits. But what I absolutely need to do is allow each person to find the actual solution on their own. I don’t want people that will just type in the solution I give them, I want people who are going to use their unique experience and expertise to create a solution *better* than the one I would have thought of on my own.

* The job is what you make it: Ultimately, making your job your dream job is up to you. Each person needs to push the boundaries of where they can contribute. The same is as true for me as it is for everyone on the team. I don’t expect people to be limited by the letter of the tasks they’ve been given or the job description. It’s knowing that it’s not really about those tasks or the job description, it’s about improving. Improving ourselves. Improving our products.

* Don’t wait for feedback: It’s fine to have an ‘open-door policy’ (which is actually difficult for me since I don’t actually have a door.) But more than that it’s necessary that I constantly seek feedback. I can’t just wait for someone to come to me with a problem or suggestion, I need to use every tool at my disposal to figure out what’s on people’s minds. That’s absolutely part of my responsbility. There isn’t a formula for this, either. There are tools I’ve found helpful for sure (like Rypple.) But the most important thing has been keep trying different things – some things work for some people and not others.

* Ask good questions: The kinds of answers you get depend a lot of the types of questions you ask. If you want the best out of people, you have to figure out how to ask the best questions. I don’t pretend to know any magic answer for figuring out what those are, but I can share some of the questions I have asked with you: Review questions.

* Let people share their progress: When people work hard to accomplish something, they should be recognized for it. Here people can send their progress to our mailing list. We also post videos and screenshots of how things are improving on our internal web page. And we have presentations (about once a month) where the team can show off the things they’ve done to the company. It’s not just about ‘credit’ – it’s about knowing that your hard work is appreciated by your peers and having the opportunity to get lots of feedback to make things even better.

* Everyone knows what is flexible and what isn’t: From personal experience I can tell you one of the most soul-sucking things that a developer can do to it’s people is to constantly change ‘the facts.’ In particular, release dates. I know a lot of places that will give their developers a ship-date only to extend it by three months when it’s closing in, then do it again and again. Which does nothing except piss people off and totally undermine people’s faith in the company. Here at Insomniac, we know when we’re going to ship well (years!) in advance and that’s when we ship. Period. And let me tell you, being able to trust that is abosolutely one of the things that makes Insomniac a great place to work.

* Know the importance of family: Family comes first. That’s it. We have a relatively mature team, many of which have family and kids. I know that in order to be a great developer, you need the support of your family. And in order to have the support of your family, you really need to be there for them. If you want an adult, professional and creative team you need to treat them that way and trust them. I know that people know what they need to do, and we can virtually always figure out how to balance that with the obligations of a family (and personal) life. [Note 15 January 2011: Since a couple people have mentioned it: Yes, I originally posted this on 25 December. But I don’t celebrate Christmas, so it’s not really ironic. Now if I ever spend all day writing on New Year’s Eve or my daughter’s birthday, feel free to hit me upside the head.]

* Give honest feedback: A lot of people have a hard time recognizing their own talents. It’s important to share with them where you think they can grow and learn, and where they really shine. A lot of bosses out there have this preoccupation with ‘being objective’ when it comes to feedback. That’s both impossible and ridiculous. Everything I say is going to be colored by my own views. I recognize that. My team knows that. But it’s also important to get a wider view of each person’s progress and I’ve found the best way to do that is simply to keep asking those who work with them most closely. For example, recently, I conducted 10 hours or so of one-on-one interviews with the team asking about their personal experiences with others. It was both enlightening to me and perhaps some comfort to others knowing that I try to always find better ways to assess the team and provide fair and honest feedback.

* Clear leadership and direction: I imagine my view on this one might be somewhat controversial. I do believe that the best ideas come from listening to everyone. I do *not* believe the best decisions are made that way. I know a lot of folks believe in ‘the wisdom of the crowds’ or ‘democratic development’ but what I’ve found is that those methods quickly degenerate into ‘design-by-committee’ and the outliers, often the ones with the best ideas, are pushed out. Not everyone’s ideas are going to make it. But I think it’s important that a team knows that the ones that do are going to be consistent with the direction and will be protected (even against a strong initial resistance by the majority view.)

* It’s not about the hours: It’s about the results. I’m not going to clock every minute of each person on the team. It’s well-established that it’s counter-productive, anyway. What we try to do is make sure everyone brings their A-game, every day. It’s about having a steady, consistent pressure that’s sustainable. And picking the right (demonstrable) goals *all the time* to ensure that we’re progressing and not wasting a bunch of time throughout the process. It’s about making the right choices about what where to place risk and where to make conservative choices. It’s about failing as fast as possible. It’s about getting as much feedback as you can as early as you can so that you’re not spending months developing something that simply won’t work or won’t get used. And it’s about working with each person on the team to figure out how they can do that best.

* Education: You can’t simply hope that people will learn new things. And just encouraging learning and research it isn’t enough either. Making continued education of our team a priority continues to be a major focus for us. Of course, we teach each other. But we also bring in guest speakers. And formal (in-house) training on new and different topics. But the biggest challenge, and in my view the most important one, is to get people to question what they think they already know.

* Think about the future: It’s also important that see (much!) further than the immediate problems at hand. While I absolutely believe in an iterative development approach, I also believe in good planning. Those things are not opposed, contrary to what you may ocassionally hear or read. You need to know what you’re iterating toward and have a good direction and principles to measure progress against. It’s important to me that everyone on the team can see what our goals are one, three or even five years in advance. So that we can focus on the right things.

* Know how to say no: I think one of the best things any director, boss, manager, etc. can do (programming or otherwise) is to say ‘no’ appropriately. If you say ‘yes’ to everything, you end up with a gigantic pile of crap that doesn’t serve any job well, without any focus or real strengths. I have to say ‘no’ to those requests that do not fit our goals, or our guiding principles. And it’s hard. There are definitely times when that upsets people. But saying ‘no’ is exactly what gives us the resources to say ‘Yes!’ to making things we are doing, much, much better. A good environment is where people know they are able to concentrate on what’s important and won’t be taken off on a bunch of conflicting or barely-relevant problems that will negatively affect everything.

* Reinforce the message: Why are we here? What are we doing? What is important? What progress have we made? What have we learned? No one likes working in a vacuum. Every week we have a team meeting where I try to tie everything together and answer these questions for the whole team. Sure it takes a lot of time to prepare this every week, but again, if helps the team, it’s exactly the kind of thing I should be spending my time on first.

* Core values: It’s a mistake to try to operate without a good set of core values. You need to define the 1-3 things that *everything* should be measured by. And re-inforce those values constantly, in every meeting, in every conversation. It helps people make decisions that are consistent with each other. Without them you end up with a team that’s constantly frustrated by choices that conflict, making everyone’s lives more difficult.

This list isn’t meant to be a prescription for how to do things. Nor is it all-inclusive. It’s simply some of the things I believe we do here at Insomniac that make it a great place to work. It’s not perfect, we have our share of conflicts and difficulties. But the one constant is that everyone knows we consistently work hard to make things *better*. Even looking back just one year, our team, our processes and our environment are radically different. Better. I hope I can say the same a year from now as well. I haven’t always done all of these things well, and I continue to struggle with many of them, but in order to create a good environment that people want to be in, I have to put the same kind of dedication toward making things better as I would into any technical problem I have as a programmer.

My advice to those people working in toxic environments? Get out. If your management isn’t working at least as hard as you are to make sure that you are always at your best, that you are constantly learning and growing, and that you are respected as a professional and as a human being then you owe it to yourself to find somewhere that will. You only get so much time on this planet, don’t waste it on them.

Oh, and I’ll also note that we’re hiring :)

Mike.

PS: You can follow me @mike_acton on Twitter.

[15 January 2011] I’ll leave you with this. Have a passion for what you do and do what you’re passionate about. Don’t let people drag you down. And it really, really doesn’t have to suck.

And watch this. A worthy reminder to all of us. Don’t you dare waste your time by Gabrielle Bouliane:

Recently we’ve gone through our annual review process here at Insomniac. I wanted to share a bit about my process this year.

In the past, my own review process felt a little unstructured and mostly focused on the individual tasks accomplished through the year. While that has had value, I didn’t feel like it really captured the whole picture. It’s especially important to me to focus on the growth of our Core team here as creative professionals. And I really want to make sure that I stay focused on that over time, too.

I have these 15 questions that I want to answer for every person on the team:
1. What did I see as your Super Power this year?
2. What do I think your top accomplishments were this year?
3. How do I think you have grown the most this year?
4. What is the biggest idea I think you have inspired this year?
5. How do I think you have improved or served the team this year?
6. How do I think you have improved or served production this year?
7. What do I think is the biggest difference you have made this year?
8. What do I think are the biggest problems you have solved this year?
9. What do I think are the best risks you have taken this year?
10. How do I think you have made Insomniac a better place to work this year?
11. What do I think the biggest challenges you have overcome this year?
12. How do I think you have inspired others this year?
13. How do I think you have best demonstrated your passions this year?
14. How do I think you have embodied our vision this year?
15. How do you stack up?

My process has been:
-> First, I answer these questions (as honestly and straightforwardly as I can) about each person on the team.
-> Then I do interviews in person about other members of the team. Basically, I sit down and ask different people on the team to answer the questions above about a peer, or peers. (Between 1 and 3 others.) Doing this face-to-face has really been the only effective way. It becomes a conversation and it doesn’t have the same barrier to entry as asking someone to write up a review. I just take lots of notes.
-> Now, the above together give me a fairly coherent picture of each person’s year and at that point I combine them into review text.
-> Some people have also answered the above questions for themselves. I think that gives a good chance to reflect.

I prefer this process a lot. It’s something I can do enthusiastically instead of grudgingly. The questions help me to get closer to the heart of how we can best build on each person’s strengths for the new year. They give some structure to the conversation and force me in particular to remember each area of impact. But most valuable are theinterviews with the team. Inevitably, even where questions seem similar, just the slight variation jogs a memory or helps them to make a connection. It’s definitely led to some great feedback and hopefully, as I push all that back to the individual, gives them perspective on how they can grow as an individual/professional.

My questions to you:
-> What methods have you used in your review process that you’ve really liked, or that you’ve thought were really super effective?
-> Or, what things have your leads done now or in the past that have really helped you reflect or grow in your reviews?

Mike.

PS: You can follow me @mike_acton on Twitter.

Lately I’ve been thinking about how we can make game development itself even more fun. As a rule, we are experts in taking something which is not inherently fun or even interesting (e.g. pushing a bunch of buttons) and making that a joyful experience. Over the years, both individually and as an industry, we have learned a lot of lessons on how to motivate and inspire people. So much so that the “gamification” of various processes outside the game industry has become a really hot business topic lately.

See for example, Seth Priebatsch’s TED presentation on “The game layer on top of the world” –

I am also inspired by Volkswagon’s FunTheory project where they have done some small, but very interesting, experiments on adjusting behavior by making the mundane and tedious tasks of life a little more fun.

For example:

Side note on the video above: One of our Engine Programmer’s made the observation that the most effective way of getting people to take the stairs would probably just be to remove the escalator. :)

The idea is to take our understanding of games, motivation, creativity and joy and apply that back on ourselves. And of course this doesn’t just apply to game development. Ask yourself: How could you make your job more fun? How could you make it a game? What kind of difference would that make?

Here’s an example from our team: Lately we’ve been experimenting with a “5-minute Show and Tell.” Every week in our Core team meeting, one person is given 5 minutes to share anything they think is interesting. It doesn’t have to be related to games or development. It’s a little bit of fun, a whole lot of just plain interesting and you just never know what you’re going to pick up and what’s going to have some relevance for people.

Mike.

PS: You can follow me @mike_acton on Twitter

Recently we've been asking ourselves some hard questions:
  • What is it that we want to focus on?
  • What's most important to us?
  • What do we want to make?
 
And the answer is simple:
 
We want to give you guys, our fans and players, the best looking games you can buy on a console.
 
You may have already seen Ratchet and Clank Future: A Crack in Time (available in US stores now!)
 
 
I'm really proud of what our art and production teams accomplished in this game. It's a great looking game, a ton of fun to play and is 60fps.
 
And it's that last point that I want to talk about today. One of the long-standing sacred cows here at Insomniac is framerate. We’ve long viewed a solid framerate as both a sign of a quality product and professionalism as developers. It’s always been point of pride in our work and considered an extremely serious part of our development process.
 
However, during development, there are hard choices to be made between higher quality graphics and framerate. And we want to make the right choices that reflect our commitment to providing you with the best looking games out there. To that end, our community team did some research into the question of framerate. The results perhaps confirmed what I’ve known for a long time, but found it difficult to accept without evidence. They found that:
 
  • A higher framerate does not significantly affect sales of a game.
  • A higher framerate does not significantly affect the reviews of a game.
 
And in particular they found that there was a clear correlation between graphics scores in reviews (where they are provided) and the final scores. And they found no such correlation between framerate and the graphics scores nor the final scores. As an interesting side-note, our team also found no direct correlation between gameplay scores and final scores, however it does appear that gameplay scores are also influenced by graphics scores. i.e. Better looking games appear to be more “fun” to reviewers, in general.
 
 
 
After reviewing our internal research, I decided to take this question to the public. I wanted to see what the players themselves thought of this question. Here are the results of that poll
 
 
The first thing I noted in reviewing these results was that 16% of the respondents said they wouldn’t buy a non 60fps game. Now, considering the top selling games and the market research, I take that to mean one of two things:
 
  • People are big fat liars. Sales numbers clearly contradict this pattern. Or,
  • The group responding to this poll in the first place was a self-selected group of people with an interest in framerate in the first place. Which may also explain why that last group is represented by such a small response rate in the poll results.
 
Based on the research, the informal polling and various conversations with fans and other game buyers, I’ve come to the following conclusions: 
 
  • Framerate is important, but not critically so. When there is a clear choice between framerate and improved graphics, graphics should win. The correlation with review scores is clear.
     
  • There is virtually no advantage in sales or reviews of a 60 fps game versus a 30 fps game.
     
  • Only a minority of players notice framerate as a significant issue of any kind.
     
  • Framerate should be as consistent as possible and should never interfere with the game. However, a drop in framerate is interestingly seen by some players as a reward for creating or forcing a complex setup in which a lot of things must happen on the screen at once. As in, “Damn! Did you see that? That was crazy!”
     
  • A solid framerate is still a sign of professional, well-made product. When there is a trade-off for framerate, it needs to be clearly worth it. i.e. It must introduce clear improvements on what the player sees, and never used as an excuse to not optimize the game or art.
 
 
What does all of this mean, really?
  
It means that framerate is still important to us here at Insomniac, but it’s not on the same pedestal it was before. And that Ratchet and Clank Future: A Crack in Time will probably be Insomniac’s last 60fps game.
 
 
Mike.
 
PS (Update 30 October 2009)
 
 
dixee wrote:
Plus, there are other advantages. You can load things twice as fast (assuming it's being streamed during gameplay) and calculate twice as many collisions. Frame rate involves more than just drawing methods, unless I'm drastically mistaken. Am I?
 
You really can't load things any faster regardless of your framerate. The speed (seek and throughput rate) of loading is limited by device you're loading from (e.g. the Bluray). But yes, a lower framerate does give you "extra" time to do other operations as well, outside of rendering. Collision, as you mentioned, is an example.
 
Rhez Darkhoof wrote:
Framerate is nice but gameplay and graphics probably have more effect on the overall experience. I mean how high can we really get the framerate before the human eye can't distinguish the difference.?
 
The human eye and brain are seriously sophisticated equipment. It's certainly possible to distinguish very high framerates. Much higher, in fact, than 60fps. But the question is more about how important is it to most players (and reviewers) when when offered the choice against "better graphics" of some kind. We want to give players better graphics, because ultimately that's what so many are asking for.
 
Arelius wrote:
Mike, I'd like to make a few strong counter points. First of all, regarding your Poll, and the fact that 16% said they wouldn't buy a non 60 FPS game. Keep in mind there were no other "I prefer a 60 FPS game" option, so while the exact text may have said "60 FPS or I won't buy" it got people saying "I prefer 60 FPS and it influences my purchases."
 
So, yeah. I should have clarified the point. That poll was extremely informal and didn't play a significant part in the decision process. It was more a point of interest. However, we can see through sales data what people are saying when they speak with their hard-earned money. And we simply don't see any significant correlation between framerate and what people want to buy.
 
But I don't want to dismiss the importance of framerate, either. As I said, it is still important to us. We certainly understand how framerate can affect the game experience. And we definitely have people here that prefer 60fps as well. But when it comes right down to it, when we have to make a choice, that choice has to be made based on what we think you guys will like better. And what you'd actually prefer to buy, in general. It's not a perfect system.
 
 
mgibson wrote:
That graph of overall scores and graphics scores makes no sense. The horizontal axis is meaningless.
 
Just to be clear, what I wrote was definitely not about the details of our research. There is clearly a lot of information that I didn't present. Nor did I present it in a way that could allow for any kind of rigorous analysis. I'm more speaking about what conclusions we've drawn and why we think it's interesting. 
 
The horizontal axis represents the individual games. The vertices are the only things that have any meaning. The lines drawn between them are not meaningful data themselves, but do help visualize the pattern for an informal view.
 
MiZa wrote:
I know, my words are wasted though as soon as I read your mission statement. If it reads "We want to give you guys, our fans and players, the best looking games you can buy on a console" instead of "We want to give you guys, our fans and players, the best games you can buy on a console" then there is no hope.

That's a good point, MiZa. And we do want to provide you the best games you can buy on a console. Though here, I'm speaking specifically about engine (and art) choices, and ultimately this question is fairly narrowly focused on the A/V experience.
 
superdynamite wrote:
Let me start off by saying that frame rate plays a tremendous part in the gameplay of a "Next-Gen" videogame. The statement that was made by Insomniac, sighting "no correlation between 60 fps benchmark to yield higher retail sales" is complete and utter nonsense,
 
I appreciate your view here, superdynamite. But I'm going to stick by our research. Certainly you can look at many current games that run at 30 fps and ask if their sales have been impacted by not being 60 fps. Additionally, I'd be surprised if you could find a single review (in similar genres of games that we work on) that penalized a game for not being 60 fps.
 
Thanks so much for all of your heartfelt feedback everyone! And keep in mind that I'm certainly not saying 60 fps has no value or that framerate isn't important. It's simply about focusing on what we think is most important and devoting resources to that. So that when you pick up our games, you know you're going to get an awesome graphical experience. Along with great fun.
 
 
 
 
One of the things we talked about this year at GDC was what we called the "Three Big Lies of Software Development." How much programmers buy into these "lies" has a pretty profound effect on the design (and performance!) of an engine, or any high-performance embedded system for that matter.
 
You should keep in mind that all of this is just my opinion, based on my own personal experience. Your mileage may vary. :)
 
(Lie #1) Software is a platform
 
I blame the universities for this one. Academics like to remove as many variables from a problem as possible and try to solve things under "ideal" or completely general conditions. It's like old physicist jokes that go "We have made several simplifying assumptions… first, let each horse be a perfect rolling sphere…"
 
The reality is software is not a platform. You can't idealize the hardware. And the constants in the "Big-O notation" that are so often ignored, are often the parts that actually matter in reality (for example, memory performance.) You can't judge code in a vacuum. Hardware impacts data design. Data design impacts code choices. If you forget that, you have something that might work, but you aren't going to know if it's going to work well on the platform you're working with, with the data you actually have.
 
(Lie #2) Code should be designed around a model of the world
 
There is no value in code being some kind of model or map of an imaginary world. I don't know why this one is so compelling for some programmers, but it is extremely popular. If there's a rocket in the game, rest assured that there is a "Rocket" class (Assuming the code is C++) which contains data for exactly one rocket and does rockety stuff. With no regard at all for what data tranformation is really being done, or for the layout of the data. Or for that matter, without the basic understanding that where there's one thing, there's probably more than one.
 
Though there are a lot of performance penalties for this kind of design, the most significant one is that it doesn't scale. At all. One hundred rockets costs one hundred times as much as one rocket. And it's extremely likely it costs even more than that! Even to a non-programmer, that shouldn't make any sense. Economy of scale. If you have more of something, it should get cheaper, not more expensive. And the way to do that is to design the data properly and group things by similar transformations.
 
(Lie #3) Code is more important than data
 
This is the biggest lie of all. Programmers have spent untold billions of man-years writing about code, how to write it faster, better, prettier, etc. and at the end of the day, it's not that significant. Code is ephimiral and has no real intrinsic value. The algorithms certainly do, sure. But the code itself isn't worth all this time (and shelf space! – have you seen how many books there are on UML diagrams?). The code, the performance and the features hinge on one thing – the data. Bad data equals slow and crappy application. Writing a good engine means first and formost, understanding the data.