Ron Pieket: A Client/Server Tools Architecture

Ron Pieket wrote a bit more about our game tools webapp architecture on his blog. Reprinted here:

A Client/Server Tools Architecture

I mentioned Insomniac’s client/server tools architecture during my GDC talk earlier this year, and this topic has generated considerable interest. This article gives a very high level overview of the system as it is currently implemented.

All interactive productivity tools developed over the last two years at Insomniac Games have been built on this architecture. The basic idea is that the edited document is not kept in the memory space of the editor application itself, but rather each individual modification is transmitted to a server application, running on the same machine as the editor. The server maintains the authoritative document. Any number of documents may be open at any time. Any number of editors may communicate with the same local server application. The server provides various document related services.

There are several benefits from this architecture:

  • Crash Proofing

    The server application is comparatively simple, and matures early on. Editors are in constant development and may therefore be unstable. But since your work is managed by the server, up to the very last edit, it will survive a crash. Just restart the editor application and all your changes will still be there. Even the undo queue will survive.

  • Multiple Views

    LunaServer allows multiple editors to display/edit the same document. Changes that are made in one will immediately appear in the others.

  • “Free” Undo/Redo

    Undo/redo is handled by the server. All editors (clients) get undo/redo without a single line of code.

  • “Free” Load/Save/Revert

    All disk operations are handled by the server. All editors get this for free.

  • “Free” Perforce Integration

    Perforce integration is handled by the server. All editors get this for free.

  • Consistent Behavior

    Because undo/redo, file operations and Perforce integration are handled by the server, the user is presented with a very consistent interface

LunaServer

 

At the heart of the system is LunaServer. This is a server application running on the user’s own machine. No network connection is required. Although LunaServer uses network protocols for communication with the editors, and it is quite capable of running remotely, we have not found it necessary to do so. It was never intended to be used over a network. Every user machine runs its own independent LunaServer.

LunaServer implements a RESTful HTTP protocol.

Any editor that we write at Insomniac Games runs as a LunaServer client. We currently have implemented a world editor, an animation set editor, a visual script editor, an effect editor, a material editor, and several more. Any combination of editors and any number of instances of the same editor may be running at the same time. The same document may be opened in any number of editors. Most editors are written in JavaScript, one is written in C++, and one is written in Flash.

JSON

 

LunaServer acts as a JSON document manager. All assets are JSON documents. All editors “speak” JSON, even the C++ and Flash based editors. Even though the C++ application maintains a version of the document in binary form, this is considered a cache for 3D rendering and mouse interaction. The JSON document is the authoritative version, and it is always synchronized immediately with any changes made in the binary version.

Delta JSON

 

Changes are transmitted between LunaServer and the clients in what we call a “delta JSON” format.

Say, LunaServer contains a document representing an asset named DreamCar. The document might look like this:

{
  "model":"veyron.dae",
  "color":"black",
  "topSpeed":267
}

Now imagine that the user is using one of the editors to change the “color” property. The editor would update its local copy of the document, and transmit the following delta JSON:

{
  "color":"red"
}

In order to transmit this change, the delta JSON itself is wrapped into a “change document” like this:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "color":"red"
  }
}

When this is received by LunaServer, it knows to update the “color” property of the DreamCar document with the new value.

New properties can be added and existing properties can be removed. We have adopted the convention that a null value in a delta JSON object means “remove”. (As a consequence, LunaServer documents cannot contain null values, only delta JSON can)

For example, if an editor needs to send a change to LunaServer to add a new property “price”, and remove property “topSpeed”, it would make those changes locally, and send the following change document to LunaServer:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "topSpeed":null,
    "price":1.7e6
  }
}

After processing both change documents, DreamCar will read the same on the client side and in LunaServer:

{
  "model":"veyron.dae",
  "color":"red",
  "price":1.7e6
}

Property “model” was unchanged, “topSpeed” was removed, “color” was modified, and “price” was added.

Synchronizing Clients

 

As I mentioned, multiple editors may be running at the same time. They may even display and edit the same document. In addition to transmitting changes as delta JSON to LunaServer, each client must poll for document changes from LunaServer. If any editor changes a document, the delta JSON that is received by LunaServer is also transmitted to others, as they poll for changes.

(LunaServer implements a RESTful protocol, and therefore cannot push changes to the clients. So clients must poll)

MongoDB

 

LunaServer itself is backed by MongoDB. This was a natural choice. JavaScript, JSON, and MongoDB work very well together.

Perforce

 

Asset documents are shared between team members through Perforce, and Perforce maintains a document version history. LunaServer will interact with Perforce when an editor attempts to modify an asset document that is not currently writable. LunaServer manages load, save, revert, check-in, check-out, and other operations. It will prompt the user for file names and confirmations as needed. LunaServer does all of this automatically, relieving all editors from these responsibilities. All the editor code needs to do is send and receive asset changes in delta JSON format.

LunaTracker and files on disk

 

Perforce does not interact directly with LunaServer or MongoDB. Instead, it synchronizes files on the user’s hard drive. Every document in LunaServer’s database has a counterpart on disk. Synchronization of the database with changes on disk is the responsibility of LunaTracker. LunaTracker is a service that watches the asset document folders, and updates LunaServer when a new version of the file is detected.

Undo/Redo

Before LunaServer applies a delta JSON to its copy of the document, it will compute an inverse delta JSON.

If you recall, our DreamCar document started out looking like this:

{
  "model":"veyron.dae",
  "color":"black",
  "topSpeed":267
}

And our first change was to set the “color” to “red”:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "color":"red"
  }
}

LunaServer will store this change document in its redo queue, and transmit it to clients that poll for changes. It will also store the inverse change in the undo queue. The inverse change document will look like this:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "color":"black"
  }
}

When told to perform an undo operation, LunaServer will simply process the last document in the undo queue, and the car is black again.

Sessions

 

If multiple editors are open at once, their undo queues must kept separate from each other. When an editor is launched, LunaServer creates a unique session ID and a session document. Undo records are organized by session ID. The session document contains various bookkeeping data such as the list of documents currently opened by the editor instance, object selection, and other settings.

Multiple editor instances may make changes to the same document. Changes that are made in one instance are transmitted to all. But undo for any particular change is only available from the instance or session where it originated.

In some cases (actually only one), what appears as a single editor to the user is in fact implemented as two (or more) separate executables. This is the case with the world editor. The 3D view is a C++ application, and the supporting 2D user interface is a separate application written in JavaScript and HTML 5. In such a case, multiple editors may share the same session ID and session document. Their undo records and selections are shared and synchronized.

Session document changes are handled in the same manner as asset document changes. They are transmitted, received and processed in the same delta JSON format. The target will be the session document. This makes changes in selection, opening and closing of documents undoable operations.

Limitations

 

Although the architecture allows for multiple simultaneous editors, it is designed for a single user on a single machine. There is therefore no need to arbitrate multiple simultaneous changes, and there is no provision for it. Although multiple editors may have the same document open, the single user can operate only one at a time. If somehow simultaneous changes are made to the same property in the same document, in two different editor instances, some changes may be lost.

Conclusion

 

The Insomniac LunaServer architecture was started about two years ago. It seemed like a good idea at the time. I don’t think that anyone involved realized back then, how good the idea really was. The server side document storage, the simple synchronization between instances, the centralized undo/redo system and disk access have made our tools more robust, more flexible, easier to use, and their development much simpler.