Pomelo data compression protocol

#Pomleo data compression protocol

In pomelo 0.3, we support two data compression method : route compression based on dictionary and message compression based on protobuf.

##Route compression based on dictionary ###The route field In Pomelo, we use the route field to determine the corresponding handler of a message. There are two kinds of route in Pomelo: The route used at server side, to dispatch a request to corresponding handler, and the route used at client side, the handle the messages published by server.

• The server side routes are generated by Pomelo at start, the content identify its corresponding handler. For example, the route "area.playerHandeler.attack" means it's handler is at server 'area', for the handler is 'playerHandler', and the method is 'attck'.

• The client side routes are used for to publishing messages, the client side used them the identify different messages, and the routes are defined by users.

When the data usage are not important, the overhead of route are acceptable. By in the bandwidth intensive environment, for example, on the mobile phones, the data traffic are much more expensive and limited. It will be necessary to compress the route field;

###The route compression in Pomelo In Pomelo, we use different strategy for the two kinds of route:

• For the route generated by the system, as 'area.playerHandler.attack', we give each route a unique id(16 bits short int).
• For the route defined by user, like 'onMove', 'onAttack'. because we don't know which routes are used, it's the duty of the developer to provide a dictionary for the route he/she used, and pomelo will assign a unique id the each routes.

When open the route compression function, when there is a message, the route will be replaced by it's id at sender side. At the server side, the id will be restore to the original route. And the procedure is invisible for the developers, you just open the dictionary function and provide a dictionary, all the job are done by pomelo. The below code is how to open the route compression function at app.js:

	app.set('connectorConfig',
		{
			connector : pomelo.connectors.hybridconnector,
			heartbeat : 3,
			useDict : true,
			handshake : function(msg, cb){
				cb(null, {});
			}
		});

The 'userDict : true' means to open the route compression. The default route compression will compress all the system generated routes, like 'area.playerHandler.move' etc.

To add the dictionary for user defined route, like 'onMove', 'onAttack', you need to add all the use defined routes at the /game-server/config/dictionary.json file, here is the example:

[
	"onDropItem",
	"onAttack",
	"onDied",
	"onMove",
	"onUpgrage",
	"onPickItem",
	"onRevive",
	"addEntities",
	"onRemoveEntities",
	"onPathCheckout"
]

At see above, the content of ditionary.json is an array of string, contends all the user defined routes need to be compressed.

##Data compress based on protobuf In pomleo 0.3, we use protobuf protocol the compress the message. Compare to other encode protocol like xml, json, the protobuf protocol has smaller size. In the project Lordofpomelo, the message size encode by protobuf can be 80% smaller than Json format on average.

###Protobuf protocol Protobuf are made by Goole, which is mainly used in rpc invoke and file encode. The Google protobuf has to parts : the protobuf protocol use binary encode, then the code generater based on protobuf meta data. The procedure of using google protobuf is as follow :

As in the above picture, to use the google protbuf, you have to generate code for each message need to be encode, and use the generated code to encode/decode the message. The disadvantage is obviously : it is complex to use, need to generate lots of code, change is very difficult.

So in pomelo protobuf, we use a differnt way to use the protobuf protocol.

###Pomelo-protobuf In Pomelo-protobuf we implement a universal protobuf encoder/decoder. By directly use the meta data in protos, we can decode/encode protobuf message without generate any code. To add/modify a message, you just need to modify the protos files. The procedure of Pomelo-protobuf is as follow :

To read more information about pomelo protobur, you can see here pomelo-protobuf.

###Use Pomelo-protobuf In pomelo 0.3, at server side, we include pomelo-protobuf as a component of pomelo, at client side, the protbuf encoder/decoder are integrated in the clients. So you don't need to directly include the Pomelo-protobuf module, it's already in your project.

As in the google protobuf, to use the protobuf protocol, the first thing you do is to write the proto files. In Pomelo-protobuf, all the protos are defined in the same file instead of a single proto for a single message in google protobuf. And there is a unique key for each proto, the format is as foolow:

"onMove" : {
    "required uInt32 entityId" : 1,
    "message Path": {
      "required uInt32 x" : 1,
      "required uInt32 y" : 2
    },
    "repeated Path path" : 2,
    "required uInt32 speed" : 3
  },
  "onAttack" : {
    "required uInt32 attacker" : 1,
    "required uInt32 target" : 2
  }

As you can see, in pomelo, the key is the route of the message. The syntax of the proto file is the same as in google protobuf, we just use the json format to write them, which is much more easy to use in JS.

In pomelo-protobuf, we support different protos for encoder and decoder. In pomelo, it means for the same route, we can use different format of a message for server side and client side. This function is useful, because for a request message, like 'area.playerHandler.attck', the request and response can be completely different, but with the same route. So there are two protos.json in pomelo, /game-server/config/serverProtos.json and /game-server/config/clientProtos.json, the relation of the them is as follow:

clientProtos.json and serverProtos.json will be created at the pomelo init command. The two protos files will be empty at create time, and it's the developer's job to add message proto in them. All of our demos : pomelo-chat, treasures, lordofpomelo are use the protobuf to encode the message.

To use the protobuf fuction, it is also need to set the useProtobuf flag in the app.js to tell pomelo to open the protobuf encode/decode function, the config is as follow:

	app.set('connectorConfig',
		{
			connector : pomelo.connectors.hybridconnector,
			heartbeat : 3,
			useProtobuf : true,
			handshake : function(msg, cb){
				cb(null, {});
			}
		});

The 'useProtobuf:true' is the flag, after set that, pomelo will transport the protos data to client, and open protobuf decompress function.

##Compatibility with older pomelo The dictionary compression and pomelo-protobuf are fully compatible with old project, you can open them in your project without any problem.

After open the dictionary compression in the project, all the request route are automatically compressed, and all the routes in publish message, which use user define route, are note compressed. When you open the protobuf function, all the message defined in the proto files will be encode by protobuf at transport time, with other messages will not be effected.