LibSerialize

LibSerialize is a Lua library for efficiently serializing/deserializing arbitrary values. It supports serializing nils, numbers, booleans, strings, and tables containing these types.

It is best paired with LibDeflate, to compress the serialized output and optionally encode it for World of Warcraft addon or chat channels. IMPORTANT: if you decide not to compress the output and plan on transmitting over an addon channel, it still needs to be encoded, but encoding via LibDeflate:EncodeForWoWAddonChannel() or LibCompress:GetAddonEncodeTable() will likely inflate the size of the serialization by a considerable amount. See the usage below for an alternative.

Note that serialization and compression are sensitive to the specifics of your data set. You should experiment with the available libraries (LibSerialize, AceSerializer, LibDeflate, LibCompress, etc.) to determine which combination works best for you.

Usage:

-- Dependencies: AceAddon-3.0, AceComm-3.0, LibSerialize, LibDeflate
MyAddon = LibStub("AceAddon-3.0"):NewAddon("MyAddon", "AceComm-3.0")
local LibSerialize = LibStub("LibSerialize")
local LibDeflate = LibStub("LibDeflate")

function MyAddon:OnEnable()
    self:RegisterComm("MyPrefix")
end

-- With compression (recommended):
function MyAddon:Transmit(data)
    local serialized = LibSerialize:Serialize(data)
    local compressed = LibDeflate:CompressDeflate(serialized)
    local encoded = LibDeflate:EncodeForWoWAddonChannel(compressed)
    self:SendCommMessage("MyPrefix", encoded, "WHISPER", UnitName("player"))
end

function MyAddon:OnCommReceived(prefix, payload, distribution, sender)
    local decoded = LibDeflate:DecodeForWoWAddonChannel(payload)
    if not decoded then return end
    local decompressed = LibDeflate:DecompressDeflate(decoded)
    if not decompressed then return end
    local success, data = LibSerialize:Deserialize(decompressed)
    if not success then return end

    -- Handle `data`
end

-- Without compression (custom codec):
MyAddon._codec = LibDeflate:CreateCodec("\000", "\255", "")
function MyAddon:Transmit(data)
    local serialized = LibSerialize:Serialize(data)
    local encoded = self._codec:Encode(serialized)
    self:SendCommMessage("MyPrefix", encoded, "WHISPER", UnitName("player"))
end
function MyAddon:OnCommReceived(prefix, payload, distribution, sender)
    local decoded = self._codec:Decode(payload)
    if not decoded then return end
    local success, data = LibSerialize:Deserialize(decoded)
    if not success then return end

    -- Handle `data`
end

API:

• LibSerialize:SerializeEx(opts, ...)

  Arguments:

  • opts: options (see below)
  • ...: a variable number of serializable values

  Returns:

  • result: ... serialized as a string

• LibSerialize:Serialize(...)

  Arguments:

  • ...: a variable number of serializable values

  Returns:

  • result: ... serialized as a string

  Calls SerializeEx(opts, ...) with the default options (see below)

• LibSerialize:Deserialize(input)

  Arguments:

  • input: a string previously returned from LibSerialize:Serialize()

  Returns:

  • success: a boolean indicating if deserialization was successful
  • ...: the deserialized value(s), or a string containing the encountered Lua error

• LibSerialize:DeserializeValue(input)

  Arguments:

  • input: a string previously returned from LibSerialize:Serialize()

  Returns:

  • ...: the deserialized value(s)

• LibSerialize:IsSerializableType(...)

  Arguments:

  • ...: a variable number of values

  Returns:

  • result: true if all of the values' types are serializable.

  Note that if you pass a table, it will be considered serializable even if it contains unserializable keys or values. Only the types of the arguments are checked.

Serialize() will raise a Lua error if the input cannot be serialized. This will occur if any of the following exceed 16777215: any string length, any table key count, number of unique strings, number of unique tables. It will also occur by default if any unserializable types are encountered, though that behavior may be disabled (see options).

Deserialize() and DeserializeValue() are equivalent, except the latter returns the deserialization result directly and will not catch any Lua errors that may occur when deserializing invalid input.

Note that none of the serialization/deseriazation methods support reentrancy, and modifying tables during the serialization process is unspecified and should be avoided. Table serialization is multi-phased and assumes a consistent state for the key/value pairs across the phases.

Options:

The following serialization options are supported:

• errorOnUnserializableType: boolean (default true)
  • true: unserializable types will raise a Lua error
  • false: unserializable types will be ignored. If it's a table key or value, the key/value pair will be skipped. If it's one of the arguments to the call to SerializeEx(), it will be replaced with nil.
• filter: function(t, k, v) => boolean (default nil)
  • If specified, the function will be called on every key/value pair in every table encountered during serialization. The function must return true for the pair to be serialized. It may be called multiple times on a table for the same key/value pair. See notes on reeentrancy and table modification.

If an option is unspecified in the table, then its default will be used. This means that if an option foo defaults to true, then:

• myOpts.foo = false: option foo is false
• myOpts.foo = nil: option foo is true

Customizing table serialization:

For any serialized table, LibSerialize will check for the presence of a metatable key __LibSerialize. It will be interpreted as a table with the following possible keys:

• filter: function(t, k, v) => boolean
  • If specified, the function will be called on every key/value pair in that table. The function must return true for the pair to be serialized. It may be called multiple times on a table for the same key/value pair. See notes on reeentrancy and table modification. If combined with the filter option, both functions must return true.

Examples:

1. LibSerialize:Serialize() supports variadic arguments and arbitrary key types, maintaining a consistent internal table identity.

   local t = { "test", [false] = {} }
   t[ t[false] ] = "hello"
   local serialized = LibSerialize:Serialize(t, "extra")
   local success, tab, str = LibSerialize:Deserialize(serialized)
   assert(success)
   assert(tab[1] == "test")
   assert(tab[ tab[false] ] == "hello")
   assert(str == "extra")

2. Normally, unserializable types raise an error when encountered during serialization, but that behavior can be disabled in order to silently ignore them instead.

   local serialized = LibSerialize:SerializeEx(
       { errorOnUnserializableType = false },
       print, { a = 1, b = print })
   local success, fn, tab = LibSerialize:Deserialize(serialized)
   assert(success)
   assert(fn == nil)
   assert(tab.a == 1)
   assert(tab.b == nil)

3. Tables may reference themselves recursively and will still be serialized properly.

   local t = { a = 1 }
   t.t = t
   t[t] = "test"
   local serialized = LibSerialize:Serialize(t)
   local success, tab = LibSerialize:Deserialize(serialized)
   assert(success)
   assert(tab.t.t.t.t.t.t.a == 1)
   assert(tab[tab.t] == "test")

4. You may specify a global filter that applies to all tables encountered during serialization, and to individual tables via their metatable.

   local t = { a = 1, b = print, c = 3 }
   local nested = { a = 1, b = print, c = 3 }
   t.nested = nested
   setmetatable(nested, { __LibSerialize = {
       filter = function(t, k, v) return k ~= "c" end
   }})
   local opts = {
       filter = function(t, k, v) return LibSerialize:IsSerializableType(k, v) end
   }
   local serialized = LibSerialize:SerializeEx(opts, t)
   local success, tab = LibSerialize:Deserialize(serialized)
   assert(success)
   assert(tab.a == 1)
   assert(tab.b == nil)
   assert(tab.c == 3)
   assert(tab.nested.a == 1)
   assert(tab.nested.b == nil)
   assert(tab.nested.c == nil)

Encoding format:

Every object is encoded as a type byte followed by type-dependent payload.

For numbers, the payload is the number itself, using a number of bytes appropriate for the number. Small numbers can be embedded directly into the type byte, optionally with an additional byte following for more possible values. Negative numbers are encoded as their absolute value, with the type byte indicating that it is negative. Floats are decomposed into their eight bytes, unless serializing as a string is shorter.

For strings and tables, the length/count is also encoded so that the payload doesn't need a special terminator. Small counts can be embedded directly into the type byte, whereas larger counts are encoded directly following the type byte, before the payload.

Strings are stored directly, with no transformations. Tables are stored in one of three ways, depending on their layout:

• Array-like: all keys are numbers starting from 1 and increasing by 1. Only the table's values are encoded.
• Map-like: the table has no array-like keys. The table is encoded as key-value pairs.
• Mixed: the table has both map-like and array-like keys. The table is encoded first with the values of the array-like keys, followed by key-value pairs for the map-like keys. For this version, two counts are encoded, one each for the two different portions.

Strings and tables are also tracked as they are encountered, to detect reuse. If a string or table is reused, it is encoded instead as an index into the tracking table for that type. Strings must be >2 bytes in length to be tracked. Tables may reference themselves recursively.

Type byte:

The type byte uses the following formats to implement the above:

• NNNN NNN1: a 7 bit non-negative int
• CCCC TT10: a 2 bit type index and 4 bit count (strlen, #tab, etc.)
  • Followed by the type-dependent payload
• NNNN S100: the lower four bits of a 12 bit int and 1 bit for its sign
  • Followed by a byte for the upper bits
• TTTT T000: a 5 bit type index
  • Followed by the type-dependent payload, including count(s) if needed