pllua

Embeds Lua into PostgreSQL as a procedural language module.

Please report any remaining bugs or missing functionality on github.

Currently it should build against (recent point releases of) pg versions 9.5 to 16. It is known that this module will never work on pg versions before 9.5 (we rely critically on memory context callbacks, which were introduced in that version).

Lua 5.4 (only 5.4.2 onwards, excluding 5.4.5), Lua 5.3, and LuaJIT 2.1beta (with COMPAT52) are fully supported at this time. (Lua 5.4.0 and 5.4.1 are checked for at runtime and rejected, because they would otherwise require runtime stack limit calculations; 5.4.5 is checked for at compile and run time and rejected because of an incompatible ABI/API change.)

Documentation is being migrated from this README to a more comprehensive document:

Documentation

COMPATIBILITY WITH 0.x (previous pllua-ng releases)

The server.* namespace is removed, to facilitate the compatibility option described below.

server.error, server.debug, etc. are now available via spi.error, spi.debug, or can be obtained from the 'pllua.elog' package via require(). If you want, you can do:

    pllua.on_common_init='server = require "pllua.elog"'

to restore the previous behavior.

COMPATIBILITY WITH 1.x (old pllua)

This version permits a degree of compatibility with pllua 1.x via the following setting:

    pllua.on_common_init='require "pllua.compat"'

This creates the server.* namespace with functions that match the 1.x calling conventions (e.g. passing args as tables). It also creates global functions fromstring(), subtransaction(), debug(), log(), info(), notice(), warning(), and setshared().

The following incompatibilities remain:

• coroutine.yield() with no result values in 1.x ended execution of the SRF; in this version it returns a NULL row and continues.

• The global error() is not modified, so using it to throw a table has the same effect it would have in plain Lua. In 1.x, a metatable was added to the thrown value in this case.

• The pgfunc library is not supported at all.

• The "readonly" parameter to server.execute and friends is ignored. All queries in a stable function are readonly, and all queries in a volatile function are read-write.

• Returning multi-dimensional arrays by doing a simple return of a Lua table is no longer supported.

The pllua.compat module is implemented in pure Lua (inside the sandbox in trusted mode), see src/compat.lua for the implementation.

Please report any incompatibilities discovered.

CHANGES

NOTE: the name of the module is now just "pllua", and its extension packaging is split into two according to usual practice for pl modules (in this case "pllua" for the trusted language and "plluau" for the untrusted language).

(Compared to the old pllua project:)

Some names and locations have been changed.

The old pllua.init table is gone. Instead we support three init strings (superuser-only): pllua.on_init, pllua.on_trusted_init, pllua.on_untrusted_init.

Note that the on_init string can be run in the postmaster process, by including pllua in shared_preload_libraries. Accordingly, on_init cannot do any database access, and the only function directly available from this module is print(), which in this environment will output to the server log as LOG: messages.

The on_init string can now access the trusted.allow() functionality, but only by doing an explicit require 'pllua.trusted'. e.g.

  local t = require 'pllua.trusted'
  t.allow{ "lpeg", "re" }

SPI functionality is now in global table spi and has different calling conventions:

  spi.execute("query text", arg, arg, ...)
  spi.execute_count("query text", maxrows, arg, arg, ...)
  spi.prepare("query text", {argtypes}, [{options}])
    - returns a statement object:
      s:execute(arg, arg, ...)  - returns a result table
      s:execute_count(maxrows, arg, arg, ...)  - returns a result table
      s:rows(arg, arg, ...) - returns iterator
      s:numargs() - returns integer
      s:argtype(argnum) - returns typeinfo
  spi.rows("query text", args...)
    - returns iterator

Execution now returns a table with no number keys (#t == 0) in the event of no matching rows, whereas the old version returned nil. The result is also currently a plain table, not an object.

spi.prepare takes an options table with these possible values:

  scroll = true or false
  no_scroll = true
  fast_start = true
  custom_plan = true
  generic_plan = true
  fetch_count = integer

Note that "scroll" and "no_scroll" are independent options to the planner, but we treat { scroll = false } as if it were { no_scroll = true } because not doing so would be too confusing. The fetch_count value is used only by the :rows iterator, to determine how much prefetch to use; the default is 50. (Smaller values might be desirable for fetching very large rows, or a value of 1 disables prefetch entirely.)

Cursors work:

  spi.findcursor("name")   - find already-open portal by name
  spi.newcursor(["name"])  - find existing cursor or create new one
  s:getcursor(args)   - get cursor from statement (can't specify name)
  c:open(stmt,args)   - open a cursor
  c:open(query,args)  - open a cursor
  c:isopen()          - is it open
  c:name()
  c:fetch([n, [dir]])  - fetch n rows in dir (default: forward 1)
  c:move([n, [dir]])

There can only be one cursor object for a given open portal - doing a findcursor on an existing cursor will always return the same object. (But note that this matching is by portal, not name - if a cursor was closed and reopened with the same name, findcursor will return a different object for the new cursor.) If a cursor is closed by external code (or transaction end), then the :isopen() state will be automatically updated (this happens when the portal is actually dropped). Cursor options are set on the statement object.

Refcursor parameters and results are transparently converted to and from SPI cursor objects.

:save on a statement is now a no-op - all statements seen by lua code have been passed through SPI_keepplan and are managed by Lua garbage collection. (It was never safe to do otherwise.)

(SPI interface is particularly subject to change - in particular to something more compatible with client-side database APIs)

print() is still a global function to print an informational message, but other error levels such as debug, notice are installed as spi.debug(), spi.warning() etc. spi.elog('error', ...) is equivalent to spi.error(...) and so on.

spi.error() and friends can take optional args:

  spi.error('message')
  spi.error('sqlstate', 'message')
  spi.error('sqlstate', 'message', 'detail')
  spi.error('sqlstate', 'message', 'detail', 'hint')
  spi.error({ sqlstate = ?,
              message = ?,
              detail = ?,
              hint = ?,
              table = ?,
              column = ?, ...})

Sqlstates can be given either as 5-character strings or as the string names used in plpgsql: spi.error('invalid_argument', 'message')

Subtransactions are implemented via pcall() and xpcall(), which now run the called function in a subtransaction. In the case of xpcall, the subtransaction is ended before running the error function, which therefore runs in the outer subtransaction. This does mean that while Lua errors in the error function cause recursion and are eventually caught by the xpcall, if an error function causes a PG error then the xpcall will eventually rethrow that to its own caller. (This is subject to change if I decide it was a bad idea.)

e.g.

  local ok,err = pcall(function() --[[ do stuff in subxact ]] end)
  if not ok then print("subxact failed with error",err) end

Currently there's also an lpcall function which does NOT create subtransactions, but which will catch only Lua errors and not PG errors (which are immediately rethrown). It's not clear yet how useful this is; it saves the (possibly significant) subxact overhead, but it can be quite unpredictable whether any given error will manifest as a Lua error or a PG error.

The readonly-global-table and setshared() hacks are omitted. As the trusted language now creates an entirely separate lua_State for each calling userid, anything the user does in the global environment can only affect themselves.

Type handling is all different. The global fromstring() is replaced by the pgtype package/function:

  pgtype(d)
    -- if d is a pg datum value, returns an object representing its
       type

  pgtype(d,n)
    -- if d is a datum, as above; if not, returns the object
       describing the type of argument N of the function, or the
       return type of the function if N==0

  pgtype['typename']
  pgtype.typename
  pgtype(nil, 'typename')
    -- parse 'typename' as an SQL type and return the object for it

  pgtype.array.typename
  pgtype.array['typename']
    -- return the type for "typename[]" if it exists

The object representing a type can then be called as a constructor for datum objects of that type:

  pgtype['mytablename'](col1,col2,col3)
  pgtype['mytablename']({ col1 = val1, col2 = val2, col3 = val3})
  pgtype.numeric(1234)
  pgtype.date('2017-12-01')
  pgtype.array.integer(1,2,3,4)
  pgtype.array.integer({1,2,3,4}, 4)        -- dimension mandatory
  pgtype.array.integer({{1,2},{3,4}},2,2)   -- dimensions mandatory
  pgtype.numrange(1,2)        -- range type constructor
  pgtype.numrange(1,2,'[]')   -- range type constructor

or the :fromstring method can be used:

  pgtype.date:fromstring('string')

In turn, datum objects of composite type can be indexed by column number or name:

  row.foo  -- value of column "foo"
  row[3]   -- note this is attnum=3, which might not be the third
              column if columns have been dropped

Arrays can be indexed normally as a[1] or a[3][6] etc. By default array indexes in PG start at 1, but values starting at other indexes can be constructed. One-dimensional arrays (but not higher dimensions) can be extended by adding elements with indexes outside the current bounds; ranges of unassigned elements between assigned ones contain NULL.

tostring() works on any datum and returns its string representation.

pairs() works on a composite datum (and actually returns the attnum as a third result):

  for colname,value,attnum in pairs(row) do ...

The result is always in column order.

ipairs() should NOT be used on a composite datum since it will stop at a null value or dropped column.

Arrays, composite types, and jsonb values support a mapping operation controlled by a configuration table:

  rowval{ map = function(colname,value,attno,row) ... return value end,
          null = (any value, default nil),
          discard = (boolean, default false)
        }
  arrayval{ map = function(elem,array,i,j,k...) ... return elem end,
            null = (any value, default nil),
            discard = (boolean, default false)
          }
  jsonbval{ map = function(key,val,...) ... return key,val end,
            null = (any value, default nil),
            discard = (boolean, default false),
            pg_numeric = (boolean, default false)
          }

The result in all cases is returned as a Lua table, not a datum, unless the "discard" option was given as true, in which case no result at all is returned.

The map function for arrays is passed as many indexes as the original array dimension.

The map function for jsonb values is passed the path leading up to the current key (not including the key) as separate additional parameters. The key is an integer if the current container is an array, a string if the container is an object, and nil if this is a single top-level scalar value (which I believe is not strictly allowed in the json spec, but pg allows it). The key/val returned by the function are used to store the result, but do not affect the path values passed to any other function call. If discard is not specified, then the function is also called for completed containers (in which case val will be a table). If pg_numeric is not true, then numeric values are converted to Lua numbers, otherwise they remain as Datum values of numeric type (for which see below).

Substitution of null values happens BEFORE the mapping function is called; if that's not what you want, then do the substitution yourself before returning the result. (If the mapping function itself returns a Lua nil, then the entry will be omitted from the result.)

As a convenience shorthand, these work:

  d(nvl)   -> d{null = nvl}
  d(func)  -> d{map = func}
  d()      -> d{}

Jsonb supports an inverse mapping operation for construction of json values from lua data:

  pgtype.jsonb(value,
               { map = function(val) ... return val end,
                 null = (any value, default nil),
                 empty_object = (boolean, default false)
                 array_thresh = (integer, default 1000)
                 array_frac = (integer, default 1000)
               }

"value" can be composed of any combination of (where "collection" means a value which is either a table or possesses a __pairs metamethod):

• Empty collections, which will convert to empty json arrays unless empty_object=true in which case they become empty objects

• Collections with only integer keys >= 1, which will convert to json arrays (with lua index 1 becoming json index 0) unless either more than array_thresh initial null values would have to be inserted, or the total size of the array would be more than array_frac times the number of table keys.

• Collections with keys which can be stringified: strings or numbers, or tables or userdata with __tostring methods, will convert to json objects.

• Values which compare raw-equal to the "null" parameter are converted to json nulls

• Values of type nil, boolean, number, string are converted to corresponding json values

• Datum values of type pgtype.numeric convert to json numbers

• Datum values of other types convert to json in the same way as they do in SQL; in particular, jsonb and json values are included directly, and values with casts to jsonb have those casts respected

• Values of other types that possess a __tostring metamethod are converted to strings

Unlike the other mapping functions, the map function for this operation is called only for values (including collections), not keys, and is not passed any path information.

Range types support the following pseudo-columns (immutable):

  r.lower
  r.upper
  r.lower_inc
  r.upper_inc
  r.lower_inf
  r.upper_inf
  r.isempty

Function arguments are converted to simple Lua values in the case of:

• integers, floats -- passed as Lua numbers

• text, varchar, char, json (not jsonb), xml, cstring, name -- all passed as strings (with the padding preserved in the case of char(n))

• enums -- passed as the text label

• bytea -- passed as a string without any escaping or conversion

• boolean -- passed as boolean

• nulls of any type -- passed as nil

• refcursor values are converted to or from SPI cursor objects (whether or not they correspond to open portals)

• domains over any of the above are treated as the base types

Other values are kept as datum objects.

The trusted language is implemented differently - rather than removing functions and packages, the trusted language evaluates all user-supplied code (everything but the init strings) in a separate environment table which contains only whitelisted content. A mini version of the package library is installed in the sandbox environment, allowing package.preload and package.searchers to work (the user can install their own function into package.searchers to load modules from database queries if they so wish).

See the main documentation for details on making additional modules available to the trusted language.

A set-returning function isn't considered to end until it either returns or throws an error; yielding with no results is considered the same as yielding with explicit nils. (Old version killed the thread in that scenario.) A set-returning function that returns on the first call with no result is treated as returning 0 rows, but if the first call returns values, those are treated as the (only) result row.

Trigger functions no longer have a global "trigger" object, but rather are compiled with the following definition:

  function(trigger,old,new,...) --[[ body here ]] end

"trigger" is now a userdata, not a table, but can be indexed as before. Trigger functions may assign a row to trigger.row, or modify fields of trigger.row or trigger.new, or may return a row or table; if they do none of these and return nothing, they're treated as returning trigger.row unchanged. Note that returning nil or assigning row=nil to suppress the triggered operation is in general a bad idea; if you need to prevent an action, then throw an error instead.

An interface to pg's "numeric" type (decimal arithmetic) is provided; see the main documentation for details.

Polymorphic and variadic functions are fully supported, including VARIADIC "any". VARIADIC of non-"any" type is passed as an array as usual.

Interpreters are shut down on backend exit, meaning that finalizers will be run for all objects at this time (including user-defined ones). Currently, SPI functionality is disabled during exit.

AUTHOR

Andrew Gierth, aka RhodiumToad

The author acknowledges the work of Luis Carvalho and other contributors to the original pllua project (of which this is a ground-up redesign).

License: MIT license