Stuart SQL

A native Lua implementation of Spark SQL.

This is a companion module for Stuart, the Spark runtime for embedding and edge computing.

Getting Started

Installing

$ luarocks install stuart-sql

If you are using Lua < 5.3 and also want to work with Parquet files, then:

$ luarocks install parquet

Using

Reading a Parquet file into an RDD

local lodash = require 'lodash'
local SparkSession = require 'stuart-sql.SparkSession'

local session = SparkSession.builder():getOrCreate()
local centroidsDataFrame = session.read:parquet('my-kmeans-model/part3.parquet')
local centroids = centroidsDataFrame:rdd():collect()
lodash.print('centroids=', centroids)
centroids= {{0, {3,4,5}}} -- rowid, values

Testing

Testing Locally

$ busted -v
●●
3 successes / 0 failures / 0 errors / 0 pending : 0.063098 seconds

Testing with a Specific Lua Version

$ docker build -f Test-Lua52.Dockerfile -t test .
$ docker run -it test busted -v
●●
3 successes / 0 failures / 0 errors / 0 pending : 0.063098 seconds