Idomaar evaluation process

Idomaar allows evaluating recommendation tasks via a three-phase process: data splitting, data streaming, result evaluation.

Data splitting

Data splitting creates the data required to run the experiment. An input dataset is processed and split into two main sets: batch training data and on-line streaming data.

Batch training data is the set of entities and relations used to bootstrap the computing environment. It basically consists of all the data available to the computing environment for the initial training of the recommendation algorithm.

On-line streaming data consists of a set of additional entities and relations, together with a list of recommendation requests to be tested. In fact, entities and relations can be provided later to the computing environment (as they become available, e.g., the injection of new items). Furthermore, a set of recommendation requests is created. Each request consists of the message to be sent to the computing environment (to request a recommendation) and the expected output (i.e., the groundtruth).

Once this phase completes, the orchestrator starts the computing enviroment and waits for its acknoledge (though the control channel) once the ennviroment has been set up.

Data streaming

The data streaming consists of two main sequential tasks: computing environment bootstrapping and on-line evaluation.

In the fist task - computing environment bootstrapping- the orchestrator reads the batch training data and sends it to the computing environment (through the data streaming channel). At the end, the orchestrator notifies (through the control channel) the computing environment that the bootstrapping data are finished. This task completes when the computing environment has bootstrapped the recommendation algorithms and it is ready to serve recommendation requests; at the end of this process, the computing environment sends an acknoledge message to the orchestrator.

During the task on-line evaluation, the orchestrator reads the additional entities and relations and streams them (through the data streaming channel) to the computing environment that has the opportunity to incremental update (if it has the right capabilities) the recommendation algorithms. In accordance to the recommendation requests created by the splitting module (during the phase data splitting) and the streaming strategy, the orchestrator sends the recommendation requests to the computing environment (through the control channel); the recommendation requests are served by the recommendation algorithm whose output is sent back to the orchestrator (through the control channel) and stored in accordance with the format described in Data format.

The recommendation output contains all the information required later by the evaluator to compute both quality and performance metrics.

When all on-line streaming data have been streamed to the computing environment, the orchestrator notifies (through the control channel) the computing environment that the experiment has completed and waits for its acknoledge (through the control channel) to conclude the data streaming task.

At the end of this phase, the computing environment can be shut down.

Result evaluation

The last phase consists in evaluating the output of the computing environment, both in terms of quality metrics (e.g., precision/recall, RMSE, etc.) and performance metrics (e.g., response time). This phase processes the output of the computing environment stored by the orchestrator and generate a set experiment results.