-
Notifications
You must be signed in to change notification settings - Fork 12
Idomaar evaluation process
Idomaar allows evaluating recommendation tasks via a three-phase process:
Data splitting creates the data required to run the experiment. An input dataset is processed by the evaluator and split into two main sets: batch training data and on-line streaming data.
Batch training data is the set of entities and relations used to bootstrap the computing environment. It basically consists of all the data available to the computing environment for the initial training of the recommendation algorithm.
On-line streaming data consists of a set of additional entities and relations, together with a list of recommendation requests to be tested. In fact, entities and relations can be provided later to the computing environment (as they become available, e.g., the injection of new items). Furthermore, a set of recommendation requests is created. Each request consists of the message to be sent to the computing environment (to request a recommendation) and the expected output (i.e., the groundtruth).
At the end of this phase, the orchestrator starts the computing enviroment and waits for its acknoledge (though the control channel, 'ready' message) notifying that the enviroment has been set up.
The data streaming consists of two main sequential tasks: computing environment bootstrapping and on-line evaluation.
In the fist task - computing environment bootstrapping - the orchestrator notifies the computing environment (through the control channel, 'TRAIN' message) that bootstrapping phase is starting. Therefore, the orchestrator reads the batch training data and sends it to the computing environment (through the data streaming channel, Kafka 'data' topic). The computing environment notifies the orchestrator ('OK' message) as soon as it is ready to provide recommendation, the computing environment can be aware that data is finished as soon as it receive from the queue the 'EOF' message. At this point the bootstrapping/train step is finished.
The task on-line evaluation starts immediately after bootstrapping is completed. The orchestrator notifies the computing environment (though the control channel, 'TEST' message) to be ready to serve recommendation requests. The orchestrator reads the additional entities and relations and streams them (through the data streaming channel, kafka 'data' topic) to the computing environment that has the opportunity to incremental update (if it has the right capabilities) the recommendation algorithms. In accordance to the recommendation requests created by the splitting module (during the phase data splitting) and the streaming strategy, the orchestrator sends the recommendation requests to the computing environment (through the control channel, message 'RECOMMEND'); the recommendation requests are served by the recommendation algorithm whose output is sent back to the orchestrator (through the control channel) and stored in accordance with the format described in Data format.
The recommendation output contains all the information required later by the evaluator to compute both quality and performance metrics.
The computing environment terminates when it reads the end of stream message (through the data streaming channel, 'EOF' message). At this point, the computing environment notifies the orchestrator (through the control channel, 'OK' message) that it has completed processing all received data.
At the end of this phase, the computing environment can be shut down.
The last phase consists in evaluating the output of the computing environment, both in terms of quality metrics (e.g., precision/recall, RMSE, etc.) and performance metrics (e.g., response time). This phase processes the output of the computing environment stored by the orchestrator and generates a set experiment results. This tasks is performed by the evaluator.