SQL #158

lbajolet · 2016-10-31T18:03:51Z

Big PR here, this should make the transition from a MongoDB model to the famed SQL model.
The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work.
Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here.

Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)

This change is

privat · 2016-10-31T18:51:13Z

plase update tests/test_*.nit

privat · 2016-10-31T18:53:24Z

.gitignore

@@ -3,7 +3,6 @@ bin/
 *.orig
 db_loader_serial.nit
 out/
-db/


we still need to ignore the directory I think

privat · 2016-10-31T18:55:27Z

.gitignore

@@ -3,7 +3,6 @@ bin/
 *.orig
 db_loader_serial.nit
 out/
-db/


we still need to ignore the db generated files

privat · 2016-10-31T18:56:14Z

the PR is huge. If you do not plan to split it, could you explain what you did exactly?

Morriar · 2016-10-31T18:57:46Z

45 files changed, 3 commits... seriously?

![1dan1m](https://cloud.githubusercontent.com/assets/1444825/19865386/650afce4-9f72-11e6-8e49-5152e7f5f2ed.jpg) Big PR here, this should make the transition from a MongoDB model to the famed SQL model. The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work. Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here. Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/morriar/missions/158)  Pull-Request: #158 Reviewed-by: Jean Privat <[email protected]> Reviewed-by: Alexandre Terrasa <[email protected]>

privat · 2016-11-01T18:04:10Z

TODO:

move db/ to something else.
add libsqlite3-dev into the Dockerfile

lbajolet · 2016-11-01T18:09:23Z

Done

![1dan1m](https://cloud.githubusercontent.com/assets/1444825/19865386/650afce4-9f72-11e6-8e49-5152e7f5f2ed.jpg) Big PR here, this should make the transition from a MongoDB model to the famed SQL model. The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work. Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here. Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/morriar/missions/158)  Pull-Request: #158 Reviewed-by: Jean Privat <[email protected]> Reviewed-by: Alexandre Terrasa <[email protected]>

ventilooo · 2016-11-01T18:22:11Z

Dockerfile

@@ -1,7 +1,7 @@
 FROM nitlang/nit

 # Needed for nitcorn and to build mongo-c-driver
-RUN apt-get update && apt-get install -y libevent-dev libssl-dev libsasl2-dev libcurl4-openssl-dev file
+RUN apt-get update && apt-get install -y libevent-dev libssl-dev libsasl2-dev libcurl4-openssl-dev file libsqlite3-dev

 # Install mongo-c-driver manually since it is not available in Debian/jessie
 RUN curl -L https://github.com/mongodb/mongo-c-driver/releases/download/1.4.0/mongo-c-driver-1.4.0.tar.gz -o mongo-c-driver-1.4.0.tar.gz \


Mongo-c-driver is no longer required if sqlite is used .

It should not be, but there seems to be some import somewhere at the moment which prevents us from compiling when mongoc is unavailable, so we left it for now

![1dan1m](https://cloud.githubusercontent.com/assets/1444825/19865386/650afce4-9f72-11e6-8e49-5152e7f5f2ed.jpg) Big PR here, this should make the transition from a MongoDB model to the famed SQL model. The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work. Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here. Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/morriar/missions/158)  Pull-Request: #158 Reviewed-by: Jean Privat <[email protected]> Reviewed-by: Alexandre Terrasa <[email protected]> Reviewed-by: Istvan SZALAÏ <[email protected]>

lbajolet · 2016-11-01T19:19:44Z

OK, for the explanation of what is happening here, the new model is articulated around a few things:

DBContext: Basically the equivalent of a Context in Hibernate or Entity terminology, a glorified-ish Connection to a Database with eventual caches and such. Our DBContext has no caches at the moment, but might be added in the future to help with performance.
Two kinds of Entity are defined, UniqueEntity which are basically entities with a single Primary Key: Mission, Player, Track, etc., and BridgeEntity, which are tables with more than one primary key: MissionStatus, TrackStatus, StarStatus, etc.
All entities have a commit method which persists the current state of an object to the Database, which based on the current state of the object (are its primary keys set or not), will be either inserted or updated to the database. For now, entities do not have a delete method, but this could be added later if needs be. Therefore, each Entity needs to have an insert and an update redefinition for it to work properly.
Objects are a bit more lightweight with the actual model as the linked entities are not eagerly fetched in most cases (the exception being the TrackStatus, which triggers the collection/creation of MissionStatus objects, which themselves trigger the collection/creation of StarStatus objects). But the usage from the API POV should remain more or less the same.
The deserialization of objects from the database is done within the Statement and StatementRow objects, provided they contain the right data for the creation of the corresponding objects in the model. If this is not the case, they will print (log when available) an error message and return null or an empty Array of the proper type.
The DBContext provides the general request methods for each object type in the model with a slightly different naming convention as before. Ex: To get a mission by its id, using the old API this would have looked like config.missions.by_id(id), now it will look like ctx.mission_by_id(id). Fundamentally, this does not change much the use of the API.
The API side of the application should be more or less equal to the old one, at the exception that most of the uses of id have been replaced by slug as slug is now the user-friendly version of an id for use in URLs. We made the decision to have the ids as Int in the database to put less burden when creating Foreign Keys, as Ints are less costly than Strings. This is also speedier when joining or selecting data on the ID. Note that all the Entities do not necessarily have a slug field, only those who are used in URLs: Player, Mission, Track, Achievement.
Concerning Achivements, the model changed a bit too, as the old model used Achievement as both the definition and the unlock of an Achievement, the operation is now split in two, Achievements are their own table, and the unlocks are too.
The frontend part changed a bit due to the re-organization of the server-side data, but all in all, is pretty much the same as before. This is actually the part I'm less comfortable about since I might have overlooked some things, and this has changed a lot since I've started working on the PR. And I suck at JS.

This should sum-up the differences, if there are any questions left, feel free to ask.

![1dan1m](https://cloud.githubusercontent.com/assets/1444825/19865386/650afce4-9f72-11e6-8e49-5152e7f5f2ed.jpg) Big PR here, this should make the transition from a MongoDB model to the famed SQL model. The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work. Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here. Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/morriar/missions/158)  Pull-Request: #158 Reviewed-by: Jean Privat <[email protected]> Reviewed-by: Alexandre Terrasa <[email protected]> Reviewed-by: Istvan SZALAÏ <[email protected]>

privat · 2016-11-01T20:10:41Z

src/api/api_auth_shibuqam.nit

@@ -127,7 +126,7 @@ class ShUser
 	serialize

 	# The *code permanent* (or the uid for non student)
-	var id: String
+	var slug: String


privat · 2016-11-01T20:11:46Z

src/api/api_auth.nit

@@ -115,10 +115,15 @@ abstract class AuthLogin
 	# Register a new player and add a first-login achievement
 	#
 	# Helper method to use when a new account is created.
-	fun register_new_player(player: Player)
+	fun register_new_player(ctx: DBContext, player: Player)


use the ctx of the player

privat · 2016-11-01T20:12:54Z

src/api/api_base.nit

@@ -30,6 +30,10 @@ class APIRouter
 	var config: AppConfig
 end

+redef class HttpRequest
+	var ctx: DBContext is lazy do return new DBContext


privat · 2016-11-01T20:29:48Z

src/model/submissions.nit


 		var old_score = self.old_score
 		if old_score != null then
 			res += " (was {old_score})"
 		end
 		return res
 	end
+
+	redef fun insert do


factorize the insert/update/whatever with the "template method" design pattern™ by extracting the query part

privat · 2016-11-01T20:32:52Z

tests/test_nitc.nit

-# Create a dummy user
-var player = new Player("John", "Doe")
-config.players.save player
+with ctx = new DBContext do


a with? Ohhhhhh

privat · 2016-11-01T20:33:57Z

www/directives/missions/mission.html

@@ -38,6 +38,6 @@ <h1 class='pull-right'>
 				</div>
 		</div>
 		<mission-submit mission='missionCtrl.mission' mission-status='missionCtrl.missionStatus'
-			ng-if='missionCtrl.playerId' />
+			ng-if='missionCtrl.playerId && missionCtrl.missionStatus.status != "locked"' />


We'll add some debug/god mode someday for this.

privat · 2016-11-01T20:34:18Z

www/directives/notification/PlayerNotification.html

@@ -4,9 +4,6 @@
 	</div>
 	<div class='card-body'>
 		<h5>{{notification.object}}</h5>
-		<p class='text-muted'>
-			{{notification.timestamp * 1000 | date: 'dd, MMM, yyyy HH:MM:ss a'}}
-		</p>


privat · 2016-11-01T20:36:27Z

src/model/missions.nit

+	var solve_reward: Int = 0 is writable, serialize_as("reward")
+	var path: String = "" is writable
+	# Template for the source code
+	var template: nullable String = null is writable


add template in the DB

privat · 2016-11-01T20:37:38Z

mission tree is broken

![1dan1m](https://cloud.githubusercontent.com/assets/1444825/19865386/650afce4-9f72-11e6-8e49-5152e7f5f2ed.jpg) Big PR here, this should make the transition from a MongoDB model to the famed SQL model. The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work. Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here. Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/morriar/missions/158)  Pull-Request: #158 Reviewed-by: Jean Privat <[email protected]> Reviewed-by: Alexandre Terrasa <[email protected]> Reviewed-by: Istvan SZALAÏ <[email protected]>

privat · 2016-11-03T20:01:35Z

+1

Morriar · 2016-11-07T14:53:24Z

@R4PaSs: I would definitely expect some benchmarks explaining why we have to endure this 1500 lines addon :)

Morriar · 2016-11-07T15:04:10Z

tests/test_nitc.nit

@@ -13,52 +13,56 @@ var opts = new AppOptions.from_args(args)
 var config = new AppConfig.from_options(opts)

 # clean bd
-config.db.drop
+#config.db.drop


Just delete it(c)

Morriar · 2016-11-07T15:04:17Z

tests/test_nitc.nit


 # Load nit
-config.load_track("tracks/nit")
+#config.load_track("tracks/nit")


Just delete it(c)

![1dan1m](https://cloud.githubusercontent.com/assets/1444825/19865386/650afce4-9f72-11e6-8e49-5152e7f5f2ed.jpg) Big PR here, this should make the transition from a MongoDB model to the famed SQL model. The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work. Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here. Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/morriar/missions/158)  Pull-Request: #158 Reviewed-by: Jean Privat <[email protected]> Reviewed-by: Alexandre Terrasa <[email protected]> Reviewed-by: Istvan SZALAÏ <[email protected]>

Signed-off-by: Alexandre Terrasa <[email protected]>

Signed-off-by: Philippe Pepos Petitclerc <[email protected]>

Signed-off-by: Lucas Bajolet <[email protected]>

![1dan1m](https://cloud.githubusercontent.com/assets/1444825/19865386/650afce4-9f72-11e6-8e49-5152e7f5f2ed.jpg) Big PR here, this should make the transition from a MongoDB model to the famed SQL model. The code base being large, there might be some parts of the application that might break, but from the last checks I did, most of the app should work. Some parts, like achievements may need a bit of an overhaul to make it production-ready, but the base is here. Note: I know the login redirect is semi-broken, I'll check to fix it, in the meantime, you are more than encouraged to take a look at the code :)  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/morriar/missions/158)  Pull-Request: #158 Reviewed-by: Jean Privat <[email protected]> Reviewed-by: Alexandre Terrasa <[email protected]> Reviewed-by: Istvan SZALAÏ <[email protected]>

privat reviewed Oct 31, 2016

View reviewed changes

lbajolet force-pushed the sql branch from 94e94ce to 5cd083b Compare November 1, 2016 18:09

ventilooo reviewed Nov 1, 2016

View reviewed changes

lbajolet force-pushed the sql branch from 5cd083b to a25867c Compare November 1, 2016 18:59

lbajolet force-pushed the sql branch from a25867c to d3bc9a9 Compare November 1, 2016 19:20

lbajolet force-pushed the sql branch from d3bc9a9 to 571af3a Compare November 1, 2016 19:28

privat reviewed Nov 1, 2016

View reviewed changes

lbajolet force-pushed the sql branch from 571af3a to 0d78da0 Compare November 2, 2016 20:22

lbajolet force-pushed the sql branch from 0d78da0 to f3b125d Compare November 3, 2016 18:38

Morriar reviewed Nov 7, 2016

View reviewed changes

lbajolet force-pushed the sql branch from f3b125d to d56e58d Compare November 30, 2016 17:07

lbajolet force-pushed the sql branch from d56e58d to 0409e34 Compare November 30, 2016 17:13

Morriar and others added 9 commits December 6, 2016 14:32

src: update config to new nit Config class

29f0249

Signed-off-by: Alexandre Terrasa <[email protected]>

Intro SQL database schema creation

0dfee71

Signed-off-by: Philippe Pepos Petitclerc <[email protected]>

src/model: Mongo-to-SQL Transition

3edadce

Signed-off-by: Lucas Bajolet <[email protected]>

src: Update loaders to use SQL instead of Mongo

dd7f69d

Signed-off-by: Lucas Bajolet <[email protected]>

src/api: Replaced all id uses by slug for better URLs

d145f20

Signed-off-by: Lucas Bajolet <[email protected]>

www: Frontend updates to use the SQL model

93d8f90

Signed-off-by: Lucas Bajolet <[email protected]>

tests: Fix config utilisation for SQL compatibility

dd566b2

Signed-off-by: Lucas Bajolet <[email protected]>

Added missing libsqlite3-dev and sqlite3 packages to Dockerfile

3d2b8ba

Signed-off-by: Lucas Bajolet <[email protected]>

src/model: Added caching for Players and Missions

39d1390

Signed-off-by: Lucas Bajolet <[email protected]>

lbajolet force-pushed the sql branch 2 times, most recently from 05115cf to 39d1390 Compare December 6, 2016 21:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL #158

SQL #158

lbajolet commented Oct 31, 2016 •

edited

Loading

privat commented Oct 31, 2016

privat Oct 31, 2016

privat Oct 31, 2016

privat commented Oct 31, 2016

Morriar commented Oct 31, 2016

privat commented Nov 1, 2016

lbajolet commented Nov 1, 2016

ventilooo Nov 1, 2016 •

edited

Loading

lbajolet Nov 1, 2016

lbajolet commented Nov 1, 2016

privat Nov 1, 2016

privat Nov 1, 2016

privat Nov 1, 2016

privat Nov 1, 2016

privat Nov 1, 2016

privat Nov 1, 2016

lbajolet Nov 3, 2016

privat Nov 1, 2016

privat Nov 1, 2016

privat commented Nov 1, 2016

privat commented Nov 3, 2016

Morriar commented Nov 7, 2016

Morriar Nov 7, 2016

Morriar Nov 7, 2016

SQL #158

Are you sure you want to change the base?

SQL #158

Conversation

lbajolet commented Oct 31, 2016 • edited Loading

privat commented Oct 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

privat commented Oct 31, 2016

Morriar commented Oct 31, 2016

privat commented Nov 1, 2016

lbajolet commented Nov 1, 2016

ventilooo Nov 1, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbajolet commented Nov 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

privat commented Nov 1, 2016

privat commented Nov 3, 2016

Morriar commented Nov 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbajolet commented Oct 31, 2016 •

edited

Loading

ventilooo Nov 1, 2016 •

edited

Loading