Setting up a development Environment

We support Mac OS X, Windows and Linux. You need to be comfortable with command-line tools.

Starting

1. Install dependencies:

• Docker (on Ubuntu, you'll need to pip install docker-compose as well)
• Git

1. git clone https://github.com/overview/overview-server.git
2. cd overview-server
3. ./dev

That last command will take a long time the first time you run it -- maybe hours -- as it downloads and compiles components. When the text stops, it's ready.

You should also set up integration tests. Install Node LTS so you can run those. (Test that node --version is greater than 8 and that npm --version does not produce an error.)

Logging in

You can set the OVERVIEW_MULTI_USER flag to false (see configuration) to bypass all login and user management. Or you can use the default admin login, [email protected]/[email protected]. Or you can register a new user. Overview won't actually email you, but you can read the email it would have sent on the console where you ran ./dev.

Getting plugins running

A fresh development install does not have plugins configured (Word cloud, Entities, etc.) To get these to appear in the Add View menu, you need to do two things:

• Add the plugins from the Admin menu, giving a URL and description for each. For a normal development environment, the url will be http://localhost:[port] where port matches the numbers in plugins.yml
• Then start the plugins running with docker-compose -f plugins.yml up

Running individual servers

Overview is a hive of servers. Here they are:

• PostgreSQL: stores our data about documents. (Actual documents are stored elsewhere.)
• Redis: caches stuff.
• overview-server: a Play web server that listens at http://localhost:9000, with source code in web/. If Play is running while you edit the source code (in web/), Play will automatically recompile the next time you load a page.
• worker: a plain Java program, with source code and tests in worker/. It won't restart when you edit code -- at least, not by default.
• plugins: Each plugin is in fact a tiny web server. To start them all, docker-compose -f plugins.yml up

If you're going to fiddle, you should probably run each command in its own console.

Incantations

• ./sbt web/test: Compiles and unit-tests the code in web/.
• ./sbt '; project web; ~testQuick': Useful when coding in web/. Recompiles and re-tests, every time a file changes. It's equally great for the worker and common projects.
  • Beware: if you run ~testQuick while dev is also hosting a web server, you can hit a race: dev will recompile when you refresh the web page, and ~testQuick will recompile when you save a file. They'll step on each other's toes and one will crash. Train yourself to wait for ./sbt ~testQuick to finish its work before you refresh the page.
• ./sbt db-evolution-applier/run: runs db-evolution-applier/src/main/resources/migration/*.sql on the development database. For the test database, use ./sbt test-db-evolution-applier/run.
• ./auto/test-coffee.sh: continually re-runs JavaScript unit tests in web/test/js/.
• ./auto/test-coffee-once.sh: just runs the tests once
• ./build overview-server.zip: creates a build, overview-server.zip.
• ./auto/psql.sh and ./auto/psql.sh overview-test: log in to the overview-dev and overview-test databases, respectively. PostgreSQL SQL Documentation

All these commands will transparently run inside a Docker container. That means you won't need to install any dependencies.

Where's my data?

Everything goes in Docker volumes. You can see those volumes in docker-compose.yml. Here are the Docker volumes we use:

• Document Sets and document metadata go in database (Postgres)
• File contents (raw uploaded files and ready-to-view PDFs) go in blob-storage (flat files; in production it's S3)
• Search indexes (one per document set) go in search (flat files)
• Build stuff goes in homedir

To make Overview stop, hit Ctrl+C. If you want everything destroyed, add the scary option -v to the non-scary command, docker-compose down. The -v option will delete all Overview data. You've been warned.

Hacking Docker services

• docker ps shows running Docker containers. (A "container" is basically a Linux process. Even if you're on Windows or Mac.)
• docker ps -a shows running and "stopped" Docker containers.
• auto/docker-exec.sh bash will run bash inside the already-running "dev" container (which you run by calling docker-compose up)
• auto/docker-run.sh bash will run bash inside a fresh "dev" container. This will have the same persisted data (built files, search indexes, etc) as the running "dev" container, but it won't be able to "see" the running "dev" container.

Hacking the web server

To run just the web server, run ./sbt run. It will restart every time you change a file in app/, test/, or common/src/main and then reload.

To run unit tests every time you change a file in app/ or test/, run ./sbt '; project web; ~testQuick'.

Beware: if you run both commands simultaneously, sbt will destroy its own files, leading to errors. Train yourself to wait for ~testQuick to finish its work before you refresh the page.

Hacking the worker

To run just the worker, run ./sbt worker/re-start. It will restart every time you change a file in worker/src/main or common/src/main. (You'll probably want to have docker-compose up -d database redis running. You might not want the web server running.)

To run unit tests every time you change a file in worker/src or common/src/main, run ./sbt '; project worker; ~testQuick'.

Beware: if you run worker/re-start and ; project worker; ~testQuick simultaneously, sbt will destroy its own files, leading to errors. Run one and then the other. Or don't auto-restart the worker; just run ./sbt worker/run instead.

Hacking common code

To run unit tests every time you change a file in common/src, run ./sbt '; project common; ~testQuick'.

Hacking CoffeeScript/JavaScript

Run auto/test-coffee.sh to auto-test all CoffeeScript/JavaScript code any time it changes. The code is in app/assets/javascripts and the tests are in test/assets/javascripts.

Running/writing integration tests

Run integration-tests/run to test everything. This expects ./dev to be running.