Skip to content

Latest commit

 

History

History
87 lines (66 loc) · 3.87 KB

MIGRATING.md

File metadata and controls

87 lines (66 loc) · 3.87 KB
Raw

Migrating from filetracker 1.x to filetracker 2.x

The storage format of the filetracker server is very different in the new server version, so non-trivial migration is required if you want to preserve your users' files (submitted sources, problem packages, etc.).

After migration is complete, you may expect total filetracker storage size to decrease by ~70% for a typical OIOIOI instance.

There are two migration options, but for both of them, you first need to make sure OIOIOI uses remote_storage_factory as FILETRACKER_CLIENT_FACTORY and FILETRACKER_CACHE_ROOT is configured in your settings.py. If you have a recent config version (23+), this is the default behaviour.

Offline migration

This is the simpler option for cases when you don't care about downtime.

On the machine that hosts your filetracker server (this is the same as OIOIOI host in default config), let ft_root be the root directory of your filetracker storage. You should see all of your files under ft_root/files.

New filetracker server depends on Berkeley DB, so you should have it installed (on Ubuntu or Debian install libdb-dev, on Arch it's db).

Activate your venv, and upgrade the filetracker package to version 2.1 (or newer). Now your $PATH should contain filetracker-migrate script (try running $ filetracker-migrate -h).

Now, run the filetracker server in the usual directory with filetracker-server -L ~/filetracker-migration.log -d ft_root -p 9998 -D, port number is arbitrary, ft_root should be substituted for your actual path). OIOIOI and other SIO2 services shouldn't be running, otherwise you will get lots of SE verdicts and HTTP 500 errors.

Navigate to ft_root and execute the following command: filetracker-migrate ./files http://127.0.0.1:9998. Enjoy the view.

CAUTION: if you have a lot of files (>100 GiB), this command may take a few days to complete, so doing this in a usual SSH session may not be the best idea. Consider using tmux or screen. Average migration speed for our servers was 10 GiB/h.

After the command above is completed you can safely remove the original ./files directory (it's not used by the new server). Your previous server config should still work, so you can get going immediately.

Online migration

If you can't afford significant downtime, it's possible to perform the migration online, but it's more involved.

You should have read and become comfortable with the section above, this section only describes the differences.

The trick is based on the new --fallback-url parameter for the filetracker server. After upgrading the filetracker server, you should configure two servers: the main server which uses the new filetracker code, and the fallback server which is plain lighttpd.

Before starting the new server and the rest of the SIO2 infrastructure, start a lighttpd server with a simple configuration for serving static files from ft_root (port and log path are arbitrary):

server.tag = "filetracker-old"
server.document-root = "/path/to/ft_root"
server.port = 59999
server.bind = "0.0.0.0"
server.modules = ( "mod_accesslog", "mod_status" )

status.status-url = "/status"
accesslog.filename = "~/filetracker-old.log"
fastcgi.debug = 1

mimetype.assign = (
    "" => "application/octet-stream"
)

Then, start the rest of the SIO2 services normally, but add an additional argument to filetracker server: --fallback-url http://127.0.0.1:59999. The server will now forward GET and HEAD requests for files it doesn't have to the fallback server. PUT and DELETE are handled normally by the new server.

Start the migration process the same way as described in the section above: filetracker-migrate ./files http://127.0.0.1:9999, if your main filetracker server runs on port 9999 (the default).

After migration is completed, you may restart the filetracker server without --fallback-url, stop the lighttpd server, and remove ft_root/files directory.