Update setup scripts for marquee to work on Debian 10

[foolip]

marquee is still running on the same Debian 9 instance, but these
scripts have been tested on a new Debian 10 instance on DigitalOcean.

[foolip]

@domenic I thought this might help with #24, but it won't.

Much of this would be easy to do with Docker, and converting it wouldn't even be all that hard. The hard part is how we'd manage the persistent storage and deploy new Docker images to DigitialOcean. It's easier to keep pushing this pile of scripts forward, but not ideal.

[domenic]

Seems reasonable, although it might be nice if we could upgrade everything to Debian 10 at once. I'll give this the approve and let you handle merging and deploying as necessary.

Much of this would be easy to do with Docker, and converting it wouldn't even be all that hard. The hard part is how we'd manage the persistent storage and deploy new Docker images to DigitialOcean. It's easier to keep pushing this pile of scripts forward, but not ideal.

I think we'd keep deploying the persistent storage to its current location, then just run the web server via docker with the storage shared with the docker container as bind mounts. The docker container would then run the web server which serves files from the bind mounts onto port 80 and 443, and we would expose those ports out of Docker onto the host machine.

However, Googling around for this seems like people are using something called Docker Compose for this, which I don't quite understand the need for. See e.g. this tutorial or this "one click droplet" configuration. It feels like it removes some of the simplicity of doing everything inside a Docker container by involving multiple containers... But everyone's doing it, so there must be a reason.

Additionally I don't know how to handle the downtime that would be involved during deploys, as the container needs to get re-built and re-started, which could take a minute or two.

I kind of enjoy this stuff though so maybe I should buy a book on sysadmining with Docker and figure it all out...

[foolip]

Every time I've looked into this (where to put disk and database) I've come across solutions just like that, multiple containers and some orchestration. Kubernetes comes up a lot, too.

We could probably define all of whatwg.org as a group of containers, having bind mounts for the data. But then you have to run that somewhere. It seems like avoiding downtime is more often done by having two instances always.

A lot of complexity. If you'd like to read that book, I'd be happy to see the results though :)

[sideshowbarker]

However, Googling around for this seems like people are using something called Docker Compose for this, which I don't quite understand the need for. See e.g. this tutorial or this "one click droplet" configuration. It feels like it removes some of the simplicity of doing everything inside a Docker container by involving multiple containers... But everyone's doing it, so there must be a reason.

I also don’t clearly understand the need for Docker Compose, but from my limited experience in adding and maintaining a Dockerfile for the HTML checker — and in documenting it at https://validator.github.io/validator/#dockerhub and discussing it with people who actually use Docker for the HTML checker — I get the impression that Docker Compose is what’s normally used in practice by more real-world production deployments of stuff using Docker.

Some of the reasons seem to be that Docker Compose allows you to run without needing to specify additional arguments on the command line — instead you just put it all into .yml file — and also instead of needing do execute both docker build and docker run, you instead can just do docker-compose up --build.

At Stack Overflow, I also found what seems a pretty good description, in the following answer:
https://stackoverflow.com/a/51058001/441757

Dockerfile describes how to build a Docker image - it's essentially Docker's version of Makefile or pom.xml or build.gradle

docker-compose.yml describes how the image is to be deployed, that is all the things necessry to create both the container and the environment it needs to run (like networks or volumes). If several containers need to work together the description of how to deploy them together, in which order, how to link them an so on can be put into a single docker-compose.yml

[domenic]

@foolip would you be willing to carry this over the finish line, and/or help upgrade our server?

[domenic]

Note that the cerbot scripts are failing now because everything related to apt seems to be failing. So this might be kinda urgent :(

[foolip]

@foolip would you be willing to carry this over the finish line, and/or help upgrade our server?

By now we should upgrade to Debian 12 instead, and every time I think of this problem I think I'd like to migrate to something like Cloudflare instead.

But I can commit to either upgrading in DigitalOcean or giving up and making it clear I'm not working on it, during January.

Note that the cerbot scripts are failing now because everything related to apt seems to be failing. So this might be kinda urgent :(

Is this from some logs?

[domenic]

Is this from some logs?

This is when I followed the process in https://github.com/whatwg/misc-server/blob/main/debian/README.md in order to re-deploy after b19abc3.

[foolip]

January and half of February has passed. I did a quick experiment and ruled out the "just use Git" approach I was hoping for in #107 (comment).

I won't be able to make time for this, but I think that upgrading the VMs and keeping most things unchanged is the only quick path forward here.