This section applies to firehol, iprange and netdata.
Note
: If there is ever a need to release a new FireHOL 1.x, see
the notes in doc/release-firehol-v1.md instead of this procedure.
Ensure local repository hooks are installed, they will take care of all the checking and much of the work.
As well as commit rights to the repository, you need a GPG key to sign and verify the git tag. To automate release signing and signature upload, you must set up an Authentication token.
Note
: If there is ever a need to release a new FireHOL 2.x, use the custom
steps in doc/release-firehol-v2.md instead of this sub-section.
Update configure.ac to remove the version suffix or set it to -rc.nn,
update the ChangeLog and any .spec.in files.
Note : Pre-release tags should be of the form vx.y.z-pre.nn or vx.y.z-rc.nn in order to comply with http://semver.org/ without limiting the number of such releases.
Run git commit in the normal way. The hooks will detect a release and
do the following:
- Test build
- Prepare a suitable message
- Ask you to sign a tag
- Update the
configure.acfor continuing development and commit it
If not satisfied, since we have not yet pushed anything, it is safe to delete the local tag, and optionally roll back history:
git tag -d vx.y.z
git log
git reset --hard commit-before-our-release
Once satisfied with the results, push the branch and tag e.g.:
git push origin
git push origin tag va.b.c-rc.d
Travis integration will automatically start the build process and upload the resulting packages as a github release.
Run the sign-github-release script e.g.:
./bin/sign-github-release firehol firehol vx.y.z
This will check the signature in git, grab any tar files, verify their checksums and the compare the contents against git to ensure it all matches.
If everything is OK, detached signatures are created for each tar-file and the results uploaded to the github release. It is now ready to be published from the website.
Announce on the mailing lists (doc/announce.template) and website.
Signed tags are created automatically provided the commit hooks are in place. It can happen that we want to remove a bad tag, however, from local and from GitHub:
git tag -d vx.y.z
git push origin :refs/tags/vx.y.z
Note that this will not delete the tag from any other mirrored repositories; they will get conflicts later on if we re-use the tag.
Tools are moving over to GitHub actions, so this is mostly out of
date. If a project has .github/workflows/publish.yml, the rest
of this section can be ignored.
The FireHOL tools are built, tested and deployed by linking the travis-ci with GitHub. Current status can be seen here, by administrators of the firehol project:
Each repository contains a .travis.yml file which controls
the environment used and build steps taken.
The CI process aims to:
- Check that all git pre-commit hooks have been validated
- Build releasable (i.e. with documentation and configure script) tar-files
- Run any unit tests
If all of these are successful, the generated files are deployed.
After a successful build, the releasable tar-files and their checksums are uploaded:
- to firehol.org, overwriting prior master/stable-nnn branch builds
- to GitHub (only when a tag is built)
Automatic deployment is set
up as part of the .travis.yml.
GitHib/Travis integration for each repository was made using an Authentication token with the "Travis CI command line client" on a development machine:
travis setup releases
Files are SFTP transferred to [email protected]:uploads/DIR, from
where a cron job will move them to the website, once a final complete.txt
file has been created.
The SSH key has been added to each repository, (separately, since each has separate secure environment variables), in encrypted form:
mkdir .travis
cd .travis
cp /from/somewhere/travis_rsa .
travis login -g `cat ~/.firehol-github-oauth`
travis encrypt-file travis_rsa
rm travis_rsa
git add travis_rsa.enc
travis logout
Note that when copying the openssl instruction into .travis.yml,
the -in and -out paths need to have the .travis folder added.
The .travis.yml decrypts the key, starts an ssh-agent and adds the key to it, then deletes the file to prevent later parts of the script gaining access to it, accidentally or otherwise.
These sections explain the extra steps someone with commit privileges to the firehol github repositories needs to take to be able to create releases.
The key you sign with needs to be in your git configuration ($HOME/.git).
If not run:
git config user.signingkey KEYID
or:
git config --global user.signingkey KEYID
If you need to locate the identifier, run:
gpg --list-keys
Any new keys that will be used for signing a project should be concatenated
to that project's packaging/gpg.keys file. This will extract the key
in a suitable format:
gpg --armor --export KEYID
The key used to sign a tag is checked against this file before a build is allowed to proceed.
Signing packages and updating encrypted files both require you to have
a github API authentication token. Store it in ~/.firehol-github-oauth.
Set up a token with repo rights here.
echo "long-hex-string" > ~/.firehol-github-oauth
chmod 600 ~/.firehol-github-oauth
This repository should be exported as the home directory of a user, firehol, who is a member of group firehol on the firehol.org server.
Create the user in the normal way, then as root:
cd /home
rm -rf firehol
git clone https://github.com/firehol/infrastructure.git firehol
chown -R firehol:firehol firehol
find firehol -type d -exec chmod g+s \{\} \;
To update:
sudo -u firehol -i
git pull
The website is built using nanoc version 3. There are newer versions which are not compatible. At some point the site will be upgraded to v4.
Install docs are essentially, this as root:
aptitude install rubygems
gem install nanoc --version 3.8.0
Then:
mkdir -p /home/web/firehol
mkdir -p /home/web/firehol/download
mkdir -p /home/web/firehol/static
mkdir -p /home/web/firehol/tmp
mkdir -p /home/web/firehol/webalizer
The sites must be manually installed in nginx the first time, e.g.:
cd /etc/nginx/sites-enabled
ln -s /home/web/firehol/master.conf 10-firehol-master
ln -s /home/web/firehol/master.conf 15-firehol-test
The master site must have a lower number than all the others so that it is used by default.
SSL is provided by Let's Encrypt
Installation:
cp -rp letsencrypt /usr/local/letsencrypt
mkdir -p /usr/local/letsencrypt/challenge
mkdir -p /usr/local/letsencrypt/certs
git clone https://github.com/lukas2511/dehydrated.git /usr/local/letsencrypt/dehydrated
echo "10 6 * * 0 root /usr/local/letsencrypt/generate.cron 2>&1" >> /etc/crontab
Testing:
openssl s_client -connect firehol.org:443
openssl s_client -connect www.firehol.org:443
openssl s_client -connect test.firehol.org:443
openssl s_client -connect lists.firehol.org:443
Website statistics are generated with webalizer
aptitude install webalizer
Initial report for all existing logs:
gunzip -c `ls -t /var/log/nginx/firehol-master-access*.gz | tac` | \
webalizer - -b -i -c /home/web/firehol/static/webalizer.conf
Rest is managed by daily cron script:
cat - > /etc/cron.daily/webalizer-firehol <<_END_
#!/bin/sh
cat /var/log/nginx/firehol-master-access*.1 /var/log/nginx/firehol-master-access*.log | \
webalizer - -q -c /home/web/firehol/static/webalizer.conf
_END_
chmod +x /etc/cron.daily/webalizer-firehol
sudo -i
useradd -m travis
mkdir /home/travis/.ssh
cp /home/firehol/travis/authorized_keys /home/travis/.ssh
chown -R travis:travis /home/travis/.ssh
chmod 0700 /home/travis/.ssh
echo "* * * * * root /home/firehol/bin/travis-project.cron /home/travis/uploads /home/web/firehol/download /home/web/firehol/static/travis-project.log" >> /etc/crontab
echo "* * * * * root /home/firehol/bin/travis-website.cron /home/travis/website /home/web/firehol /home/web/firehol/static/travis-website.log" >> /etc/crontab
The scripts create a log of the most recent deployment attempt:
To see them from the command line:
curl https://firehol.org/travis-project.log
curl https://firehol.org/travis-website.log
The FireHOL lists use mailman as installed by default on Ubuntu/Debian and customised as described below.
The mailman documentation is here: http://list.org/docs.html
Connecting mailman to nginx also requires installation of fcgiwrap.
The nginx configuration for https://lists.firehol.org/ is stored as
nginx/sites-available/firehol-lists. It should be copied to the
live server and linked as sites-enabled/15-firehol-lists.
Ensure lists.firehol.org is added to /etc/postfix/main.cf for:
- mydestination
- relay_domains
and restart postfix:
/etc/init.d/postfix restart
The .cfg files are stored under the mailman directory.
For each of firehol-devs and firehol-support:
/usr/lib/mailman/bin/newlist --urlhost=lists.firehol.org
/usr/lib/mailman/bin/config_list -i firehol-xyz.cfg firehol-xyz
Visit the website and add descriptions, review all settings as required and re-save them with:
/usr/lib/mailman/bin/config_list -o firehol-xyz.cfg firehol-xyz
To import an existing mbox:
/usr/lib/mailman/bin/arch --wipe firehol-devs firehol-devs.mbox
/usr/lib/mailman/bin/arch --wipe firehol-support firehol-support.mbox
N.B. Whenever arch is run, use this to ensure that incoming mails are correctly handled:
chown -R list:web /var/lib/mailman/archives/private/*
The Gmane archives of the mailing list are searchable. To set up a search box on the mailing list archive replace the default templates:
mkdir /var/lib/mailman/lists/firehol-devs/en
cp mailman/devs-archtoc.html \
/var/lib/mailman/lists/firehol-devs/en/archtoc.html
cp mailman/devs-archtocnombox.html \
/var/lib/mailman/lists/firehol-devs/en/archtocnombox.html
mkdir /var/lib/mailman/lists/firehol-support/en
cp mailman/support-archtoc.html \
/var/lib/mailman/lists/firehol-support/en/archtoc.html
cp mailman/support-archtocnombox.html \
/var/lib/mailman/lists/firehol-support/en/archtocnombox.html
Ensure the new templates are used for incoming mail:
/etc/init.d/mailman restart
Rebuild the archives:
/usr/lib/mailman/bin/arch firehol-devs
/usr/lib/mailman/bin/arch firehol-support
chown -R list:web /var/lib/mailman/archives/private/*
These are the key files for re-creating the mailing list:
/var/lib/mailman/archives/private/firehol-devs.mbox/firehol-devs.mbox
/var/lib/mailman/archives/private/firehol-support.mbox/firehol-support.mbox
For the list members use the mailman-subscribers.py script which is from:
Usage:
./mailman-subscribers.py -c -o out.csv lists.firehol.org firehol-devs pass
Note that the script does not deal well with the complex passwords, so the adminpw may need editing internally.