Borg fails when file changes on disk during archive creation

[dmaroulidis]

Describe the bug

Borg_ynh fails to backup an application when files on disk change during archive creation. Probably related to borgbackup/borg#6989. In my situation specifically, as you can see from the backup logs, matrix-synapse adds to homeserver.log during archive creation which causes borg to emit a warning and probably a signal that backup-with-borg receives and stops the process. I tried stopping synapse before starting the borg.service and the backup completes successfully.

Context

• Hardware: VM with VPN
• YunoHost version: 11.2.12
• I have access to my server: Through SSH | through the webadmin | direct access via keyboard / screen
• Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no
• Using, or trying to install package version/branch: borg 1.2.8~ynh1

Steps to reproduce

Upgrade borg_ynh to the latest version 1.2.8~ynh1 and get notified of the issue by email every morning.

Expected behavior

A solution would be to stop each service before backing it up and restart it after the backup is finished. A better setup would be to take a snapshot of the filesystem and backup the snapshot which would result in zero downtime, but that cannot be accomplished without a filesystem that supports this.

Logs

https://paste.yunohost.org/raw/duzoyalaho

[utzer]

I got the same problem since the late upgrade of borg backup.

2024-06-03 10:17:49,897: DEBUG - + /var/www/borg/venv/bin/borg init -e repokey ssh://user01@server01/./backup
2024-06-03 10:17:50,872: WARNING - A repository already exists at ssh://user01@server01/./backup.
2024-06-03 10:17:50,890: DEBUG - + set -e
2024-06-03 10:17:50,890: DEBUG - + /var/www/borg/venv/bin/borg create ssh://user01@server01/./backup::_auto_synapse-2024-06-03_10:17 ./
2024-06-03 10:17:50,890: DEBUG - + log_with_timestamp
2024-06-03 10:17:50,891: DEBUG - + tee -a /var/log/backup_borg.err
2024-06-03 10:17:50,891: DEBUG - ++ date +%Y-%m-%d_%H:%M:%S
2024-06-03 10:17:50,892: DEBUG - + sed -e 's/^/[2024-06-03_10:17:50] /'
2024-06-03 10:18:09,837: DEBUG - [2024-06-03_10:17:50] apps/synapse/backup/var/log/matrix-synapse/homeserver.log: file changed while we backed it up
2024-06-03 10:18:10,838: ERROR - Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 298, in hook_callback
    hook_return = hook_exec(
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 416, in hook_exec
    raise YunohostError("hook_exec_failed", path=path)
yunohost.utils.error.YunohostError: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app

[CodeShakingSheep]

Same.

[utzer]

Anyone knows if there is some work done on this and if this here it the right place to report, maybe it should be reported via synapase package?

[utzer]

Sorry, it just bugs me that I get a "backup failed" email every two hours now. :-(

[utzer]

is there a possibility to exclude files from the backup, I mean the log is nice to have in some cases, but not the most important as well, so maybe we can simply exclude it for now?

[dmaroulidis]

@utzer I've looked through the backup script of synapse (the file that exports the file list to be included in the backup) but I couldn't figure out how to exclude the log from backups. That's something the maintainers of ynh_synapse could help with, but that would be a patch.

The bigger issue is that this thing can happen with any service that changes its files while the backup is running. That's why I suggested stopping each service before backing it up. I think this can be done in the yunohost backup framework, so that might be something to take to the yunohost repo or at least the forum.

I reverted to my pre-upgrade backup of borg and I've been using the previous version for the last 3 weeks. I'll update when this is fixed. Note that the same situation (a file is changed during backup) can happen with the previous version as well but the backup won't fail if this happens.

[Thatoo]

I face the same issue since I last updated synapse indeed.

I opened an issue on synapse repository.

[Thatoo]

@Josue-T answered on the issue opened here YunoHost-Apps/synapse_ynh#474 (comment)
Hoping it will help.

[utzer]

@fflorent I wonder what the status of the PR is?

I also wonder if it would be a good idea to add a function to the borg app to executed a command before the borg create starts and one after borg create finished? This way it would also be possible to run "systemctl stop synapse.service" before borg create and "systemctl start synapse.service" after borg create finished.