xorriso: "Malformed environment variable SOURCE_DATE_EPOCH encountered"

[kaihendry]

Causing the binary target to fail.

Writing to 'stdio:live-image-i386.hybrid.iso' completed successfully.

xorriso : NOTE : -return_with SORRY 32 triggered by problem severity SORRY
Makefile:40: recipe for target 'binary' failed
make: *** [binary] Error 32

[kaihendry]

Seems to have no affect on the final ISO, so I guess this message can be ignored ?

[lamby]

That's a generic error message. The one with actual details will be a few lines up.

[kaihendry]

Thank you Chris. You're right.

Real issue seems to be:

xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.

xorriso : SORRY : Malformed environment variable SOURCE_DATE_EPOCH encountered
xorriso : HINT : Unset SOURCE_DATE_EPOCH before starting xorriso or see https://reproducible-builds.org/specs/source-date-epoch/
xorriso : NOTE : Tolerated problem event of severity 'SORRY'

[matthijskooijman]

@kaihendry, did you discover what the problem with this issue is exactly? Your fix doesn't seem to match the symptoms exactly. Apart from generating SOURCE_DATE_EPOCH for only the find command and apart from breaking building with sudo, your fix only seems to change the SOURCE_DATE_EPOCH to be generated from the chroot repo instead of the Debian-live-config repo, which I think would change the date, but the problem here seems to be that it is not set at all?

Though, looking at the Dockerfile, it seems you only add the webconverger subdirectory of this repo to docker, which would probably mean that the .git directory of this repo is not available inside Docker, which would indeed cause a problem?

[matthijskooijman]

I have created an alternative fix here. I hope this fixes the build problems @usr was having, once he tested this I'll create a PR.

[lamby]

when running the build using the Dockerfile, the .git directory of this repo would not be available, so no date could be determined, which might break the build.

Ah, which was probably setting SOURCE_DATE_EPOCH to the empty string. Which sounds "malformed" :)

[matthijskooijman]

Ah, which was probably setting SOURCE_DATE_EPOCH to the empty string. Which sounds "malformed" :)

Yup, I think that's what would happen.

[kaihendry]

Sorry, I failed to figure out how to get a variable passing around a Makefile. Will test your branch when I get a chance. Thank you for looking into this! 😅

[kaihendry]

Wait, I just tested 9edba83 and I am getting different sha1sums for the resulting ISOs using https://github.com/Webconverger/Debian-Live-config/blob/master/build.sh ... am I missing something?

[matthijskooijman]

I haven't actually confirmed reproducible builds (since I don't really care about that), but it the old export SOURCE_DATE_EPOCH stuff used to work, I'd think it would work again. Not sure about any of that, though. Perhaps the xorriso version installed by Docker isn't sufficiently patched or sufficiently new after all. To confirm, you could try reverting 40c7de8 and see if that fixes the reproducible builds (on top of my branch, since without my fixes I think SOURCE_DATE_EPOCH won't be properly forwarded anyway).

[kaihendry]

Sorry for the delay, I git checkout -b revert 40c7de8a85ca992a3f341d9e04f32dd7cdb279671 and it didn't seem to make stable builds. Probably need to @lamby on the case, since I'm out of my depth / time.

[lamby]

I don't have any spare bandwidth in my free time to spend on this, alas...

[matthijskooijman]

I had a look just now, and it seems there seems to be a timestamp in some rock-ridge related file that's different, diffing gives me this:

$ cmp -l live-image-i386.hybrid.iso foo/live-image-i386.hybrid.iso 
    74001  71  60
   108817  71  60
$ hd foo/live-image-i386.hybrid.iso -s 73000 -n 2000
00011d28  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00011ff8  00 00 00 00 00 00 00 00  45 52 ed 01 0a 54 87 01  |........ER...T..|
00012008  52 52 49 50 5f 31 39 39  31 41 54 48 45 20 52 4f  |RRIP_1991ATHE RO|
00012018  43 4b 20 52 49 44 47 45  20 49 4e 54 45 52 43 48  |CK RIDGE INTERCH|
00012028  41 4e 47 45 20 50 52 4f  54 4f 43 4f 4c 20 50 52  |ANGE PROTOCOL PR|
00012038  4f 56 49 44 45 53 20 53  55 50 50 4f 52 54 20 46  |OVIDES SUPPORT F|
00012048  4f 52 20 50 4f 53 49 58  20 46 49 4c 45 20 53 59  |OR POSIX FILE SY|
00012058  53 54 45 4d 20 53 45 4d  41 4e 54 49 43 53 50 4c  |STEM SEMANTICSPL|
00012068  45 41 53 45 20 43 4f 4e  54 41 43 54 20 44 49 53  |EASE CONTACT DIS|
00012078  43 20 50 55 42 4c 49 53  48 45 52 20 46 4f 52 20  |C PUBLISHER FOR |
00012088  53 50 45 43 49 46 49 43  41 54 49 4f 4e 20 53 4f  |SPECIFICATION SO|
00012098  55 52 43 45 2e 20 20 53  45 45 20 50 55 42 4c 49  |URCE.  SEE PUBLI|
000120a8  53 48 45 52 20 49 44 45  4e 54 49 46 49 45 52 20  |SHER IDENTIFIER |
000120b8  49 4e 20 50 52 49 4d 41  52 59 20 56 4f 4c 55 4d  |IN PRIMARY VOLUM|
000120c8  45 20 44 45 53 43 52 49  50 54 4f 52 20 46 4f 52  |E DESCRIPTOR FOR|
000120d8  20 43 4f 4e 54 41 43 54  20 49 4e 46 4f 52 4d 41  | CONTACT INFORMA|
000120e8  54 49 4f 4e 2e 41 4c 2f  01 00 00 03 04 64 69 00  |TION.AL/.....di.|
000120f8  07 02 fe 01 03 0d e0 0c  00 03 04 73 74 00 0a 31  |...........st..1|
00012108  35 30 39 30 30 38 35 30  30 00 03 04 6e 74 00 04  |509008500...nt..|
00012118  01 01 01 ff 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00012128  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000124f8
$ hd foo/live-image-i386.hybrid.iso -s 108000 -n 2000
0001a5e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0001a800  45 52 ed 01 0a 54 87 01  52 52 49 50 5f 31 39 39  |ER...T..RRIP_199|
0001a810  31 41 54 48 45 20 52 4f  43 4b 20 52 49 44 47 45  |1ATHE ROCK RIDGE|
0001a820  20 49 4e 54 45 52 43 48  41 4e 47 45 20 50 52 4f  | INTERCHANGE PRO|
0001a830  54 4f 43 4f 4c 20 50 52  4f 56 49 44 45 53 20 53  |TOCOL PROVIDES S|
0001a840  55 50 50 4f 52 54 20 46  4f 52 20 50 4f 53 49 58  |UPPORT FOR POSIX|
0001a850  20 46 49 4c 45 20 53 59  53 54 45 4d 20 53 45 4d  | FILE SYSTEM SEM|
0001a860  41 4e 54 49 43 53 50 4c  45 41 53 45 20 43 4f 4e  |ANTICSPLEASE CON|
0001a870  54 41 43 54 20 44 49 53  43 20 50 55 42 4c 49 53  |TACT DISC PUBLIS|
0001a880  48 45 52 20 46 4f 52 20  53 50 45 43 49 46 49 43  |HER FOR SPECIFIC|
0001a890  41 54 49 4f 4e 20 53 4f  55 52 43 45 2e 20 20 53  |ATION SOURCE.  S|
0001a8a0  45 45 20 50 55 42 4c 49  53 48 45 52 20 49 44 45  |EE PUBLISHER IDE|
0001a8b0  4e 54 49 46 49 45 52 20  49 4e 20 50 52 49 4d 41  |NTIFIER IN PRIMA|
0001a8c0  52 59 20 56 4f 4c 55 4d  45 20 44 45 53 43 52 49  |RY VOLUME DESCRI|
0001a8d0  50 54 4f 52 20 46 4f 52  20 43 4f 4e 54 41 43 54  |PTOR FOR CONTACT|
0001a8e0  20 49 4e 46 4f 52 4d 41  54 49 4f 4e 2e 41 4c 2f  | INFORMATION.AL/|
0001a8f0  01 00 00 03 04 64 69 00  07 02 fe 01 03 0d e0 0c  |.....di.........|
0001a900  00 03 04 73 74 00 0a 31  35 30 39 30 30 38 35 30  |...st..150900850|
0001a910  30 00 03 04 6e 74 00 04  01 01 01 ff 00 00 00 00  |0...nt..........|
0001a920  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0001adb0

Note that this just dumps one iso for some context, the difference is
in the timestamp (1509008500). @lamby, perhaps this rings a bell for you?

In any case, I confirmed that the SOURCE_DATE_EPOCH is correctly passed to xorriso, with and without my changes, and the same reproducibility problem exists in the current master branch, so I think my changes are not introducing this problem. I'll go ahead and submit them as a PR to be merged (so at least builds as non-root work again) and if I have some leftover time, look at the reproducibility issue more closely.

I also noticed once that git created pack files with different filenames (hashes), but perhaps I was doing something differently then (I could not reproduce this anymore, with or without my changes).

[lamby]

Are you sure you are using the patched xorisso?

[kaihendry]

I'm not sure, how do I tell ?

[matthijskooijman]

Are you sure you are using the patched xorisso?

I removed the patches and backports-from-sid in 40c7de8. At that time, there were no xorriso patches left in this repository, but it still backported xorriso from sid, so I assumed that the sid version has the necessary patches. At that time, the xorriso version (and other backported packages) in sid was the same as in stretch, so it seemed I could safely remove this backporting.

The changelog for the stretch version (1.4.6-1, which I am running) has:

libisoburn (1.4.6-1) unstable; urgency=low

  * New upstream release
    (...)
    + New environment variable SOURCE_DATE_EPOCH

Which I think is the patch you are referring to?

I also tried upgrading xorriso and its dependencies (libburn4, libisoburn1 and libisofs6) to the sid versions, but that did not make any difference.

[matthijskooijman]

Actually, it seems that with the sid versions of these packages, I get a lot more differences in the resulting iso files (around 50).

[lamby]

I get a lot more differences in the resulting iso files (around 50).

Isn't that expected, ie. you are missing the patches that I added to some WebConverger repo?

(This was well over a year ago, sorry if my memory is hazy... plus slammed at the moment for time!)

[matthijskooijman]

@lamby, it seems you removed those patches here (75ebaab), instead using the sid version, so I assumed the patches were merged. You mention 1.4.6 in the commit message, which I think is also what I was using before, so apparently that version is not completely ok yet (or perhaps some new issue popped up)?

[lamby]

Quite possible. But I'm sorry but I just simply can't remember and don't have the bandwidth/time to reload/try all this out right now…

[matthijskooijman]

That's fine, I'm not expecting you to :-) Just trying to get as much info in this ticket for when someone does have time to investigate further (I'm also a bit busy with other stuff right now).

[lamby]

nodnod!