Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-36302: Sort list of sources #12341

Merged
merged 1 commit into from
Aug 1, 2019
Merged

Conversation

bmwiedemann
Copy link
Contributor

@bmwiedemann bmwiedemann commented Mar 15, 2019

When building packages (e.g. for openSUSE Linux)
(random) filesystem order of input files
influences ordering of functions in the output .so files.
Thus without the patch, builds (in disposable VMs) would usually differ.

Without this patch, all callers have to be patched individually
ofalk/libdnet#42
sass/libsass-python#212
tahoe-lafs/pycryptopp#41
yt-project/yt#2206
pyproj4/pyproj#142
pytries/datrie#49
but that is an infinite effort.

See https://reproducible-builds.org/ for why this matters.

https://bugs.python.org/issue36302

@merwok
Copy link
Member

merwok commented Mar 15, 2019

Could there be negative consequences to the sorting?

I think the two most common idioms for defining sources in setup.py are 1) an explicit list with fixed order (e.g. sources = ["lib/thing.c", "lib/utils.c"]) and 2) a glob result.

Is it ok in 100% of cases to change the order for case 1) ?

@bmwiedemann
Copy link
Contributor Author

Let's say the case 2 people seem to not have noticed any negative consequences from random order, so I would assume that it is fine for case 1, too. I also mailed this to our expert to be on the safe side.

@bmwiedemann
Copy link
Contributor Author

Our compiler expert said, order does not matter for .so files. It only matters when using .a files in case of duplicate symbols - but those are for static linking and not for dynamic linking as required by python.

@bmwiedemann
Copy link
Contributor Author

ping. Anything left to do on my side?

@csabella csabella requested review from merwok and dstufft May 30, 2019 19:40
@bmwiedemann
Copy link
Contributor Author

bmwiedemann commented Jul 27, 2019

ping. Still wanting this. Hoping for @vstinner or who else could have a look.

Because I found another of the infinite callers.

bmwiedemann added a commit to bmwiedemann/pyreadstat that referenced this pull request Jul 29, 2019
so that _readstat_parser.so builds in a reproducible way
in spite of indeterministic filesystem readdir order
and missing python/cpython#12341

See https://reproducible-builds.org/ for why this is good.
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to modify where self.sources is used rather than sorting Extension constructor? For example, read_setup_file() function seems to modify the sources attribute afterwards.

Maybe modify CCompiler.compile()?

@bmwiedemann
Copy link
Contributor Author

bmwiedemann commented Jul 30, 2019

patching build_ext would probably work (edit: does not need test updates). read_setup_file code looks deterministic enough (unless someone autogenerates the setup file nondeterministically).
CCompiler.compile is overridden in msvccompiler.py so might not catch all cases.

link_shared_object might also be a good place to patch instead, because after all it is the linking order of .o files into .so that matters - it just happens to follow the source file order.

@bmwiedemann
Copy link
Contributor Author

moved the sorting into build_ext

when building packages (e.g. for openSUSE Linux)
(random) filesystem order of input files
influences ordering of functions in the output .so files.
Thus without the patch, builds (in disposable VMs) would usually differ.

Without this patch, all callers have to be patched individually
ofalk/libdnet#42
sass/libsass-python#212
tahoe-lafs/pycryptopp#41
yt-project/yt#2206
pyproj4/pyproj#142
pytries/datrie#49
Roche/pyreadstat#37
but that is an infinite effort.

See https://reproducible-builds.org/ for why this matters.
@vstinner
Copy link
Member

vstinner commented Aug 1, 2019

Thanks for updated PR: sorting in build_extension() looks more reliable than sorting in Extension constructor.

@vstinner vstinner merged commit 0d30ae1 into python:master Aug 1, 2019
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
When building packages (e.g. for openSUSE Linux)
(random) filesystem order of input files
influences ordering of functions in the output .so files.
Thus without the patch, builds (in disposable VMs) would usually differ.

Without this patch, all callers have to be patched individually
ofalk/libdnet#42
sass/libsass-python#212
tahoe-lafs/pycryptopp#41
yt-project/yt#2206
pyproj4/pyproj#142
pytries/datrie#49
Roche/pyreadstat#37
but that is an infinite effort.

See https://reproducible-builds.org/ for why this matters.
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
When building packages (e.g. for openSUSE Linux)
(random) filesystem order of input files
influences ordering of functions in the output .so files.
Thus without the patch, builds (in disposable VMs) would usually differ.

Without this patch, all callers have to be patched individually
ofalk/libdnet#42
sass/libsass-python#212
tahoe-lafs/pycryptopp#41
yt-project/yt#2206
pyproj4/pyproj#142
pytries/datrie#49
Roche/pyreadstat#37
but that is an infinite effort.

See https://reproducible-builds.org/ for why this matters.
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020
When building packages (e.g. for openSUSE Linux)
(random) filesystem order of input files
influences ordering of functions in the output .so files.
Thus without the patch, builds (in disposable VMs) would usually differ.

Without this patch, all callers have to be patched individually
ofalk/libdnet#42
sass/libsass-python#212
tahoe-lafs/pycryptopp#41
yt-project/yt#2206
pyproj4/pyproj#142
pytries/datrie#49
Roche/pyreadstat#37
but that is an infinite effort.

See https://reproducible-builds.org/ for why this matters.
@bmwiedemann bmwiedemann deleted the distutils branch January 29, 2024 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants