Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Numerical stability is better when array sizes are powers of two #38

Open
syncrostone opened this issue Jul 13, 2022 · 0 comments
Open

Comments

@syncrostone
Copy link
Collaborator

syncrostone commented Jul 13, 2022

Problem: when running the same code multiple times with the same seeds, there are small numerical differences that arise over the course of training. This is fixed if array sizes are powers of two.

Suggestion: Use array sizes that are powers of two for now

Eventually I would like to implement a workaround (if tensorflow doesn't have a way to activate a built in one) where if the array size is not a power of two, in the background an array with dimensions that are powers of two is made and unneeded entries are set to 0. If this is relevant to you and you want to work on that workaround please do (and drop a comment here so people don't duplicate work).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant