Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using a seeded Faker instance with Recipes breaks in versions 1.3.3 and greater #349

Closed
epicserve opened this issue Sep 29, 2022 · 6 comments
Assignees

Comments

@epicserve
Copy link

Describe the issue
When using a seeded faker instance to generate strings (e.g. username) the assertions randomly fail, in versions 1.3.3 and greater of Model Bakery.

To Reproduce
You can view my example project that has tests that reproduce the bug using tox to run the tests in multiple versions of Model Bakery and Django. It also has a detailed README.md with everything that has been tried.

But you can reproduce it with a test like the following and running it multiple times using something like: for i in {1..20}; do pytest tests/test_recipes.py::test_faker_recipe; done

import pytest
from faker import Faker
from model_bakery.recipe import Recipe

@pytest.mark.django_db
def test_faker_recipe(self):
    faker = Faker()
    faker.seed_instance(10)

    user_recipe = Recipe(
        "auth.User",
        username=faker.user_name,
        email=faker.email,
    )

    user = user_recipe.make()
    assert user.username == "pattersonbelinda"
    assert user.email == "[email protected]"

Expected behavior
It would be expected that no matter how many times you run the tests, they would pass every time and not randomly fail.

Versions

  • Python: 3.9
  • Django 3.2, 4.0
  • Model Bakery >=1.3.3
@amureki
Copy link
Collaborator

amureki commented Oct 4, 2022

Hey @epicserve thanks for raising the issue!

I am not familiar with Faker, but happy to try to understand what is going on here.
Definitely kudos for providing test scenario and even making an example repository! 🏅
I tried to run it and indeed sometimes it fails.

May I ask why would you not define faker methods in recipe as callable?

    user_recipe = Recipe(
        "auth.User",
        username=faker.user_name(),
        email=faker.email(),
    )

So far, that's the way of using methods in the recipes. However, you might be affected by the issue described in #49. Is that the case?

Best,
Rust

@epicserve
Copy link
Author

Hi @amureki,

Thank you for looking into this issue. The test repo is just a reproduction of the issue we're having in our main product and doesn't really capture the use case. Our product has exports where users can export their data. To test the export, we run a create data function that creates records using Model Bakery and Faker. When we create the data the first time, we create a snapshot, and we use faker.seed_instance(10) to make sure when we run the test again, the Faker methods we use to create the same values every time instead of random data.

As to your direct question ... why don't we call the faker methods? If we called the faker method when the recipe was initialized, it wouldn't be seeded yet and would be random every time. Or at least I think that is the case.

#49 may be related, I can do some more digging on it.

@amureki
Copy link
Collaborator

amureki commented Oct 11, 2022

@epicserve okay, I think, I found where the issue comes from:

for field in self.get_fields():

self.get_fields() is not ordered (set), so sometimes email field comes before username, so with the given seed, faker generates [email protected] as an email and rli as a username.

We could consider reworking this bit and making it ordered (so for example, simply wrapping it in sorted()), but I need to think on the possible consequences...

@epicserve
Copy link
Author

@amureki, THANK YOU SO MUCH! I tried your PR in the bug project, and it seems to have solved it. I'll also try it out in our product to see if it solves it there. It seems like all the tests pass in the PR. Any chance this could get merged and get a release pushed out? This bug is blocking us on some work we're trying to get done.

@amureki amureki self-assigned this Oct 13, 2022
@amureki
Copy link
Collaborator

amureki commented Oct 13, 2022

@epicserve https://pypi.org/project/model-bakery/1.8.0/ is out :)
Looking forward to getting your feedback from the product test. 👍

@epicserve
Copy link
Author

@amureki, it seems to be working in my project as well!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants