Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add documentation for Task / Thread occupancy #549

Merged
merged 6 commits into from
Jul 30, 2024

Conversation

wolthom
Copy link
Contributor

@wolthom wolthom commented Jul 13, 2024

This PR adds a section to the Task Spawning documentation page to explain the behavior observed in #548.

@jpsamaroo jpsamaroo linked an issue Jul 14, 2024 that may be closed by this pull request
Copy link
Member

@jpsamaroo jpsamaroo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I had trouble running it locally - Documenter hangs at "ExpandTemplates: expanding markdown templates" - any ideas why?

docs/src/task-spawning.md Outdated Show resolved Hide resolved
@wolthom
Copy link
Contributor Author

wolthom commented Jul 19, 2024

@jpsamaroo I think I may have an idea:

Did you run the make.jl script from the commandline? E.g. via julia --threads=12 --project=. make.jl?

For some reason that hangs for me as well. However, if I start a REPL, activate the docs environment and include("make.jl"), it builds normally for me.

That also explains why the Documentation CI step timed out. (Just realized that).

Could this potentially be an interaction between the Dagger scheduler / the main thread of the REPL / ...?

@jpsamaroo
Copy link
Member

Yeah I ran it from the CLI as well. I'm running with the JULIA_DEBUG=Dagger env var set, but nothing prints - this makes me think that either Documenter is filtering it out. I forced it in Dagger directly, yet it still doesn't show anything, so probably it's being redirected into a file/pipe for output. This makes things particularly hard to debug... maybe I'll need to implement a debug helper that sends output to a file or something?

@wolthom
Copy link
Contributor Author

wolthom commented Jul 19, 2024

Interestingly enough, I do get output when running from the CLI with JULIA_DEBUG=Dagger:
image

When enabling the debug messages, it does not hang anymore though. Instead it actually fails with an exception in the example.

If this is specific to this environment / setup and not some underlying actual bug, I could also turn this from executed code to "dumb" code snippets as a workaround.

@jpsamaroo
Copy link
Member

jpsamaroo commented Jul 22, 2024

@wolthom yeah let's do that for now - I don't have a whole lot of time to investigate this right now, and I don't want to block adding this information to the docs, so let's just use "dumb" code blocks, and maybe you can just include the result from running on your local system?

@wolthom
Copy link
Contributor Author

wolthom commented Jul 28, 2024

@jpsamaroo I have updated the task-spawning.md note as discussed above. Now the code blocks should no longer be executed as part of the documentation build (and thus not hang).

Could you verify if this also works as expected for you?

@jpsamaroo
Copy link
Member

Worked great, thank you!

@jpsamaroo jpsamaroo merged commit 95febd2 into JuliaParallel:master Jul 30, 2024
9 of 12 checks passed
@wolthom wolthom deleted the docs-occupancy branch July 30, 2024 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallelization across threads does not appear to work correctly
2 participants