Test sometimes fail

[KristofferC]

It seems the test:

metropolis: Test Failed at /home/pkgeval/.julia/packages/KissMCMC/z6iaT/test/runtests.jl:37
  Expression: all(abs.(std(thetas) - std_) .< abs.(std_ * tol))
Stacktrace:
 [1] test_mean_std(::Array{Array{Float64,1},1}, ::ATest, ::Float64) at /home/pkgeval/.julia/packages/KissMCMC/z6iaT/test/runtests.jl:37
 [2] macro expansion at /home/pkgeval/.julia/packages/KissMCMC/z6iaT/test/metro.jl:16 [inlined]
 [3] macro expansion at /workspace/srcdir/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1116 [inlined]
 [4] top-level scope at /home/pkgeval/.julia/packages/KissMCMC/z6iaT/test/metro.jl:3
Test Summary: | Pass  Fail  Total
metropolis    |   27     1     28

sometimes randomly fail (perhaps the tolerance is too tight or something).

[mauro3]

And I thought no-one would notice... It's stochastic, so it fails at times. I certainly can adjust the tolerance. What would be a acceptable failure rate?

[KristofferC]

Can't you initialize with an explicit random seed?

[mauro3]

Yes, that should work. But only for single threaded (which is all I test at the moment). Or is there a way to make threaded also deterministic? This simple test is not:

julia> using Random, Base.Threads; Random.seed!(1); n = 1000;

julia> a = zeros(n);

julia> @threads for i = 1:n
                  a[i] = rand()
         end; 

julia> sum(a)

[KristofferC]

Maybe instead of Random.seed!(1) do

@threads for i in 1:nthreads()
    Random.seed!(Random.default_rng(), i)
end

[mauro3]

Cool, yes, that seems to work:

using Random, Base.Threads
n = 1000
a = zeros(n);
@threads for i in 1:nthreads()
    Random.seed!(Random.default_rng(), i)
end
@threads for i = 1:n
    a[i] = rand()
end;
sum(a) # -> 511.1700285629997

I'm a bit surprised by this, as I thought that the RNG would be setup as in the docs, i.e. using several locations in one chain (i.e. using Future.randjump); thus I thought one seed would be enough. Instead separate chains are used: https://github.com/JuliaLang/julia/blob/6144a5d7117f9e407bca2e8d7476cce3d586e858/stdlib/Random/src/RNGs.jl#L297

Do you know why the rnadjump strategy was not used? For instance wikipedia (4th bullet) recons that for MCMC doing the separate chains might be bad.

[KristofferC]

Nope, I know very little about the RNG stuff. Might be good to open an issue?

[mauro3]

All I know about RNGs is from the rabbit hole I just descended after wondering why the documentation-example used the randjump approach. I'll post a comment in JuliaLang/julia#32407 and open an issue otherwise.

[mauro3]

Issue JuliaLang/julia#34386