Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calculation of P_W in LRS Estimate in certain environments #170

Closed
g-g-sakura opened this issue Oct 29, 2021 · 3 comments · Fixed by #175
Closed

Calculation of P_W in LRS Estimate in certain environments #170

g-g-sakura opened this issue Oct 29, 2021 · 3 comments · Fixed by #175

Comments

@g-g-sakura
Copy link

If the data type long int is 32-bit, the denominator for calculating $P_W$ in LRS estimate overflows, i.e. the binomial coefficient $\binom{L - W + 1}{2}$ is beyond the upper limit of 32-bit taking into account that $L$ is order of 10^6 and actual values of $W$.

As a result, H_original becomes infinity and H_bitstring becimes 0.13 for truerand_8bit.bin.

@joshuaehill
Copy link
Contributor

joshuaehill commented Oct 29, 2021

Yes, this tool only works on platforms on which "long int" is at least a 64 bit quantity. There are several places where this assumption is made, this is only one of them.

@joshuaehill
Copy link
Contributor

I should mention that the error-free rollover is clearly a bug, but the resolution to this bug would be to trigger an assert on such environments, so it wouldn't resolve in a particularly useful way for such architectures.

@g-g-sakura
Copy link
Author

Thank you for its investigation. The temporary solution will be to use the proposed code in Windows 10 branch.
Taking into account that the denominator can not be expressed by 32-bit, the numerator (vector S in code) should also be expressed by 64-bit for consistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants