gh-102701: Fix overflow in dictobject.c

[methane]

• Issue: dict() resize failure #102701

[markshannon]

The change looks good, but it will need a test and a machine large enough to test it on.

[methane]

I used GCP n2d-standard-16 (64GiB RAM):

inada.naoki@instance-1:~/cpython$ /usr/bin/time ./python -m test -vM 64g -m DictTest test_bigmem
== CPython 3.12.0a6+ (heads/main-dirty:1c9f3391b9, Mar 16 2023, 08:21:15) [GCC 12.2.0]
== Linux-5.19.0-1015-gcp-x86_64-with-glibc2.36 little-endian
== Python build: debug
== cwd: /home/inada.naoki/cpython/build/test_python_3512æ
== CPU count: 16
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 0.51 Run tests sequentially
0:00:00 load avg: 0.51 [1/1] test_bigmem
test_dict (test.test_bigmem.DictTest.test_dict) ...
 ... expected peak memory use: 53.3G
 ... process data size: 0.1G
 ... process data size: 0.5G
 ... process data size: 0.9G
 ... process data size: 1.4G
(snip)
 ... process data size: 51.7G
 ... process data size: 51.7G
 ... process data size: 41.0G
(snip)
 ... process data size: 20.0G
ok

----------------------------------------------------------------------
Ran 1 test in 123.835s

OK
test_bigmem passed in 2 min 3 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 min 3 sec
Tests result: SUCCESS
106.85user 17.31system 2:04.05elapsed 100%CPU (0avgtext+0avgdata 53939612maxresident)k
0inputs+0outputs (0major+16111371minor)pagefaults 0swaps

53939612maxresident)k is 51.44GiB. So expected peak memory use: 53.3G is accurate enough.

[furkanonder]

I am asking for my curiosity(to become familiar with cpython); How does casting the result of 1<<log2_size to size_t fix the overflow in the dict?

[methane]

1 is int. And int is 32bit on most platforms. So 1 << 32 will overflow.
Here is example:

$ cat x.c
#include <stdio.h>
#include <stdlib.h>

#define USABLE_FRACTION(n) (((n) << 1)/3)

int main() {
    size_t x = USABLE_FRACTION(1 << 31);
    size_t y = USABLE_FRACTION((size_t)1 << 31);
    printf("x=%zd y=%zd\n", x, y);
}

$ gcc x.c && ./a.out
x.c:7:16: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
    size_t x = USABLE_FRACTION(1 << 31);
               ^~~~~~~~~~~~~~~~~~~~~~~~
x.c:4:34: note: expanded from macro 'USABLE_FRACTION'
#define USABLE_FRACTION(n) (((n) << 1)/3)
                             ~~~ ^
1 warning generated.
x=0 y=1431655765

[furkanonder]

Thanks!

[miss-islington]

Thanks @methane for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[bedevere-bot]

GH-102777 is a backport of this pull request to the 3.11 branch.