Add Jenkin's lookup3 as a 32-bit checksum for HDF5

[mkitti]

Minimal, reproducible code sample, a copy-pastable example if possible

# Jenkin's lookup3 can be useful for verifying HDF5 data structures such as the superblock below
import jenkins_cffi

with open("original_hdf5_zarr_shard_demo.h5", "rb") as f:
    b = f.read(48)

hash_bytes = jenkins_cffi.hashlittle(bytes(b[:-4])).to_bytes(4, "little")
print(b[-4:] == hash_bytes) # True

Problem description

Jenkin's lookup3 is an integral component of the HDF5 specification for its internal datastructures.

This becomes relevant if we would like to reuse HDF5 data structures. For example, the HDF5 Fixed Array Data Block can made byte compatible with the proposed Zarr shard specification, except for the four byte checksum. Currently, the only permitted checksum is crc32.

zarr-developers/zarr-specs#152 (comment)

An implementation of Bob Jenkin's lookup3 is widely available across many languages.

• https://en.wikipedia.org/wiki/Jenkins_hash_function
• https://www.burtleburtle.net/bob/hash/doobs.html
• C: https://www.burtleburtle.net/bob/c/lookup3.c
• C: https://github.com/HDFGroup/hdf5/blob/3af8bb267d6ad2d4e8e5d77e16d5a8f7625ad34d/src/H5checksum.c#L364-L458
• Python: https://pypi.org/project/jenkins/ (old PyPi package)
• Python: https://pypi.org/project/jenkins-cffi/
• JavaScript: https://www.npmjs.com/package/jenkins-hash-lookup3
• Julia: https://github.com/JuliaIO/JLD2.jl/blob/master/src/Lookup3.jl

Version and installation information

Please provide the following:

jenkins-cffi              1.0.2.1                  pypi_0    pypi
python                    3.11.0          he550d4f_1_cpython    conda-forge

[mkitti]

Would the best route be to use cython as @martindurant did in #412 for fletcher32 or would it make sense to add jenkins-cffi as a dependency https://github.com/what-studio/jenkins-cffi/ ?

[rabernat]

We tend to favor either vendoring outside libraries or reimplementing algorithms in Cython. (Numcodecs has no dependencies other than numpy.) Both options seem plausible here. I'd go for the path of least resistance.

@mkitti - are you interested in working on PR to add this codec? We will be glad to support you.

[mkitti]

I would be happy to work on it if there is a clear path forward. I would prefer to avoid the outcome of the SZIP/libaec effort with the pull request: #420 / #422