Issue runnig engine with undefined symbol: EC_GROUP_new_curve_GF2m

[hifi-ua]

Engine built and installed successfully but test going to fail on Fedora Linux x64

$ sudo make install

...
Install the project...
-- Install configuration: ""
-- Installing: /usr/lib64/engines-1.1/dstu.so
-- Installing: /usr/local/lib/libkeylib.so
-- Up-to-date: /usr/local/include/keylib/keylib.h
-- Up-to-date: /usr/local/include/keylib/keystore.h
-- Up-to-date: /usr/local/include/keylib/key6.h
-- Up-to-date: /usr/local/include/keylib/jks.h
-- Up-to-date: /usr/local/include/keylib/pkcs12.h

$ openssl cms -verify -engine dstu -in tests/cms.pem -inform PEM -noverify

invalid engine "dstu"
140498579156480:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(/usr/lib64/engines-1.1/dstu.so): /usr/lib64/engines-1.1/dstu.so: undefined symbol: EC_GROUP_new_curve_GF2m
140498579156480:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
140498579156480:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:434:
140498579156480:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:421:id=dstu
140498579156480:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(libdstu.so): libdstu.so: cannot open shared object file: No such file or directory
140498579156480:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
140498579156480:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:434:
Verification failure
140498579156480:error:0609E09C:digital envelope routines:pkey_set_type:unsupported algorithm:crypto/evp/p_lib.c:210:
140498579156480:error:0B09406F:x509 certificate routines:x509_pubkey_decode:unsupported algorithm:crypto/x509/x_pubkey.c:114:
140498579156480:error:2E098086:CMS routines:CMS_SignerInfo_verify:no public key:crypto/cms/cms_sd.c:733:

$ openssl version
OpenSSL 1.1.1q FIPS 5 Jul 2022

Could you clarify if it indicates some missing debs, a new version of OpenSSL, or distro specific issue?

[madf]

The error looks strange. EC_GROUP_new_curve_GF2m is a function from the OpenSSL libcrypto.so, and the engine depends on it. It looks like I need to install Fedora and look at the issue myself. Meanwhile, could you please show me the output of the following commad:

ldd /usr/lib64/engines-1.1/dstu.so

Also it would be helpful to see the full build log, including the configuration part.

[madf]

It seems that RedHat excluded support of elliptic curves over binary fields from OpenSSL package for some obscure legal reasons. With this, it is not possible to use dstu engine with stock OpenSSL in Fedora. If switching to other Linux distributive is not an option, you can try to compile your own version of OpenSSL without no-ec2m configuration option (though I don't recommend it).

[madf]

Closing in favor of #20