-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
presto-postgresql tests fail on ppc64le #14550
presto-postgresql tests fail on ppc64le #14550
Comments
After some debugging by adding some logs in the code https://github.com/prestodb/testing-postgresql-server (tag 9.6.3-4) and updating the class file in ~/.m2/repository/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar, I found that the issue is with initdb() call. It's unable to find a library which causes the initdb to fail and hence close() gets called. Here's the log:
I checked LD_LIBRARY_PATH and also tried setting the libdir as env to point to the local postgresql library location (/opt/rh/rh-postgresql96/root/usr/lib64). But the issue still persists. Any pointers? |
Apart from this, can someone please let me know whether prestodb community handles up-streaming of the necessary jars to maven repos? e.g. https://repo.maven.apache.org/maven2/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar using https://github.com/prestodb/testing-postgresql-server. If yes, what's the procedure for up-streaming the jars to the respective maven repos? This information will be of great help. Thanks! |
I got past the $libdir related issue by updating postgresql binaries, libraries in postgresql-Linux-ppc64le.tar.gz. Next error was: FATAL: could not create lock file "/var/run/postgresql/.s.PGSQL.33158.lock": Permission denied . So, changed ownership of /var/run/postgresql/ to non-root testuser. Post that, I got the following error:
The reason for this error was unavailability of file_fdw extension which is part of contrib package and needs to be copied to the jar. The line that was causing the exception and resulting in failure is After copying the file_fdw extension and the related file_fdw.so to the jar, I got syntax error at or near "MATERIALIZED". And that resulted in 1 test failure, rest all tests passed. After some debugging, I found that the error was because the expected postgresql version for tests is 9.6.x and the one that I installed/copied binaries for was 9.2.x. So, I copied 9.6.x to the jar as:
This resolved all test failures. |
The only info I need now is: Whether prestodb community handles up-streaming of the necessary jars to maven repos. e.g. https://repo.maven.apache.org/maven2/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar using https://github.com/prestodb/testing-postgresql-server. If yes, what's the procedure for up-streaming the jars to the respective maven repos? Thanks! |
Can you explain what you mean by upstreaming the jars? When we release, we deploy the new versions to maven central. |
Thanks for the response @rschlussel ! By upstreaming, I meant pushing/deploying the generated jars to respective maven repositories. Based on your answer, I'm guessing that I just need to raise PR to https://github.com/prestodb/testing-postgresql-server in order to get the ppc64le binaries/libraries packed in the jar and once you merge+release+deploy, I'll need to raise a PR to https://github.com/prestodb/presto with updated testing-postgresql-server dependency. Is that correct understanding? Also, could you please confirm if the following are correct repositories to be followed for mysql, redis, hive related build/tests: I needed to make similar changes as postgresql to testing-mysql-server-5-0.6.jar, embedded-redis-0.6.jar, hadoop-apache2-2.7.4-7.jar, snappy-java-1.1.7.1.jar. |
yes, that all looks correct, and we can create new releases for any of the prestodb repos |
Thanks @rschlussel for the confirmation! I'll raise PRs on the respective repos then. |
@rschlussel I have a question regarding the cross OS viability of the jars generated by these repos. e.g. hadoop-apache2-2.7.4-7.jar which gets downloaded from central maven repo has libhadoop.so, libsnappy.so in nativelib/Linux-ppc64le/ directory. But since those are not compatible with RHEL 7.6 ppc64le (maybe compatible with ubuntu), it resulted in an error So, my question is, is it OK to focus on a specific OS for ppc64le similar to what's done for x86_64 here? |
Also, one correction, for future references, the repo that is currently being used for embedded-redis is https://github.com/gbrd/embedded-redis. There was a plan to switch to https://github.com/kstyrc/embedded-redis. But that didn't see completion. Ref: kstyrc/embedded-redis#71 |
yeah, if the jars are not compatible across different oses , it's fine to focus on a particular os |
Thanks @rschlussel for the feedback! |
@rschlussel I tried to raise PR on testing-postgresql-server repo (prestodb/testing-postgresql-server#2) yesterday. There was no mention of a CLA in the repo or old PR, but an EASYCLA check popped up in the PR. Could you please help me with the following info?
|
@amitsadaphule it's the same cla across prestodb. It's possible old prs are from before the presto foundation was formed, so they didn't have the cla configured. The link should take you to a page where you select whether you are contributing on behalf of an individual or corporation, and then take you to the appropriate page to sign. Once it's been signed, you'll need to push an update to your pr so the check runs again (unfortunately it won't detect the change automatically). |
OK. Thanks for the feedback! I'll raise another PR and follow your instructions. |
ENV:
arch=ppc64le
os=RHEL7.6
maven=3.6.3
I was able to compile presto commit c51b285 on ppc64le RHEL 7.6 as
mvn install -DskipTests
. However, there were test failures for presto-postgresql caused by the following error:After some debugging, I found that the issue was due to missing postgresql-Linux-ppc64le.tar.gz in ~/.m2/repository/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar. I could get this particular issue resolved by adding the necessary binaries, libraries, etc. from rh-postgresql96 installation from my setup to postgresql-Linux-ppc64le.tar.gz and then adding that as a part of testing-postgresql-server-9.6.3-4.jar in the local maven repo.
However, I started getting a different error after that:
I tried creating a directory data as a part of the tarball in the jar itself alongside the bin, lib, etc. directories. But that didn't help either.
Please provide me a pointer to help resolve this issue.
Thanks in advance!
The text was updated successfully, but these errors were encountered: