Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

presto-postgresql tests fail on ppc64le #14550

Closed
amitsadaphule opened this issue May 18, 2020 · 16 comments · Fixed by prestodb/testing-postgresql-server#3
Closed

presto-postgresql tests fail on ppc64le #14550

amitsadaphule opened this issue May 18, 2020 · 16 comments · Fixed by prestodb/testing-postgresql-server#3

Comments

@amitsadaphule
Copy link
Contributor

ENV:
arch=ppc64le
os=RHEL7.6
maven=3.6.3

I was able to compile presto commit c51b285 on ppc64le RHEL 7.6 as mvn install -DskipTests. However, there were test failures for presto-postgresql caused by the following error:

io.airlift.command.CommandFailedException: [/tmp/testing-postgresql-server5907429493503395537/bin/pg_ctl, stop, -D, /tmp/testing-postgresql-server5907429493503395537/data, -m, fast, -t, 5, -w] failed to start: Cannot run program "/tmp/testing-postgresql-server5907429493503395537/bin/pg_ctl" (in directory "/root/presto/presto-postgresql/."): error=2, No such file or directory

After some debugging, I found that the issue was due to missing postgresql-Linux-ppc64le.tar.gz in ~/.m2/repository/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar. I could get this particular issue resolved by adding the necessary binaries, libraries, etc. from rh-postgresql96 installation from my setup to postgresql-Linux-ppc64le.tar.gz and then adding that as a part of testing-postgresql-server-9.6.3-4.jar in the local maven repo.

However, I started getting a different error after that:

2020-05-13T10:33:14.269-0500 SEVERE could not stop postmaster in /tmp/testing-postgresql-server9157088070576433197
java.lang.RuntimeException: io.airlift.command.CommandFailedException: [/tmp/testing-postgresql-server9157088070576433197/bin/pg_ctl, stop, -D, /tmp/testing-postgresql-server9157088070576433197/data, -m, fast, -t, 5, -w] exited with 1
pg_ctl: directory "/tmp/testing-postgresql-server9157088070576433197/data" does not exist

I tried creating a directory data as a part of the tarball in the jar itself alongside the bin, lib, etc. directories. But that didn't help either.

Please provide me a pointer to help resolve this issue.

Thanks in advance!

@amitsadaphule
Copy link
Contributor Author

After some debugging by adding some logs in the code https://github.com/prestodb/testing-postgresql-server (tag 9.6.3-4) and updating the class file in ~/.m2/repository/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar, I found that the issue is with initdb() call. It's unable to find a library which causes the initdb to fail and hence close() gets called. Here's the log:

2020-05-26T16:51:17.937-0500 INFO !!!!!!caught Exception in EmbeddedPostgreSql
java.lang.RuntimeException: io.airlift.command.CommandFailedException: [/tmp/testing-postgresql-server4098338493358230158/bin/initdb, -A, trust, -U, postgres, -D, /tmp/testing-postgresql-server4098338493358230158/data, -E, UTF-8] exited with 1
The files belonging to this database system will be owned by user "prestouser".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /tmp/testing-postgresql-server4098338493358230158/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... FATAL:  could not access file "$libdir/ascii_and_mic": No such file or directory
STATEMENT:  -- SQL_ASCII --> MULE_INTERNAL
        CREATE OR REPLACE FUNCTION ascii_to_mic (INTEGER, INTEGER, CSTRING, INTERNAL, INTEGER) RETURNS VOID AS '$libdir/ascii_and_mic', 'ascii_to_mic' LANGUAGE C STRICT PARALLEL SAFE;
        COMMENT ON FUNCTION ascii_to_mic(INTEGER, INTEGER, CSTRING, INTERNAL, INTEGER) IS 'internal conversion function for SQL_ASCII to MULE_INTERNAL';
        CREATE DEFAULT CONVERSION pg_catalog.ascii_to_mic FOR 'SQL_ASCII' TO 'MULE_INTERNAL' FROM ascii_to_mic;
        COMMENT ON CONVERSION pg_catalog.ascii_to_mic IS 'conversion for SQL_ASCII to MULE_INTERNAL';

child process exited with exit code 1
initdb: removing data directory "/tmp/testing-postgresql-server4098338493358230158/data"

        at com.facebook.airlift.testing.postgresql.EmbeddedPostgreSql.system(EmbeddedPostgreSql.java:309)
        at com.facebook.airlift.testing.postgresql.EmbeddedPostgreSql.initdb(EmbeddedPostgreSql.java:174)
        at com.facebook.airlift.testing.postgresql.EmbeddedPostgreSql.<init>(EmbeddedPostgreSql.java:94)
        at com.facebook.airlift.testing.postgresql.TestingPostgreSqlServer.<init>(TestingPostgreSqlServer.java:43)
        at com.facebook.presto.plugin.postgresql.TestPostgreSqlIntegrationSmokeTest.<init>(TestPostgreSqlIntegrationSmokeTest.java:49)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:29)
        at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:363)
        at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:275)
        at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:126)
        at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:191)
        at org.testng.TestClass.getInstances(TestClass.java:100)
        at org.testng.TestClass.initTestClassesAndInstances(TestClass.java:86)
        at org.testng.TestClass.init(TestClass.java:78)
        at org.testng.TestClass.<init>(TestClass.java:41)
        at org.testng.TestRunner.initMethods(TestRunner.java:425)
        at org.testng.TestRunner.init(TestRunner.java:252)
        at org.testng.TestRunner.init(TestRunner.java:222)
        at org.testng.TestRunner.<init>(TestRunner.java:163)
        at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:585)
        at org.testng.SuiteRunner.init(SuiteRunner.java:189)
        at org.testng.SuiteRunner.<init>(SuiteRunner.java:136)
        at org.testng.TestNG.createSuiteRunner(TestNG.java:1375)
        at org.testng.TestNG.createSuiteRunners(TestNG.java:1355)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1209)
        at org.testng.TestNG.runSuites(TestNG.java:1133)
        at org.testng.TestNG.run(TestNG.java:1104)
        at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
        at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
        at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
        at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)

I checked LD_LIBRARY_PATH and also tried setting the libdir as env to point to the local postgresql library location (/opt/rh/rh-postgresql96/root/usr/lib64). But the issue still persists. Any pointers?

@amitsadaphule
Copy link
Contributor Author

Apart from this, can someone please let me know whether prestodb community handles up-streaming of the necessary jars to maven repos? e.g. https://repo.maven.apache.org/maven2/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar using https://github.com/prestodb/testing-postgresql-server. If yes, what's the procedure for up-streaming the jars to the respective maven repos? This information will be of great help. Thanks!

@amitsadaphule
Copy link
Contributor Author

I got past the $libdir related issue by updating postgresql binaries, libraries in postgresql-Linux-ppc64le.tar.gz. Next error was: FATAL: could not create lock file "/var/run/postgresql/.s.PGSQL.33158.lock": Permission denied . So, changed ownership of /var/run/postgresql/ to non-root testuser. Post that, I got the following error:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
2020-05-27T12:55:27.295-0500 INFO postgres (PostgreSQL) 9.2.24
2020-05-27T12:55:29.571-0500 INFO postmaster started on port 45687. Waiting up to 10.00s for startup to finish.
2020-05-27T12:55:29.942-0500 INFO PostgreSQL server ready: jdbc:postgresql://localhost:45687/tpch?user=testuser
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23.737 s
[INFO] Finished at: 2020-05-27T17:55:30Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.0:test (default-test) on project presto-postgresql: There are test failures.
[ERROR]
[ERROR] Please refer to /presto-root/prestosrc/presto-postgresql/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] There was an error in the forked process
[ERROR]
[ERROR] Cannot instantiate class com.facebook.presto.plugin.postgresql.TestPostgreSqlIntegrationSmokeTest
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked proces

The reason for this error was unavailability of file_fdw extension which is part of contrib package and needs to be copied to the jar. The line that was causing the exception and resulting in failure is
https://github.com/prestodb/presto/blob/master/presto-postgresql/src/test/java/com/facebook/presto/plugin/postgresql/TestPostgreSqlIntegrationSmokeTest.java#L57

After copying the file_fdw extension and the related file_fdw.so to the jar, I got syntax error at or near "MATERIALIZED". And that resulted in 1 test failure, rest all tests passed. After some debugging, I found that the error was because the expected postgresql version for tests is 9.6.x and the one that I installed/copied binaries for was 9.2.x. So, I copied 9.6.x to the jar as:

mv ~/.m2/repository/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar .
jar xf testing-postgresql-server-9.6.3-4.jar
rm -rf testing-postgresql-server-9.6.3-4.jar
mkdir postgresql-Linux-ppc64le
tar xzf postgresql-Linux-amd64.tar.gz -C postgresql-Linux-ppc64le
cd postgresql-Linux-ppc64le
rm -rf bin/*
rm -rf lib/*
rm -rf share/*
cp /usr/pgsql-9.6/bin/pg_ctl bin/
cp /usr/pgsql-9.6/bin/postgres bin/
cp /usr/pgsql-9.6/bin/initdb bin/
cp -r /usr/pgsql-9.6/lib/* lib/
cp -r /usr/pgsql-9.6/share/* share/
tar czf postgresql-Linux-ppc64le.tar.gz bin lib share
mv postgresql-Linux-ppc64le.tar.gz ../
cd ..
jar cf testing-postgresql-server-9.6.3-4.jar com META-INF postgresql-Linux-amd64.tar.gz postgresql-Linux-ppc64le.tar.gz postgresql-Mac_OS_X-x86_64.tar.gz
cp testing-postgresql-server-9.6.3-4.jar ~/.m2/repository/com/facebook/airlift/testing-postgresql-server/9.6.3-4/

This resolved all test failures.

@amitsadaphule
Copy link
Contributor Author

The only info I need now is:

Whether prestodb community handles up-streaming of the necessary jars to maven repos. e.g. https://repo.maven.apache.org/maven2/com/facebook/airlift/testing-postgresql-server/9.6.3-4/testing-postgresql-server-9.6.3-4.jar using https://github.com/prestodb/testing-postgresql-server. If yes, what's the procedure for up-streaming the jars to the respective maven repos? Thanks!

@rschlussel
Copy link
Contributor

Can you explain what you mean by upstreaming the jars? When we release, we deploy the new versions to maven central.

@amitsadaphule
Copy link
Contributor Author

amitsadaphule commented Jun 5, 2020

Thanks for the response @rschlussel ! By upstreaming, I meant pushing/deploying the generated jars to respective maven repositories. Based on your answer, I'm guessing that I just need to raise PR to https://github.com/prestodb/testing-postgresql-server in order to get the ppc64le binaries/libraries packed in the jar and once you merge+release+deploy, I'll need to raise a PR to https://github.com/prestodb/presto with updated testing-postgresql-server dependency. Is that correct understanding?

Also, could you please confirm if the following are correct repositories to be followed for mysql, redis, hive related build/tests:
https://github.com/prestodb/testing-mysql-server
https://github.com/kstyrc/embedded-redis
https://github.com/prestodb/presto-hadoop-apache2.git

I needed to make similar changes as postgresql to testing-mysql-server-5-0.6.jar, embedded-redis-0.6.jar, hadoop-apache2-2.7.4-7.jar, snappy-java-1.1.7.1.jar.

@amitsadaphule
Copy link
Contributor Author

Cc @gerrith3 @seth-priya

@rschlussel
Copy link
Contributor

yes, that all looks correct, and we can create new releases for any of the prestodb repos

@amitsadaphule
Copy link
Contributor Author

Thanks @rschlussel for the confirmation! I'll raise PRs on the respective repos then.

@amitsadaphule
Copy link
Contributor Author

amitsadaphule commented Jun 9, 2020

@rschlussel I have a question regarding the cross OS viability of the jars generated by these repos. e.g. hadoop-apache2-2.7.4-7.jar which gets downloaded from central maven repo has libhadoop.so, libsnappy.so in nativelib/Linux-ppc64le/ directory. But since those are not compatible with RHEL 7.6 ppc64le (maybe compatible with ubuntu), it resulted in an error java.lang.RuntimeException: failed to load Hadoop native library when I tried to run tests for presto-hive, etc. and when I tried executing the presto-server which was built.

So, my question is, is it OK to focus on a specific OS for ppc64le similar to what's done for x86_64 here?

@amitsadaphule
Copy link
Contributor Author

Also, one correction, for future references, the repo that is currently being used for embedded-redis is https://github.com/gbrd/embedded-redis. There was a plan to switch to https://github.com/kstyrc/embedded-redis. But that didn't see completion. Ref: kstyrc/embedded-redis#71

@rschlussel
Copy link
Contributor

@rschlussel I have a question regarding the cross OS viability of the jars generated by these repos. e.g. hadoop-apache2-2.7.4-7.jar which gets downloaded from central maven repo has libhadoop.so, libsnappy.so in nativelib/Linux-ppc64le/ directory. But since those are not compatible with RHEL 7.6 ppc64le (maybe compatible with ubuntu), it resulted in an error java.lang.RuntimeException: failed to load Hadoop native library when I tried to run tests for presto-hive, etc. and when I tried executing the presto-server which was built.

So, my question is, is it OK to focus on a specific OS for ppc64le similar to what's done for x86_64 here?

yeah, if the jars are not compatible across different oses , it's fine to focus on a particular os

@amitsadaphule
Copy link
Contributor Author

Thanks @rschlussel for the feedback!

@amitsadaphule
Copy link
Contributor Author

@rschlussel I tried to raise PR on testing-postgresql-server repo (prestodb/testing-postgresql-server#2) yesterday. There was no mention of a CLA in the repo or old PR, but an EASYCLA check popped up in the PR. Could you please help me with the following info?

  • Is this CLA specific to the testing-postgresql-server repo or is it applicable across https://github.com/prestodb?
  • Does this point to a different CLA link internally?

@rschlussel
Copy link
Contributor

@amitsadaphule it's the same cla across prestodb. It's possible old prs are from before the presto foundation was formed, so they didn't have the cla configured. The link should take you to a page where you select whether you are contributing on behalf of an individual or corporation, and then take you to the appropriate page to sign. Once it's been signed, you'll need to push an update to your pr so the check runs again (unfortunately it won't detect the change automatically).

@amitsadaphule
Copy link
Contributor Author

OK. Thanks for the feedback! I'll raise another PR and follow your instructions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants