[KYLIN-4864] Enable running Kylin tests on ARM64 platform #1558

liusheng · 2021-01-26T08:28:49Z

Proposed changes

This change modify the Travis CI configurations to enable the Kylin testing on ARM64 server.

A special rocksdbjni version for aarch64 has been released by rocksdb community. [1][2]

This patch bumps the rocksdbjni version to 5.18.4 to make the Storm can be built in aarch64.

[1] https://github.com/facebook/rocksdb/releases/tag/v5.18.4
[2] facebook/rocksdb#6250

Types of changes

What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have create an issue on Kylin's jira, and have described the bug/feature there in detail
Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If this change need a document change, I will prepare another pr against the document branch
Any dependent changes have been merged

Further comments

If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

coveralls · 2021-01-26T12:16:40Z

Pull Request Test Coverage Report for Build 6970

0 of 0 changed or added relevant lines in 0 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.002%) to 28.174%

Files with Coverage Reduction	New Missed Lines	%
tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java	1	81.58%
core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java	2	68.46%

Totals
Change from base Build 6963:	-0.002%
Covered Lines:	26708
Relevant Lines:	94798

💛 - Coveralls

liusheng · 2021-01-27T07:44:21Z

let's wait for #1557 merged firstly

codecov-io · 2021-03-09T07:09:32Z

Codecov Report

Merging #1558 (57546e1) into master (0250081) will decrease coverage by 0.00%.
The diff coverage is 18.18%.

@@             Coverage Diff              @@
##             master    #1558      +/-   ##
============================================
- Coverage     25.42%   25.42%   -0.01%     
+ Complexity     6765     6762       -3     
============================================
  Files          1508     1508              
  Lines         93918    93924       +6     
  Branches      13158    13160       +2     
============================================
- Hits          23877    23876       -1     
- Misses        67662    67667       +5     
- Partials       2379     2381       +2

Impacted Files	Coverage Δ	Complexity Δ
.../java/org/apache/kylin/common/KylinConfigBase.java	`12.46% <0.00%> (ø)`	`51.00 <0.00> (ø)`
...che/kylin/storage/gtrecord/CubeTupleConverter.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...e/kylin/rest/controller/StreamingV2Controller.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...apache/kylin/rest/metrics/QueryMetrics2Facade.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
.../apache/kylin/rest/metrics/QueryMetricsFacade.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...apache/kylin/storage/hbase/HBaseResourceStore.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...lin/stream/core/query/StreamingTupleConverter.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...lin/stream/source/kafka/TimedJsonStreamParser.java	`67.39% <0.00%> (ø)`	`12.00 <0.00> (ø)`
...stream/core/query/MultiThreadsResultCollector.java	`63.91% <66.66%> (+0.37%)`	`4.00 <0.00> (ø)`
...in/java/org/apache/kylin/query/util/QueryUtil.java	`60.00% <100.00%> (-0.40%)`	`25.00 <1.00> (-1.00)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 448d38a...57546e1. Read the comment docs.

This change modify the Travis CI configurations to enable the Kylin testing on ARM64 server.

liusheng · 2021-03-12T06:11:58Z

Hi @hit-lacus It looks the ARM CI job in Travis works OK (see above check status), all the tests can pass in ARM64, could you please help review this ? Thank you.

liusheng · 2021-03-15T01:53:41Z

Hi @zhangayqian @zzcclp Sorry to bother you, could you please help to take a look this PR ?

liusheng · 2021-03-19T01:17:40Z

Hi, could any can help to review this ?

liusheng · 2021-03-22T07:45:11Z

Hi @hit-lacus @zhangayqian Could you please help to review this PR? Thank you.

shaofengshi · 2021-03-30T05:55:47Z

hi sheng, I see it includes a version upgrade for rocksdb; it is a must-have for this JIRA? @liusheng

liusheng · 2021-03-30T06:18:18Z

Hi @shaofengshi , yes, I previously made the version upgrade for rocksdbjni in #1557, now I have merged the small change in this PR, I just updated the reason of this upgrade in this PR.

hit-lacus · 2021-03-30T07:01:41Z

Hi, from the issue title Support building and testing Kylin on ARM64 architecture platform, is aims to support build and run Kylin on ARM64 architecture platform.

Does this patch ensure user can build new binary and run smoothly on ARM64 platfrom? Could you provided a test report for verification ?

Besides, I was concerned by if this upgrade will break current functions.

liusheng · 2021-03-30T07:51:00Z

Hi @hit-lacus ,
Thanks for your reply. yes, we are aim to promoting some popular open source projects support running on ARM platform. for Kylin, I have verified the building, and runing all tests(seems also includes integration tests), if needed, I will verified the deployment and basic functionalities of Kylin on ARM64 server ans post testing reports.

For the upgrade of rocksdbjni from version 5.9.2~5.18.4, I wil also try to check if there is side effect, rocksdb multi-arch support and version upgrade has already be performed on Kafka/Storm/Flink[1][2][3].

The Rocksdb adds Arm64 support [4] since version 6.4.6, and also backports all Arm64
related commits to 5.18.4 and release a all platforms support version.

So, from multi-arch support view, the better rocksdb version is the version since
v6.4.6, or 5.X version is v5.18.4.

[1] https://issues.apache.org/jira/browse/STORM-3599
[2] apache/kafka#8284
[3] https://issues.apache.org/jira/browse/FLINK-13598
[4] facebook/rocksdb#6250

martin-g · 2021-03-30T08:25:28Z

.travis.yml

@@ -40,6 +46,7 @@ before_script:
  - echo "MAVEN_OPTS='-Xms1024m -Xmx3072m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m'" > ~/.mavenrc
  - sed -i 's/log4j.logger.org.apache.kylin=INFO/log4j.logger.org.apache.kylin=WARN/g' build/conf/kylin-server-log4j.properties
  - sed -i 's/log4j.logger.org.apache.kylin=INFO/log4j.logger.org.apache.kylin=WARN/g' build/conf/kylin-tools-log4j.properties
+  - if [[ $(uname -m) == 'aarch64' ]];then export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64";fi


This line could be simplified to:
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-${TRAVIS_CPU_ARCH}"

hit-lacus · 2021-03-30T08:44:57Z

One of your commit 's commit message is "Use openjdk11 for ARM64 test", did your patch use openjdk11 for test? I didn't find it.

hit-lacus · 2021-03-30T08:53:59Z

seems also includes integration tests

Currently, integration test need a HDP 2.4 sandbox for Hadoop setup. Please check http://kylin.apache.org/development/howto_test.html for the detail of how to run integration test.

Besides, I wish you can test deployment and basic functionalities of Kylin on ARM64 server, and share testing reports to us. Sadly to say, I do not know too much on ARM64 technical and do not have a ARM64 server for verification.

For how to run manual test, please check this link https://cwiki.apache.org/confluence/display/KYLIN/Regression+Testing+for+Kylin+Release .

martin-g · 2021-06-11T12:59:31Z

I will do the testing and report back !

martin-g · 2021-06-15T10:59:21Z

@hit-lacus It seems Hortonworks resources are no more available. Do you have a Howto with a replacement of HDP ?

hit-lacus · 2021-06-17T01:45:27Z

@hit-lacus It seems Hortonworks resources are no more available. Do you have a Howto with a replacement of HDP ?

Sorry for late reply, I think docker image is a good replacement of HDP, would you like have a try? https://github.com/apache/kylin/tree/master/docker

If you prefer HDP sandbox, here is the download link : https://issues.apache.org/jira/browse/KYLIN-4040?focusedCommentId=16861673&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16861673 .

martin-g · 2021-06-22T11:21:36Z

@hit-lacus I was able to run the Docker image on Linux ARM64 with these changes: https://github.com/apache/kylin/compare/master...martin-g:KYLIN-4864-add-support-for-aarch64-to-Dockerfiles?expand=1

To build and run it I use:

#!/usr/bin/env bash


if test "$1" = "rebuild"; then
  echo -e "\n\n-- Building the Hadoop image ... \n\n"
  docker build --build-arg BASE_IMAGE=arm64v8/centos:7.9.2009 --build-arg ARCH=aarch64 --build-arg JDK_ARCH=aarch64 -t kylin-hadoop:2.7 -f Dockerfile_hadoop .

  echo -e "\n\n-- Building the Kylin image ... \n\n"
  docker build --build-arg BASE_IMAGE=kylin-hadoop:2.7 -t kylin:2.7 -f Dockerfile .
fi


echo -e "\n\n-- Starting Kylin ... \n\n"
docker run --rm -it --name kylin \
  --memory=8G \
  -p 7070:7070 \
  -p 8088:8088 \
  -p 50070:50070 \
  -p 8032:8032 \
  -p 8042:8042 \
  -p 16010:16010 \
  --user root \
  kylin:2.7

I had to upgrade MySQL to 8.x because there was no aarch64 binaries at http://repo.mysql.com/yum/mysql-5.7-community/el/7/

I've also updated Kylin, Livy, JDK and Maven to their latest versions.

I've tested it with the steps described at http://kylin.apache.org/docs/tutorial/kylin_sample.html but it got stuck at 90.91% . The logs just repeat the following again and again:

==> logs/kylin.log <==
2021-06-22 11:12:42,935 INFO  [BadQueryDetector] service.BadQueryDetector:148 : Detect bad query.
2021-06-22 11:12:45,832 INFO  [FetcherRunner 1102308739-43] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others

==> logs/streaming_coordinator.log <==
2021-06-22 11:12:45,832 INFO  [FetcherRunner 1102308739-43] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others

==> logs/kylin.log <==
2021-06-22 11:13:15,832 INFO  [FetcherRunner 1102308739-43] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others

==> logs/streaming_coordinator.log <==
2021-06-22 11:13:15,832 INFO  [FetcherRunner 1102308739-43] threadpool.DefaultFetcherRunner:117 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 0 already succeed, 0 error, 0 discarded, 0 others

==> logs/kylin.log <==

I did not face any problems related to RocksDB ! Only the unit tests fail with 5.9.2 and pass with 5.18.4!

Now I am trying to test with 3.1.3-SNAPSHOT but the build fails at kylin-it module due to :

java.lang.IllegalArgumentException: java.net.UnknownHostException: sandbox.hortonworks.com
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:355)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:284)
	at org.apache.kylin.dict.ITGlobalDictionaryBuilderTest$SharedBuilderThread.run(ITGlobalDictionaryBuilderTest.java:133)

I cannot find what build step is supposed to build the **-bin.tar.gz for

kylin/docker/Dockerfile_dev

Line 25 in efedf07

COPY apache-kylin-$KYLIN_VERSION-bin.tar.gz /home/admin/

martin-g · 2021-06-22T13:13:34Z

I've created apache-kylin-3.1.3-SNAPSHOT-bin.tar.gz by using ./build/* and ./tool/target/kylin-tool-3.1.3-SNAPSHOT.jar. But when starting Kylin it fails with Please set kylin.env.hdfs-working-dir in kylin.properties. Neither Dockerfile_dev nor Dockerfile sets kylin.env.hdfs-working-dir. What should be its recommended value ?

I've added kylin.env.hdfs-working-dir=hdfs://localhost:9000/kylin/work to conf/kylin.properties but it still fails the same way. I've doubled checked by docker exec .. bash and cat apache-kylin**/conf/kylin.properties

martin-g · 2021-07-05T11:55:57Z

@hit-lacus Ping!

hit-lacus

LGTM

liusheng · 2021-07-27T11:43:43Z

Thanks for @martin-g help and @hit-lacus review!

martin-g · 2021-11-03T08:02:22Z

The changes introduced with this PR have been lost! There is no arm64 in https://github.com/apache/kylin/blob/main/.travis.yml and its git history.

@hit-lacus Were there any Git filtering operations on the repo ? I see that master branch has been renamed to main but I doubt this is the reason.

* [KYLIN-4864] Enable running Kylin tests on ARM64 platform This change modify the Travis CI configurations to enable the Kylin testing on ARM64 server. * Use openjdk11 for ARM64 test

hit-lacus added the Work In Progress This patch is incomplete, maybe author need update this patch ? label Feb 7, 2021

liusheng force-pushed the KYLIN-4864 branch 6 times, most recently from 8aecdb5 to 9145147 Compare March 9, 2021 06:39

liusheng added 2 commits March 12, 2021 11:46

[KYLIN-4864] Enable running Kylin tests on ARM64 platform

1cf2cc1

This change modify the Travis CI configurations to enable the Kylin testing on ARM64 server.

Use openjdk11 for ARM64 test

57546e1

liusheng force-pushed the KYLIN-4864 branch from 9145147 to 57546e1 Compare March 12, 2021 03:46

liusheng mentioned this pull request Mar 12, 2021

[KYLIN-4886] Bump the rocksdbjni dependency version to 5.18.4 #1557

Closed

10 tasks

zhangayqian closed this Mar 15, 2021

zhangayqian reopened this Mar 15, 2021

martin-g reviewed Mar 30, 2021

View reviewed changes

martin-g approved these changes Mar 30, 2021

View reviewed changes

hit-lacus approved these changes Jul 27, 2021

View reviewed changes

hit-lacus merged commit 6f5c31b into apache:master Jul 27, 2021

liusheng deleted the KYLIN-4864 branch July 27, 2021 11:43

martin-g mentioned this pull request Jul 27, 2021

KYLIN-4864 Fix .travis.yml syntax #1700

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KYLIN-4864] Enable running Kylin tests on ARM64 platform #1558

[KYLIN-4864] Enable running Kylin tests on ARM64 platform #1558

liusheng commented Jan 26, 2021 •

edited

Loading

coveralls commented Jan 26, 2021 •

edited

Loading

liusheng commented Jan 27, 2021

codecov-io commented Mar 9, 2021 •

edited

Loading

liusheng commented Mar 12, 2021

liusheng commented Mar 15, 2021

liusheng commented Mar 19, 2021

liusheng commented Mar 22, 2021

shaofengshi commented Mar 30, 2021

liusheng commented Mar 30, 2021

hit-lacus commented Mar 30, 2021 •

edited

Loading

liusheng commented Mar 30, 2021 •

edited

Loading

martin-g Mar 30, 2021

hit-lacus commented Mar 30, 2021

hit-lacus commented Mar 30, 2021 •

edited

Loading

martin-g commented Jun 11, 2021

martin-g commented Jun 15, 2021

hit-lacus commented Jun 17, 2021 •

edited

Loading

martin-g commented Jun 22, 2021 •

edited

Loading

martin-g commented Jun 22, 2021

martin-g commented Jul 5, 2021

hit-lacus left a comment

liusheng commented Jul 27, 2021

martin-g commented Nov 3, 2021

[KYLIN-4864] Enable running Kylin tests on ARM64 platform #1558

[KYLIN-4864] Enable running Kylin tests on ARM64 platform #1558

Conversation

liusheng commented Jan 26, 2021 • edited Loading

Proposed changes

Types of changes

Checklist

Further comments

coveralls commented Jan 26, 2021 • edited Loading

Pull Request Test Coverage Report for Build 6970

💛 - Coveralls

liusheng commented Jan 27, 2021

codecov-io commented Mar 9, 2021 • edited Loading

Codecov Report

liusheng commented Mar 12, 2021

liusheng commented Mar 15, 2021

liusheng commented Mar 19, 2021

liusheng commented Mar 22, 2021

shaofengshi commented Mar 30, 2021

liusheng commented Mar 30, 2021

hit-lacus commented Mar 30, 2021 • edited Loading

liusheng commented Mar 30, 2021 • edited Loading

martin-g Mar 30, 2021

Choose a reason for hiding this comment

hit-lacus commented Mar 30, 2021

hit-lacus commented Mar 30, 2021 • edited Loading

martin-g commented Jun 11, 2021

martin-g commented Jun 15, 2021

hit-lacus commented Jun 17, 2021 • edited Loading

martin-g commented Jun 22, 2021 • edited Loading

martin-g commented Jun 22, 2021

martin-g commented Jul 5, 2021

hit-lacus left a comment

Choose a reason for hiding this comment

liusheng commented Jul 27, 2021

martin-g commented Nov 3, 2021

liusheng commented Jan 26, 2021 •

edited

Loading

coveralls commented Jan 26, 2021 •

edited

Loading

codecov-io commented Mar 9, 2021 •

edited

Loading

hit-lacus commented Mar 30, 2021 •

edited

Loading

liusheng commented Mar 30, 2021 •

edited

Loading

hit-lacus commented Mar 30, 2021 •

edited

Loading

hit-lacus commented Jun 17, 2021 •

edited

Loading

martin-g commented Jun 22, 2021 •

edited

Loading