HDDS-1333. OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security classes #653
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…compatible security classes. Contributed by Elek, Marton.
|
There are many unrelated changes in the PR? Can you rebase and update? |
|
This is a branch for 0.4.0 and I created the PR accidentally to trunk. I fixed it now. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
ajayydv
reviewed
Mar 30, 2019
| <resources> | ||
| <resource> | ||
| <directory>src/main/compose</directory> | ||
| <filtering>true</filtering> |
There was a problem hiding this comment.
Are filtering any files from compose dir?
There was a problem hiding this comment.
Yes, I think it's better to filter all the docker-compose and docker-config files.
| ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION | ||
| command: ["/opt/hadoop/bin/ozone","scm"] | ||
| hadoop3: | ||
| hadoop32: |
There was a problem hiding this comment.
Shall we separate the hadoop 2 and 3 compose files first class (i.e create two separate compose dirs for them)?
There was a problem hiding this comment.
Can do, if you prefer it, but I can't see any advantage. It would duplicate the work to maintain these docker-compose file. What I would do instead of this (long-term) is to create a parameter (.env file) for hadoop version and parameterize the test. But it will be way more slower (instead of starting one ozone cluster we need to start it again and again for each version)
There was a problem hiding this comment.
One advantage i can think of is to have separate test suites for hadoop 2 and 3. With hadoop 2 and 3 being first class we can change there configs independently in future. However i am ok with this approch for current patch. We can split it later as desired.
| *** Keywords *** | ||
|
|
||
| Test hadoop dfs | ||
| [arguments] ${imagename} |
There was a problem hiding this comment.
since each time imagename is different shall we move this to variable section? (i.e keynames will still not collide)
There was a problem hiding this comment.
You are right. It's not an imagename, in fact this is just any prefix of the name. Will be fixed in the last commit.
| Should not contain ${result} Failed | ||
| Should contain ${result} Creating Volume: ${volume} | ||
| Create bucket | ||
| Execute ozone sh bucket create /${volume}/${bucket} |
There was a problem hiding this comment.
After this shall we check if bucket creation is successful by listing it?
There was a problem hiding this comment.
Thanks. Added an additional bucket info call.
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java
Show resolved
Hide resolved
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java
Show resolved
Hide resolved
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java
Show resolved
Hide resolved
| /** | ||
| * Adapter to convert OzoneKey to a safe and simple Key implementation. | ||
| */ | ||
| public static class IteratorAdapter implements Iterator<BasicKeyInfo> { |
There was a problem hiding this comment.
Is OzoneKeyIterator more relevant?
There was a problem hiding this comment.
Not sure what is the question. Can you please elaborate.
There was a problem hiding this comment.
I mean shall we rename IteratorAdapter to OzoneKeyIterator.
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java
Show resolved
Hide resolved
xiaoyuyao
reviewed
Apr 1, 2019
xiaoyuyao
reviewed
Apr 1, 2019
|
💔 -1 overall
This message was automatically generated. |
xiaoyuyao
reviewed
Apr 2, 2019
| </configuration> | ||
| ``` | ||
|
|
||
| _Note_: You may also use `org.apache.hadoop.fs.ozone.OzoneFileSystem` without the `Basic` prefix. The `Basic` version doesn't support FS statistics and security tokens but can work together with older hadoop versions. |
There was a problem hiding this comment.
bq. "The Basic version doesn't support FS statistics and security tokens but can work together with older hadoop versions."
This is not accurate. If I understand correctly, the BasicOzoneFileSystem also support delegation token APIs but not FS statistics.
There was a problem hiding this comment.
Ok, thanks. It's just because I have limited knowledge about the KeyProviderTokenIssuer. Can I write:
The `Basic` version doesn't support FS statistics and encryption zones but....
|
💔 -1 overall
This message was automatically generated. |
xiaoyuyao
reviewed
Apr 3, 2019
|
💔 -1 overall
This message was automatically generated. |
|
+1 |
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
prateekm let me know if there are other places that logging should also be improved in this patch. Author: Daniel Chen <dchen1@linkedin.com> Author: Daniel Chen <xrchen@uwaterloo.ca> Reviewers: Prateek Maheshwari <pmaheshwari@apache.org> Closes apache#653 from dxichen/add-restore-logging-info
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current ozonefs compatibility layer is broken by: HDDS-1299.
The spark jobs (including hadoop 2.7) can't be executed any more:
{code}
2019-03-25 09:50:08 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProviderTokenIssuer
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:45)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:332)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:715)
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:757)
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:724)
at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.crypto.key.KeyProviderTokenIssuer
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 43 more
{code}
See: https://issues.apache.org/jira/browse/HDDS-1333