merge #2
Merged
merge #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…heckPermission. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
…ounters on IO." This reverts commit e2c7ac7. Change-Id: I5b5a93f5a36cdb0c3d56d1b3f747c318f089de20
This reverts commit 367833c.
…tributed by Yang Yun.
…ror (#1902). Contributed by zhengchenyu.
…ider Contributed by Nicholas Chammas.
…cheduler. Contributed by Gergely Pollak.
…om DecayRpcScheduler. Contributed by Fengnan Li.
…rage policies. Contributed by Siddharth Wagle. Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
…buted by Xiaoqiao He.
…checkPermissionWithContext API. (#1909)
…ax-am-resource percent value. Contributed by Peter Bacsko
…& appendblob" This reverts commit 3612317. Change-Id: Ie0d36f25de0b55a937894f4d9963c495bae0576a
#1914) add unit test, new ITest and then fix the issue: different schema, bucket == skip factored out the underlying logic for unit testing; also moved maybeAddTrailingSlash to S3AUtils (while retaining/forwarnding existing method in S3AFS). tested: london, sole failure is testListingDelete[auth=true](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations) filed HADOOP-16853 Change-Id: I4b8d0024469551eda0ec70b4968cba4abed405ed
…ly. Contributed by xuzq.
…ontributed by Steven Rand.
…-address.$NAMESERVICE`. Contributed by Ctest.
… Contributed by hemanthboyina.
…mblies.version in hadoop-project/pom.xml (see HADOOP-15369)
… invalid queue mapping. Contributed by Gergely Pollak" This reverts commit ac4540d. Reverting because of missing Jira-ID in the commit message.
…s in invalid queue mapping. Contributed by Gergely Pollak
…d collect all distinct blocks. Contributed by hemanthboyina.
Contributed by Sneha Vijayarajan
Contributed by Sneha Vijayarajan
…mapreduce.map.speculative and mapreduce.reduce.speculative are false Contributed by Tarun Parimi.
…educe Active load. Contributed by Xiaoqiao He.
…ed occasionally. Contributed by Xiaoqiao He.
…. Contributed by Andras Bokor.
…cumentation. Contributed by Andras Bokor.
… Contributed by Wei-Chiu Chuang.
…edAddressException) causing clients to abort (#2036) Contributed by Dhiraj Hegde. Signed-off-by: Mingliang Liu <liuml07@apache.org>
…ted by Andras Gyori
…ctory in LogServlet. Contributed by Adam Antal
… queue change. Contributed by Gergely Pollak
Contributed by: Mehakmeet Singh. Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42
…eme. Contributed by Uma Maheswara Rao G.
…r by batch. Contributed by Fengnan Li.
…orkTopology. Contributed by Jim Brennan.
…shutdown twice. Contributed by Wei-Chiu Chuang.
In the docker build image, skip GPG verification when downloading Yetus tarball via yetus-wrapper.
Contributed by Mikhail Pryakhin
…als are unset. (#1441) Contributed by Steve Loughran. Move the loading to deployUnbonded (where they are required) and add a safety check when a new DT is requested
Deegue
pushed a commit
that referenced
this pull request
Jun 4, 2020
Contributed by Steve Loughran. This fixes two problems with S3Guard authoritative mode and the auth directory flags which are stored in DynamoDB. 1. mkdirs was creating dir markers without the auth bit, forcing needless scans on newly created directories and files subsequently added; it was only with the first listStatus call on that directory that the dir would be marked as authoritative -even though it would be complete already. 2. listStatus(path) would reset the authoritative status bit of all child directories even if they were already marked as authoritative. Issue #2 is possibly the most expensive, as any treewalk using listStatus (e.g globfiles) would clear the auth bit for all child directories before listing them. And this would happen every single time... essentially you weren't getting authoritative directory listings. For the curious, that the major bug was actually found during testing -we'd all missed it during reviews. A lesson there: the better the tests the fewer the bugs. Maybe also: something obvious and significant can get by code reviews. modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/BulkOperationState.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/NullMetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardWriteBack.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardFsck.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java Change-Id: Ic3ffda13f2af2430afedd50fd657b595c83e90a7
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.