Skip to content

HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) #1208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Sep 12, 2019
Merged

HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) #1208

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
931137c
added integration tests for violation pairs. minor refactor
Jul 3, 2019
3cfd2c9
added AUTHORITATIVE_DIRECTORY_CONTENT_MISMATCH with test, removed ver…
Aug 7, 2019
4016351
factored out the big switch block from S3GuardFsckViolationHandler. U…
Aug 7, 2019
debe4c3
teardown added to itest
Aug 7, 2019
6565328
CLI test is working in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardT…
Aug 12, 2019
7d32a40
Remove ITestS3GuardDraftingFsck - move to a gist
Aug 12, 2019
fe0f7c4
fix findbugs and checkstyle issues. long line lenght checks are skipp…
Aug 13, 2019
60a25d3
some more comments
Aug 13, 2019
398c5d8
fix treewalk, add some tests, address Steve's comments. Tested in CLI…
Sep 4, 2019
61bbe4f
fix wrong missing versionid handling
Sep 6, 2019
7f904bf
check etag only for files, and not directories
Sep 6, 2019
d4e0c73
fixed based on comments part1 - there will be another part soon
Sep 10, 2019
6378165
ITestS3GuardFsck refactor
Sep 10, 2019
a9e9eb4
MOD_TIME_MISMATCH allowed within range
Sep 10, 2019
633d6cd
added TOMBSTONED_IN_MS_NOT_DELETED_IN_S3, fixed some checkstyle issues
Sep 11, 2019
1dfc3a1
fixed test failures
Sep 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1492,7 +1492,7 @@ public boolean hasMetadataStore() {
* is set for this filesystem.
*/
@VisibleForTesting
boolean hasAuthoritativeMetadataStore() {
public boolean hasAuthoritativeMetadataStore() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now this is done on a per path basis is this probe adequate?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is used in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck#testIAuthoritativeDirectoryContentMismatch and I think that's the easiest to check in runtime. But I can use something else like getting it from the fs's config.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If someone has configured the test path to be authoritative, then even if the store says non-auth you'll get auth-store-behaviours. Best to either unset the per-bucket/global auth mode settings in test config creation or check the auth mode of the actual test path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're only using this for tests, so I'm not as worried as I was. +1 for this change

return hasMetadataStore() && allowAuthoritativeMetadataStore;
}

Expand Down
Loading