HADOOP-16684. Bucket info to add more on authoritative mode #1698

steveloughran · 2019-11-06T18:37:40Z

-it also adds a lot on committers based on the chosen committer.
-layout tuning

Location: eu-west-1
Filesystem s3a://hwdev-steve-ireland-new is using S3Guard with store DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new}
Authoritative Metadata Store: fs.s3a.metadatastore.authoritative=false
Authoritative Path: fs.s3a.authoritative.path=/tables
Qualified Authoritative Paths:
	s3a://hwdev-steve-ireland-new/tables/

	Metadata time to live: fs.s3a.metadatastore.metadata.ttl=900000 milliseconds
Metadata Store Diagnostics:
	ARN=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
	billing-mode=per-request
	description=S3Guard metadata store in DynamoDB
	name=hwdev-steve-ireland-new
	persist.authoritative.bit=true
	read-capacity=0
	region=eu-west-1
	retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=250 MILLISECONDS)
	size=32261
	status=ACTIVE
	table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-ireland-new,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Oct 02 18:51:07 BST 2019,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes: 32261,ItemCount: 219,TableArn: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new,TableId: 9b81d364-bef0-4ce0-afb8-c8115c473903,BillingModeSummary: {BillingMode: PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Wed Oct 02 18:51:07 BST 2019},}
	write-capacity=0

S3A Client
	Signing Algorithm: fs.s3a.signing-algorithm=(unset)
	Endpoint: fs.s3a.endpoint=s3-eu-west-1.amazonaws.com
	Encryption: fs.s3a.server-side-encryption-algorithm=SSE-KMS
	Input seek policy: fs.s3a.experimental.input.fadvise=normal
	Change Detection Source: fs.s3a.change.detection.source=etag
	Change Detection Mode: fs.s3a.change.detection.mode=server

S3A Committers
	The "magic" committer is supported in the filesystem
	S3A Committer factory class: mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
	S3A Committer name: fs.s3a.committer.name=directory
	Cluster filesystem staging directory: fs.s3a.committer.staging.tmp.path=tmp/staging
	Local filesystem buffer directory: fs.s3a.buffer.dir=/tmp/hadoop-stevel/s3a
	File conflict resolution: fs.s3a.committer.staging.conflict-mode=append

Security
	Delegation Support enabled: token kind = S3ADelegationToken/Session
	Hadoop security mode: SIMPLE

-it also adds a lot on committers and tunes layout Change-Id: I3c048799063a14e1ed13fadc9507c972a9f05259

hadoop-yetus · 2019-11-06T19:36:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	39	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1095	trunk passed
+1	compile	35	trunk passed
+1	checkstyle	27	trunk passed
+1	mvnsite	40	trunk passed
+1	shadedclient	808	branch has no errors when building and testing our client artifacts.
+1	javadoc	29	trunk passed
0	spotbugs	59	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	58	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	36	the patch passed
+1	compile	28	the patch passed
+1	javac	28	the patch passed
-0	checkstyle	20	hadoop-tools/hadoop-aws: The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5)
+1	mvnsite	33	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	809	patch has no errors when building and testing our client artifacts.
+1	javadoc	27	the patch passed
-1	findbugs	67	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
-1	unit	85	hadoop-aws in the patch failed.
+1	asflicense	31	The patch does not generate ASF License warnings.
		3359

Reason	Tests
FindBugs	module:hadoop-tools/hadoop-aws
	Switch statement found in org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(String[], PrintStream) where one case falls through to the next case At S3GuardTool.java:PrintStream) where one case falls through to the next case At S3GuardTool.java:[lines 1299-1306]
Failed junit tests	hadoop.fs.s3a.s3guard.TestPathMetadataDynamoDBTranslation
	hadoop.fs.s3a.s3guard.TestNullMetadataStore
	hadoop.fs.s3a.TestS3AUnbuffer
	hadoop.fs.s3a.commit.staging.TestStagingCommitter

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/2/artifact/out/Dockerfile
GITHUB PR	#1698
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2bf8e9d7fe2c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `dd90025`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/2/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/2/testReport/
Max. process+thread count	451 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-11-06T20:05:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1891	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1104	trunk passed
+1	compile	38	trunk passed
+1	checkstyle	24	trunk passed
+1	mvnsite	42	trunk passed
+1	shadedclient	797	branch has no errors when building and testing our client artifacts.
+1	javadoc	31	trunk passed
0	spotbugs	61	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	59	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	35	the patch passed
+1	compile	28	the patch passed
+1	javac	28	the patch passed
-0	checkstyle	22	hadoop-tools/hadoop-aws: The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5)
+1	mvnsite	33	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	806	patch has no errors when building and testing our client artifacts.
+1	javadoc	27	the patch passed
-1	findbugs	65	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1	unit	86	hadoop-aws in the patch passed.
+1	asflicense	32	The patch does not generate ASF License warnings.
		5224

Reason	Tests
FindBugs	module:hadoop-tools/hadoop-aws
	Switch statement found in org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(String[], PrintStream) where one case falls through to the next case At S3GuardTool.java:PrintStream) where one case falls through to the next case At S3GuardTool.java:[lines 1299-1306]

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/1/artifact/out/Dockerfile
GITHUB PR	#1698
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux a2b5c4571c43 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `dd90025`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/1/testReport/
Max. process+thread count	420 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-11-08T13:51:08Z

checkstyle

./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:1202:    public int run(String[] args, PrintStream out):5: Method length is 174 lines (max allowed is 150). [MethodLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:1249:        if (!authoritativePaths.isEmpty()): 'if' construct must use '{}'s. [NeedBraces]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:1300:            "The 'staging' committer is used -prefer the 'directory' committer");: Line is longer than 80 characters (found 81). [LineLength]

the if() clause may be a bug

Change-Id: Ib3313100e335e1a59eaaed631be7c702cb48431c

Change-Id: I7a8a3b36007a99d61b98501e27c231e8a70d970a

hadoop-yetus · 2019-11-08T15:12:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	38	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1080	trunk passed
+1	compile	36	trunk passed
+1	checkstyle	28	trunk passed
+1	mvnsite	40	trunk passed
+1	shadedclient	812	branch has no errors when building and testing our client artifacts.
+1	javadoc	30	trunk passed
0	spotbugs	59	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	58	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	34	the patch passed
+1	compile	28	the patch passed
+1	javac	28	the patch passed
-0	checkstyle	21	hadoop-tools/hadoop-aws: The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5)
+1	mvnsite	32	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	792	patch has no errors when building and testing our client artifacts.
+1	javadoc	26	the patch passed
-1	findbugs	66	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1	unit	85	hadoop-aws in the patch passed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		3342

Reason	Tests
FindBugs	module:hadoop-tools/hadoop-aws
	Switch statement found in org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(String[], PrintStream) where one case falls through to the next case At S3GuardTool.java:PrintStream) where one case falls through to the next case At S3GuardTool.java:[lines 1299-1306]

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/3/artifact/out/Dockerfile
GITHUB PR	#1698
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ef0e1176d33e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `42fc888`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/3/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/3/testReport/
Max. process+thread count	411 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-11-08T15:25:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	39	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1152	trunk passed
+1	compile	38	trunk passed
+1	checkstyle	28	trunk passed
+1	mvnsite	41	trunk passed
+1	shadedclient	816	branch has no errors when building and testing our client artifacts.
+1	javadoc	27	trunk passed
0	spotbugs	62	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	59	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	36	the patch passed
+1	compile	32	the patch passed
+1	javac	32	the patch passed
-0	checkstyle	19	hadoop-tools/hadoop-aws: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1	mvnsite	32	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	786	patch has no errors when building and testing our client artifacts.
+1	javadoc	24	the patch passed
-1	findbugs	67	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1	unit	85	hadoop-aws in the patch passed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		3406

Reason	Tests
FindBugs	module:hadoop-tools/hadoop-aws
	Switch statement found in org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(String[], PrintStream) where one case falls through to the next case At S3GuardTool.java:PrintStream) where one case falls through to the next case At S3GuardTool.java:[lines 1300-1307]

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/4/artifact/out/Dockerfile
GITHUB PR	#1698
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 12384b26c5db 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `42fc888`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/4/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/4/testReport/
Max. process+thread count	451 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1698/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-11-08T18:53:42Z

pulled into the bigger #1707 PR

HADOOP-15619. Bucket info to add more on authoritative mode

9c128e8

-it also adds a lot on committers and tunes layout Change-Id: I3c048799063a14e1ed13fadc9507c972a9f05259

steveloughran changed the title ~~HADOOP-15619. Bucket info to add more on authoritative mode~~ HADOOP-16684. Bucket info to add more on authoritative mode Nov 8, 2019

HADOOP-16684. tweak TTL entry

c3de1ff

Change-Id: Ib3313100e335e1a59eaaed631be7c702cb48431c

steveloughran force-pushed the s3/HADOOP-15619-bucket-info-auth branch from a0fc5d2 to c3de1ff Compare November 8, 2019 14:16

HADOOP-16664 address checkstyle where appropriate

372b826

Change-Id: I7a8a3b36007a99d61b98501e27c231e8a70d970a

steveloughran mentioned this pull request Nov 8, 2019

HADOOP-16697. Tune/audit S3A authoritative mode #1707

Closed

steveloughran closed this Nov 11, 2019

steveloughran deleted the s3/HADOOP-15619-bucket-info-auth branch October 15, 2021 19:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16684. Bucket info to add more on authoritative mode #1698

HADOOP-16684. Bucket info to add more on authoritative mode #1698

Uh oh!

steveloughran commented Nov 6, 2019

Uh oh!

hadoop-yetus commented Nov 6, 2019

Uh oh!

hadoop-yetus commented Nov 6, 2019

Uh oh!

steveloughran commented Nov 8, 2019

Uh oh!

hadoop-yetus commented Nov 8, 2019

Uh oh!

hadoop-yetus commented Nov 8, 2019

Uh oh!

steveloughran commented Nov 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HADOOP-16684. Bucket info to add more on authoritative mode #1698

HADOOP-16684. Bucket info to add more on authoritative mode #1698

Uh oh!

Conversation

steveloughran commented Nov 6, 2019

Uh oh!

hadoop-yetus commented Nov 6, 2019

Uh oh!

hadoop-yetus commented Nov 6, 2019

Uh oh!

steveloughran commented Nov 8, 2019

Uh oh!

hadoop-yetus commented Nov 8, 2019

Uh oh!

hadoop-yetus commented Nov 8, 2019

Uh oh!

steveloughran commented Nov 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants