Output directories are sometimes out of order when using remote execution

[jjudd]

Description of the problem:

When using remote execution (buildfarm) output directories are sometimes out of order. The spec requires them to be lexicographically sorted.

I have a PR that fixes this issue and will link it after creating this issue.

When using remote executing, a parent node is built from action inputs in TreeNodeRepository. The files are sorted by path in RemoteSpawnRunner. These paths are then broken down into segments and a tree structure is created from the segments.

Problem is, the segments at each level of the tree structure are not sorted before they are added to the parent node. This can result in an unordered directory tree.

For example, the sort order of this list of files

/foo/bar-client/bar-client_ijar.jar
/foo/bar/bar_ijar.jar

is maintained when it becomes a tree structure

foo ->
    bar-client ->
        bar-client_ijar.jar
    bar
        bar_ijar.jar

which is out of order.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Create a Bazel project with two packages

/foo/bar-client
/foo/bar

that produce outputs named in a similar fashion. For example, two Java targets that create:

/foo/bar-client/bar-client_ijar.jar
/foo/bar/bar_ijar.jar

Then build those targets using buildfarm or another remote executor. You will receive an error like the following:

Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: The files in an input `Directory` are not correctly sorted by `name`.
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:221)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:202)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:131)
	at com.google.devtools.remoteexecution.v1test.ExecutionGrpc$ExecutionBlockingStub.execute(ExecutionGrpc.java:370)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$0(GrpcRemoteExecutor.java:133)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:220)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:101)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:133)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:220)
	... 21 more

What operating system are you running Bazel on?

Ubuntu 16.04 x86_64

What's the output of bazel info release?

$ bazel info release
.........
release 0.12.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

james@james:~/opensource/bazel$ git remote get-url origin ; git rev-parse master ; git rev-parse HEAD
git@github.com:bazelbuild/bazel.git
ca76428fe83f73f2428d4fdc1382884108de4fed
ca76428fe83f73f2428d4fdc1382884108de4fed

[jjudd]

Here is a PR that fixes this issue: #5110

[iirina]

Thanks for the PR! I assigned you since you sent a fix.

[driveraid]

Is this fixed? I still see the issue on Bazel 0.13.0 with bazel buildfarm rev 24a5d7e29.

[jjudd]

The fix for this was merged a few hours ago. It hasn't been released as far as I know. If you're running off of HEAD, then it should be fixed.

[driveraid]

Sorry, I was confused about the reference to the 0.13.0 release above.. :)

[driveraid]

Seems released in https://github.com/bazelbuild/bazel/releases/tag/0.13.1 right?

"Important changes:

Remote Execution: Fixes a regression that produces directories with unsorted file/directory lists"

[buchgr]

@driveraid yes!