Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move oak_runtime and oak_loader crates #1369

Merged
merged 1 commit into from
Aug 19, 2020
Merged

Move oak_runtime and oak_loader crates #1369

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/upload_coverage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,4 @@ jobs:

- name: Upload to codecov.io
run: |
bash <(curl -s https://codecov.io/bash) -f ./oak/server/target/debug/lcov.info
bash <(curl -s https://codecov.io/bash) -f ./oak_loader/target/debug/lcov.info
2 changes: 1 addition & 1 deletion docs/programming-oak.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ the following script:
The Oak Application is then started using the Oak Loader:

```bash
./oak/server/target/x86_64-unknown-linux-musl/release/oak_loader --application=./examples/hello_world/bin/hello_world.oak
./oak_loader/target/x86_64-unknown-linux-musl/release/oak_loader --application=./examples/hello_world/bin/hello_world.oak
```

The Oak Loader will launch an [Oak Runtime](concepts.md#oak-runtime), and this
Expand Down
3 changes: 1 addition & 2 deletions docs/sdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,7 @@ as a `Duration` since the UNIX epoch.

### `oak::proto` Module

The
[`oak::proto`](https://project-oak.github.io/oak/sdk/doc/oak/proto/index.html)
The [`oak::proto`](https://project-oak.github.io/oak/sdk/doc/proto/index.html)
module holds auto-generated submodules for dealing with protocol buffers.

### `oak::logger` Module
Expand Down
2 changes: 1 addition & 1 deletion examples/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ oak = { path = "../sdk/rust/oak" }
oak_abi = { path = "../oak_abi" }
oak_derive = { path = "../sdk/rust/oak_derive" }
oak_io = { path = "../oak_io" }
oak_runtime = { path = "../oak/server/rust/oak_runtime" }
oak_runtime = { path = "../oak_runtime" }
oak_services = { path = "../oak_services" }
oak_tests = { path = "../sdk/rust/oak_tests" }
oak_utils = { path = "../oak_utils" }
Expand Down
2 changes: 1 addition & 1 deletion examples/aggregator/scripts/docker_build_and_push
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ readonly BACKEND_DOCKER_IMAGE_NAME='gcr.io/oak-ci/oak-aggregator-backend'
# Build aggregator image

# Build the base server.
docker build --tag=oak_docker --file=./oak/server/Dockerfile ./oak/server
docker build --tag=oak_docker --file=./oak_loader/Dockerfile ./oak_loader
# Build and save the example
docker build --tag=aggregator --file=./examples/Dockerfile "./examples/aggregator"
docker tag aggregator "${AGGREGATOR_DOCKER_IMAGE_NAME}":latest
Expand Down
2 changes: 1 addition & 1 deletion examples/aggregator/scripts/run_server
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ source "${GLOBAL_SCRIPTS_DIR}/common"
"${GLOBAL_SCRIPTS_DIR}/build_example" -e aggregator

readonly APPLICATION="${PWD}/examples/aggregator/bin/aggregator.oak"
exec cargo run --release --target=x86_64-unknown-linux-musl --manifest-path=oak/server/rust/oak_loader/Cargo.toml -- \
exec cargo run --release --target=x86_64-unknown-linux-musl --manifest-path=oak_loader/Cargo.toml -- \
--application="${APPLICATION}" \
--grpc-tls-private-key="${GLOBAL_SCRIPTS_DIR}/../examples/certs/local/local.key" \
--grpc-tls-certificate="${GLOBAL_SCRIPTS_DIR}/../examples/certs/local/local.pem" \
Expand Down
2 changes: 1 addition & 1 deletion examples/authentication/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The Oak server can be run with the OpenID Connect authentication service enabled
using the --oidc-client flag:

```bash
cargo run --manifest-path=oak/server/rust/oak_loader/Cargo.toml -- \
cargo run --manifest-path=oak_loader/Cargo.toml -- \
--application=<APP_CONFIG_PATH> \
--grpc-tls-private-key=<PRIVATE_KEY_PATH> \
--grpc-tls-certificate=<CERTIFICATE_PATH> \
Expand Down
2 changes: 1 addition & 1 deletion examples/authentication/client/build.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ use oak_utils::{generate_grpc_code, CodegenOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
generate_grpc_code(
"../../../oak/proto",
"../../../proto",
&["authentication.proto"],
CodegenOptions {
build_client: true,
Expand Down
2 changes: 1 addition & 1 deletion examples/chat/proto/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ proto_library(
name = "chat_proto",
srcs = ["chat.proto"],
deps = [
"//oak/proto:handle_proto",
"//oak_services/proto:grpc_encap_proto",
"//proto:handle_proto",
"@com_google_protobuf//:empty_proto",
],
)
Expand Down
2 changes: 1 addition & 1 deletion examples/chat/proto/chat.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ syntax = "proto3";
package oak.examples.chat;

import "google/protobuf/empty.proto";
import "oak/proto/handle.proto";
import "proto/handle.proto";
import "oak_services/proto/grpc_encap.proto";

message CreateRoomRequest {
Expand Down
4 changes: 2 additions & 2 deletions examples/hello_world/client/android/java/com/google/oak/hello_world/MainActivity.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@

/** Main activity for the Oak Android "Hello, World" app. */
public class MainActivity extends Activity {
// Keep in sync with /oak/server/rust/oak_runtime/src/node/grpc/server/mod.rs.
// Keep in sync with /oak_runtime/src/node/grpc/server/mod.rs.
private static final String OAK_LABEL_GRPC_METADATA_KEY = "x-oak-label-bin";

private HandlerThread backgroundHandlerThread;
private Handler backgroundHandler;

Expand Down
36 changes: 17 additions & 19 deletions examples/hello_world/client/nodejs/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ const CERT_PATH = __dirname + '/../../../certs/local/ca.pem';
const SERVICE_PROTO_PATH = __dirname + '/../../proto/hello_world.proto';
const OAK_LABEL_PROTO_PATH = __dirname + '/../../../../oak_abi/proto/label.proto';

// Keep in sync with /oak/server/rust/oak_runtime/src/node/grpc/server/mod.rs.
// Keep in sync with /oak_runtime/src/node/grpc/server/mod.rs.
const oakLabelGrpcMetadataKey = 'x-oak-label-bin';

async function main() {
Expand Down Expand Up @@ -52,31 +52,29 @@ async function main() {
return metaData;
}

const helloWorldProto = grpc.loadPackageDefinition(helloWorldDefinition).oak
.examples.hello_world;
const helloWorldProto = grpc.loadPackageDefinition(helloWorldDefinition).oak.examples.hello_world;
const credentials = grpc.credentials.createSsl(fs.readFileSync(CERT_PATH));

const client = new helloWorldProto.HelloWorld('localhost:8080', credentials);

// For documentation on client calls see the `@grpc/grpc-js` documentation:
// https://grpc.github.io/grpc/node/grpc.Client.html#~CallProperties
client.sayHello(
// The arguments passed to the gRPC service. Corresponds to the
// `HelloRequest` message type in hello_world.proto file.
{ greeting: 'Node.js' },
// The metadata for this gRPC call.
getGrpcMetadata(),
// Callback invoked with the response.
(error, response) => {
if (error) {
console.error(error);
process.exit(1);
} else {
console.log(response.reply);
process.exit(0);
}
}
);
// The arguments passed to the gRPC service. Corresponds to the
// `HelloRequest` message type in hello_world.proto file.
{greeting: 'Node.js'},
// The metadata for this gRPC call.
getGrpcMetadata(),
// Callback invoked with the response.
(error, response) => {
if (error) {
console.error(error);
process.exit(1);
} else {
console.log(response.reply);
process.exit(0);
}
});
}

main();
2 changes: 1 addition & 1 deletion examples/hello_world/module/cpp/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,6 @@ cc_binary(
"//oak/module:oak_abi",
"//oak/module:oak_main",
# TODO(#422): Sort out inclusion of protobuf files
# "//oak/proto:oak_api_cc_proto",
# "//proto:oak_api_cc_proto",
],
)
2 changes: 1 addition & 1 deletion examples/injection/proto/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ proto_library(
name = "injection_proto",
srcs = ["injection.proto"],
deps = [
"//oak/proto:handle_proto",
"//proto:handle_proto",
"@com_google_protobuf//:empty_proto",
],
)
Expand Down
2 changes: 1 addition & 1 deletion examples/injection/proto/injection.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

syntax = "proto3";

import "oak/proto/handle.proto";
import "proto/handle.proto";

package oak.examples.injection;

Expand Down
4 changes: 2 additions & 2 deletions examples/tensorflow/module/cpp/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ cc_binary(
"//oak/module:oak_main",
"//oak/module:placeholders",
# TODO(#422): Sort out inclusion of protobuf files
# "//oak/proto:oak_api_cc_proto",
# "//proto:oak_api_cc_proto",
"@org_tensorflow//tensorflow/lite:framework",
"@org_tensorflow//tensorflow/lite/kernels:builtin_ops",
],
Expand Down Expand Up @@ -82,7 +82,7 @@ cc_binary(
"//oak/module:oak_main",
"//oak/module:placeholders",
# TODO(#422): Sort out inclusion of protobuf files
# "//oak/proto:oak_api_cc_proto",
# "//proto:oak_api_cc_proto",
"@org_tensorflow//tensorflow/lite/micro:micro_framework",
"@org_tensorflow//tensorflow/lite/micro/kernels:all_ops_resolver",
],
Expand Down
2 changes: 1 addition & 1 deletion examples/translator/client/translator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ var (
caCert = flag.String("ca_cert", "", "Path to the PEM-encoded CA root certificate")
)

// Keep in sync with /oak/server/rust/oak_runtime/src/node/grpc/server/mod.rs.
// Keep in sync with /oak_runtime/src/node/grpc/server/mod.rs.
const oakLabelGrpcMetadataKey = "x-oak-label-bin"

func translate(ctx context.Context, client translator_pb.TranslatorClient, text, fromLang, toLang string) {
Expand Down
2 changes: 1 addition & 1 deletion oak/common/label.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ namespace oak {
//
// See https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md.
//
// Keep in sync with /oak/server/rust/oak_runtime/src/node/grpc/server/mod.rs.
// Keep in sync with /oak_runtime/src/node/grpc/server/mod.rs.
const char kOakLabelGrpcMetadataKey[] = "x-oak-label-bin";

// The `-bin` suffix allows sending binary data for this metadata key.
Expand Down
2 changes: 1 addition & 1 deletion oak/module/oak_main.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ extern "C" void process_invocation(const uint8_t* req_buf, uint32_t req_size,
oak_abi::Handle rsp_handle);

// TODO(#422): Sort out inclusion of protobuf files
// #include "oak/proto/oak_api.pb.h"
// #include "proto/oak_api.pb.h"

// Placeholder main entrypoint for C++ Oak Modules that have simple unary
// method semantics. The Oak Module needs to provide an implementation
Expand Down
22 changes: 0 additions & 22 deletions oak/server/Cargo.toml
View file
Open in desktop

This file was deleted.

9 changes: 5 additions & 4 deletions oak_abi/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,15 @@ license = "Apache-2.0"

[dependencies]
log = "*"
prost = "*"
prost-types = "*"
prost = { path = "../third_party/prost" }
prost-types = { path = "../third_party/prost/prost-types" }

[build-dependencies]
oak_utils = { path = "../oak_utils" }
prost-build = "*"
prost-build = { path = "../third_party/prost/prost-build" }

[patch.crates-io]
prost-build = { path = "../third_party/prost/prost-build" }
prost = { path = "../third_party/prost" }
prost-build = { path = "../third_party/prost/prost-build" }
prost-derive = { path = "../third_party/prost/prost-derive" }
prost-types = { path = "../third_party/prost/prost-types" }
12 changes: 2 additions & 10 deletions oak_io/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions oak_io/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,9 @@ license = "Apache-2.0"

[dependencies]
oak_abi = { path = "../oak_abi" }

[patch.crates-io]
prost = { path = "../third_party/prost" }
prost-build = { path = "../third_party/prost/prost-build" }
prost-derive = { path = "../third_party/prost/prost-derive" }
prost-types = { path = "../third_party/prost/prost-types" }
Comment on lines +12 to +17
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daviddrysdale I have added these patches to various crates, which actually we should have done in the past already; if you look at the diff in the corresponding Cargo.lock, the previous version used to depend on prost from github, while now it depends on the local vendored copy (which is expected).

The problem is that when we run cargo fmt --all, it tries to format all those third party crates too for whatever reason. Have you come across this issue before? Do you have any idea how to prevent it?

The main thing I can think of is to reverse the logic in runner so that it only runs cargo fmt on non-workspace Cargo.toml files, i.e. the opposite of what it does now. I guess it may make that step more fine grain and therefore possibly a bit slower, but I don't think it matters that much.

WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would cargo fmt --all --exclude prost --exclude prost-build --exclude prost-derive --exclude prost-types work?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My version of cargo fmt does not seem to have an --exclude flag, does yours? If so, what version is it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah no, sorry – I didn't try it, I just followed rust-lang/cargo#2878 & rust-lang/cargo#4031 and assumed --exclude applied everywhere.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe create a third_party/prost/.rustfmt.toml file with an ignore directive that matches *.rs ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea! I actually created it under third_party directly, and ignored everything. Thanks for the suggestion!

0 oak/server/.dockerignore → oak_loader/.dockerignore
View file
Open in desktop
File renamed without changes.
26 changes: 0 additions & 26 deletions oak/server/Cargo.lock → oak_loader/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading