Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate scala control messages to protobuf #2950

Merged
merged 61 commits into from
Oct 31, 2024
Merged

Conversation

shengquan-ni
Copy link
Collaborator

@shengquan-ni shengquan-ni commented Oct 17, 2024

This PR migrates all Scala control messages and their corresponding handlers to use gRPC.


Main Changes (For both Scala and Python Engine)

  1. Rewrote AsyncRPCServer and AsyncRPCClient

    • Adjusted both to fit into the generated gRPC interface, while keeping RPCs sent and received through actor messages.
  2. Centralized RPC Definitions

    • All RPC messages and method definitions are now located in:
      src/main/protobuf/edu/uci/ics/amber/engine/architecture/rpc
      

Steps to Create New RPCs

  1. Create a request proto message in controlcommands.proto with all necessary inputs.
  2. Create a response proto message in controlreturns.proto.
  3. Define the RPC:
    • For controller-handled RPCs, add the definition to controllerservice.proto.
    • For worker-handled RPCs, add it to workerservice.proto.
  4. Generate Scala code by running protocGenerate in the SBT shell.
  5. Generate Python code by running ./scripts/python-proto-gen.sh from core folder.
  6. Implement the RPC handler:
    • Scala Engine: in either ControllerAsyncRPCHandlerInitializer or WorkerAsyncRPCHandlerInitializer, and move it to the promisehandlers folder for better organization.
    • Python Engine: create a new handler under python/core/architecture/handlers/control and override corresponding async method for the new rpc defined in worker service.

Changes in Sending RPCs (Scala Engine)

Example: Using the StartWorkflow RPC, which is handled by the controller.

  • Before this PR:

    send(StartWorkflow(), CONTROLLER).map { ret =>
      // handle the return value
    }
  • After this PR:

    controllerInterface.startWorkflow(EmptyRequest(), mkContext(CONTROLLER)).map { resp =>
        // handle the response
    }

@shengquan-ni shengquan-ni self-assigned this Oct 17, 2024
@shengquan-ni shengquan-ni force-pushed the shengquan-rpc-refactor branch 2 times, most recently from 8d5f5eb to 9922c15 Compare October 20, 2024 23:32
Copy link
Collaborator

@Yicong-Huang Yicong-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Had some tests locally and left some comments in code.

Copy link
Collaborator

@Yicong-Huang Yicong-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Had some tests locally and left some comments in code.

@shengquan-ni shengquan-ni merged commit 84eba2b into master Oct 31, 2024
9 checks passed
@shengquan-ni shengquan-ni deleted the shengquan-rpc-refactor branch October 31, 2024 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
engine refactoring Refactor the code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants