From 38657fb913649f834cf2fffbd44941de657a71f4 Mon Sep 17 00:00:00 2001 From: Yadong Ding Date: Fri, 8 Sep 2023 11:47:14 +0800 Subject: [PATCH 1/2] doc: move Driver to development.md 1. rename api.md to development.md 2. move Driver content from README to development.md Signed-off-by: Yadong Ding --- README.md | 2 +- docs/{api.md => development.md} | 56 ++++++++++++++++++++++++++------- 2 files changed, 46 insertions(+), 12 deletions(-) rename docs/{api.md => development.md} (51%) diff --git a/README.md b/README.md index 10559a29..4e79edfc 100644 --- a/README.md +++ b/README.md @@ -148,5 +148,5 @@ We can specify the driver name by modifying `converter.driver` in the configurat ## Documentation -* [API](./docs/api.md) +* [Development](./docs/development.md) * [Garbage Collection](./docs/garbage-collection.md) \ No newline at end of file diff --git a/docs/api.md b/docs/development.md similarity index 51% rename from docs/api.md rename to docs/development.md index 372b48a0..4b0303f4 100644 --- a/docs/api.md +++ b/docs/development.md @@ -1,4 +1,6 @@ -# API +# Development + +## API Acceld acts as an HTTP server to serve webhook HTTP requests from Harbor for image conversions, acceld exposes following APIs: @@ -10,9 +12,9 @@ from Harbor for image conversions, acceld exposes following APIs: --- -## Create Task +### Create Task -### Request +#### Request ``` POST /api/v1/conversions?sync=$sync @@ -31,7 +33,7 @@ POST /api/v1/conversions?sync=$sync `$sync`: boolean, `true` enable waiting for api to respond until the conversion task is completed. -### Response +#### Response ``` Ok @@ -45,15 +47,15 @@ Ok -## List Task +### List Task -### Request +#### Request ``` GET /api/v1/conversions ``` -### Response +#### Response ``` [ @@ -79,15 +81,15 @@ GET /api/v1/conversions -## Check Healthy +### Check Healthy -### Request +#### Request ``` GET /api/v1/health ``` -### Response +#### Response ``` Ok @@ -96,4 +98,36 @@ Ok | Status | Description | | ------ | --------------------------- | | 200 | Accled service is healthy | -| 500 | Accled service is unhealthy | \ No newline at end of file +| 500 | Accled service is unhealthy | + +## Driver + +### Interface + +Acceleration Service Framework provides a built-in extensible method called driver, which allows the integration of various types of accelerated image format conversions, the framework will automatically handle operations such as pulling source image and pushing converted image, the format providers need to implement the following interface in [pkg/driver](./pkg/driver): + +```golang +// Driver defines image conversion interface, the following +// methods need to be implemented by image format providers. +type Driver interface { + // Convert converts the source image to target image, where + // content parameter provides necessary image utils, image + // content store and so on, where source parameter is the + // original image reference. If conversion successful, the + // converted image manifest will be returned, otherwise a + // non-nil error will be returned. + Convert(ctx context.Context, content content.Provider, source string) (*ocispec.Descriptor, error) + + // Name gets the driver type name, it is used to identify + // different accelerated image formats. + Name() string + + // Version gets the driver version, it is used to identify + // different accelerated image format versions with same driver. + Version() string +} +``` + +### Testing + +We can specify the driver name by modifying `converter.driver` in the configuration file, and modify the fields in `converter.config` to specify the driver-related configuration, see [example configuration file](./misc/config/config.estargz.yaml). From 0b2cb771b598d1c16b109380753b4752ce5d4eda Mon Sep 17 00:00:00 2001 From: Yadong Ding Date: Tue, 12 Sep 2023 10:48:58 +0800 Subject: [PATCH 2/2] doc: make quickstart more friendly for newbies Signed-off-by: Yadong Ding --- README.md | 222 ++++++++++++++++++-------------------------- docs/development.md | 4 +- 2 files changed, 94 insertions(+), 132 deletions(-) diff --git a/README.md b/README.md index 4e79edfc..1b064f2e 100644 --- a/README.md +++ b/README.md @@ -1,110 +1,100 @@ # Acceleration Service -Acceleration Service provides a general service to Harbor with the ability to automatically convert user images to accelerated images. When a user does something such as artifact push, Harbor will request the service to complete the corresponding image conversion through its integrated Nydus, eStargz, etc. drivers. +Acceleration Service provides a general service to Harbor with the ability to automatically convert user images to accelerated images. When a user does something such as artifact push, Harbor will request the service to complete the corresponding image conversion through its integrated [Nydus](https://github.com/dragonflyoss/image-service), +[eStargz](https://github.com/containerd/stargz-snapshotter), etc. drivers. ⚠️ This project is still in progress, see more details in this [proposal](https://github.com/goharbor/community/pull/167). ## Quickstart - -### Build binary - -``` shell -$ make -``` - -### Configuration (Harbor side) - -Login to Harbor web interface to complete the following operation. - -1. Add webhook configuration - -Projects -> Webhooks -> `+ NEW WEBHOOK`, add webhook configuration with following fields: - -- Notify Type: http -- Event Type: artifact pushed -- Endpoint URL: ``/api/v1/conversions -- Auth Header: `` - -2. Add robot account - -Administration -> Robot Accounts -> `+ NEW ROBOT ACCOUNT`, add robot account with following fields: - -- Expiration time: `` -- Reset permissions: select `Push Artifact`, `Pull Artifact`, `Create Tag` - -Generate a base64 encoded auth string using robot account name and secret like this: - -``` shell -$ printf ':' | base64 -``` - -### Configuration (Acceleration service side) - -1. Copy template config file - -nydus: - -``` shell -$ cp misc/config/config.nydus.yaml misc/config/config.yaml -``` - -estargz: - -``` shell -$ cp misc/config/config.estargz.yaml misc/config/config.yaml -``` - -2. Edit config file - -Following comments in `misc/config/config.yaml`, ensure base64 encoded auth string and webhook auth header be configured correctly. - -### Boot service -Boot daemon to serve webhook request in PUSH_ARTIFACT event - -``` shell -$ ./acceld --config ./misc/config/config.yaml -``` - -### Test image conversion - -1. Trigger image conversion - -We can either push an image to Harbor: - -``` shell -$ docker push /library/nginx:latest -``` - -Or convert an existing image manually using the `accelctl` CLI tool: - -``` shell -$ ./accelctl task create /library/nginx:latest -$ ./accelctl task list -``` - -2. Got converted image - -If everything is ready, we can see the log output in acceleration service like this: - -``` shell -INFO[2021-09-17T05:43:49.783769943Z] Version: ecbfefc312ea78d81eb895ef7768a60caa30a281.20210917.0543 -INFO[2021-09-17T05:43:49.786146553Z] [API] HTTP server started on 0.0.0.0:2077 -INFO[2021-09-17T05:44:01.82592358Z] received webhook request from 192.168.1.1:46664 module=api -INFO[2021-09-17T05:44:01.826490232Z] POST /api/v1/conversions 200 552.695µs 392>5bytes 192.168.1.1 module=api -INFO[2021-09-17T05:44:01.830893103Z] pulling image 192.168.1.1/library/nginx:latest module=converter -INFO[2021-09-17T05:44:01.916123236Z] pulled image 192.168.1.1/library/nginx:latest module=converter -INFO[2021-09-17T05:44:01.916175337Z] converting image 192.168.1.1/library/nginx:latest module=converter -INFO[2021-09-17T05:44:04.317992888Z] converted image 192.168.1.1/library/nginx:latest-nydus module=converter +### GETTING STARTED + +#### Get Harbor + +Deploy a local harbor service if you don't have one, please refer to the harbor [documentation](https://goharbor.io/docs/latest/install-config). + +#### Get binaries from release page + +Currently, Acceleration Service includes the following tools: + +- An `acceld` daemon to work as an HTTP service to handle image conversion requests from harbor or `accelctl`. +- An `accelctl` CLI tool to manage acceleration service (`acceld`) and can do image conversion in one-time mode. + +Get `accelctl` and `acceld` binaries from [acceleration-service release](https://github.com/goharbor/acceleration-service/releases/latest). + +### Configuration + +#### Configure Habor + +1. Login to the Harbor web interface. +2. Select one project and add a new Webhook configuration with the following fields: + * Notify Type: choose HTTP + * Event Type: Enable artifact pushed + * Endpoint URL: ``/api/v1/conversions + * Auth Header: `` + > Note: The webhook can help to convert images automatically by acceleration service. + > Also you can trigger an image conversion task by sending an HTTP request manually or using accelctl. + +3. Create a system robot account with following fields: + * Expiration time: `` + * Reset permissions: select Push Artifact, Pull Artifact, Create Tag + + When you get the robot account `robot$`, please copy the secret and + generate a base64 encoded auth string like this: + + ```bash + $ echo ':' | base64 + ``` + > Note: the encoded auth string will be used in configuring acceleration service on the next step. + +#### Configure Acceleration Service + +1. Copy the [template config file](https://github.com/goharbor/acceleration-service/tree/main/misc/config). +2. Modify the config file. + * Change `provider.source` with your own harbor service hostname, the `auth` and + `webhook.auth_header` should also be configured as the one generated by the above step. + * Change settings in the `converter.driver` filed according to your requirements. + > Please follow the comments in the template config file. + +### Convert Image with Acceleration Service + +#### Convert by acceld service +1. Boot acceld daemon in config file directory + ```bash + $ ./acceld --config ./config.yaml + ``` +2. Trigger image conversion + * Push an image to trigger webhook. + ```bash + $ docker push /library/nginx:latest + ``` + * Convert manually by `accelctl` CLI tool. + > Please make sure the source OCI v1 images exist in your harbor registry. + ```bash + $ ./accelctl task create /library/nginx:latest + ``` + Or you can create a conversion task over the HTTP API by `curl`. Please + refer to the [development document](./docs/development.md#api). + ```bash + $ curl --location 'http:///api/v1/conversions?sync=$snyc' \ + --header 'Content-Type: application/json' \ + --data '{ + "type": "PUSH_ARTIFACT", + "event_data": { + "resources": [ + { + "resource_url": "/dfns/alpine:latest" + } + ] + } + } + ' + ``` + +#### One-time mode + +One-time mode allows to do a conversion without starting the acceld service, using accelctl like this: ``` - -After that, we will find converted artifact in Harbor interface with name `/nginx:latest-nydus`. - -### One-time mode - -One-time mode allows to do a conversion without starting the `acceld` service, using `accelctl` like this: - -``` -$ accelctl convert --config ./misc/config/config.yaml 192.168.1.1/library/nginx:latest +$ ./accelctl convert --config ./config.yaml 192.168.1.1/library/nginx:latest INFO[2022-01-28T03:39:28.039029557Z] pulling image 192.168.1.1/library/nginx:latest module=converter INFO[2022-01-28T03:39:28.075375146Z] pulled image 192.168.1.1/library/nginx:latest module=converter @@ -114,37 +104,9 @@ INFO[2022-01-28T03:39:29.561197593Z] pushing image 192.168.1.1/library/nginx:lat INFO[2022-01-28T03:39:29.587585066Z] pushed image 192.168.1.1/library/nginx:latest-nydus module=converter ``` -## Driver - -### Interface - -Acceleration Service Framework provides a built-in extensible method called driver, which allows the integration of various types of accelerated image format conversions, the framework will automatically handle operations such as pulling source image and pushing converted image, the format providers need to implement the following interface in [pkg/driver](./pkg/driver): - -```golang -// Driver defines image conversion interface, the following -// methods need to be implemented by image format providers. -type Driver interface { - // Convert converts the source image to target image, where - // content parameter provides necessary image utils, image - // content store and so on, where source parameter is the - // original image reference. If conversion successful, the - // converted image manifest will be returned, otherwise a - // non-nil error will be returned. - Convert(ctx context.Context, content content.Provider, source string) (*ocispec.Descriptor, error) - - // Name gets the driver type name, it is used to identify - // different accelerated image formats. - Name() string - - // Version gets the driver version, it is used to identify - // different accelerated image format versions with same driver. - Version() string -} -``` - -### Testing +### Check Converted Image -We can specify the driver name by modifying `converter.driver` in the configuration file, and modify the fields in `converter.config` to specify the driver-related configuration, see [example configuration file](./misc/config/config.estargz.yaml). +You can see the converted image and source oci image in the some repo, they have different tag suffix. ## Documentation diff --git a/docs/development.md b/docs/development.md index 4b0303f4..5cc45e00 100644 --- a/docs/development.md +++ b/docs/development.md @@ -104,7 +104,7 @@ Ok ### Interface -Acceleration Service Framework provides a built-in extensible method called driver, which allows the integration of various types of accelerated image format conversions, the framework will automatically handle operations such as pulling source image and pushing converted image, the format providers need to implement the following interface in [pkg/driver](./pkg/driver): +Acceleration Service Framework provides a built-in extensible method called driver, which allows the integration of various types of accelerated image format conversions, the framework will automatically handle operations such as pulling source image and pushing converted image, the format providers need to implement the following interface in [pkg/driver](../pkg/driver): ```golang // Driver defines image conversion interface, the following @@ -130,4 +130,4 @@ type Driver interface { ### Testing -We can specify the driver name by modifying `converter.driver` in the configuration file, and modify the fields in `converter.config` to specify the driver-related configuration, see [example configuration file](./misc/config/config.estargz.yaml). +We can specify the driver name by modifying `converter.driver` in the configuration file, and modify the fields in `converter.config` to specify the driver-related configuration, see [example configuration file](../misc/config/config.estargz.yaml).