Skip to content

Commit

Permalink
Merge pull request #188 from Desiki-high/doc-tutorial
Browse files Browse the repository at this point in the history 
doc: make quickstart more friendly for newbies
  • Loading branch information
@imeoer
imeoer authored Sep 12, 2023
2 parents fe66f09 + 0b2cb77 commit 49bae7d
Show file tree
Hide file tree
Showing 2 changed files with 138 additions and 142 deletions.
224 changes: 93 additions & 131 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,110 +1,100 @@
# Acceleration Service

Acceleration Service provides a general service to Harbor with the ability to automatically convert user images to accelerated images. When a user does something such as artifact push, Harbor will request the service to complete the corresponding image conversion through its integrated Nydus, eStargz, etc. drivers.
Acceleration Service provides a general service to Harbor with the ability to automatically convert user images to accelerated images. When a user does something such as artifact push, Harbor will request the service to complete the corresponding image conversion through its integrated [Nydus](https://github.com/dragonflyoss/image-service),
[eStargz](https://github.com/containerd/stargz-snapshotter), etc. drivers.

⚠️ This project is still in progress, see more details in this [proposal](https://github.com/goharbor/community/pull/167).

## Quickstart

### Build binary

``` shell
$ make
```

### Configuration (Harbor side)

Login to Harbor web interface to complete the following operation.

1. Add webhook configuration

Projects -> Webhooks -> `+ NEW WEBHOOK`, add webhook configuration with following fields:

- Notify Type: http
- Event Type: artifact pushed
- Endpoint URL: `<acceleration service address>`/api/v1/conversions
- Auth Header: `<configured in acceleration service>`

2. Add robot account

Administration -> Robot Accounts -> `+ NEW ROBOT ACCOUNT`, add robot account with following fields:

- Expiration time: `<by your choice>`
- Reset permissions: select `Push Artifact`, `Pull Artifact`, `Create Tag`

Generate a base64 encoded auth string using robot account name and secret like this:

``` shell
$ printf '<robot-name>:<robot-secret>' | base64
```

### Configuration (Acceleration service side)

1. Copy template config file

nydus:

``` shell
$ cp misc/config/config.nydus.yaml misc/config/config.yaml
```

estargz:

``` shell
$ cp misc/config/config.estargz.yaml misc/config/config.yaml
```

2. Edit config file

Following comments in `misc/config/config.yaml`, ensure base64 encoded auth string and webhook auth header be configured correctly.

### Boot service
Boot daemon to serve webhook request in PUSH_ARTIFACT event

``` shell
$ ./acceld --config ./misc/config/config.yaml
```

### Test image conversion

1. Trigger image conversion

We can either push an image to Harbor:

``` shell
$ docker push <harbor-service-address>/library/nginx:latest
```

Or convert an existing image manually using the `accelctl` CLI tool:

``` shell
$ ./accelctl task create <harbor-service-address>/library/nginx:latest
$ ./accelctl task list
```

2. Got converted image

If everything is ready, we can see the log output in acceleration service like this:

``` shell
INFO[2021-09-17T05:43:49.783769943Z] Version: ecbfefc312ea78d81eb895ef7768a60caa30a281.20210917.0543
INFO[2021-09-17T05:43:49.786146553Z] [API] HTTP server started on 0.0.0.0:2077
INFO[2021-09-17T05:44:01.82592358Z] received webhook request from 192.168.1.1:46664 module=api
INFO[2021-09-17T05:44:01.826490232Z] POST /api/v1/conversions 200 552.695µs 392>5bytes 192.168.1.1 module=api
INFO[2021-09-17T05:44:01.830893103Z] pulling image 192.168.1.1/library/nginx:latest module=converter
INFO[2021-09-17T05:44:01.916123236Z] pulled image 192.168.1.1/library/nginx:latest module=converter
INFO[2021-09-17T05:44:01.916175337Z] converting image 192.168.1.1/library/nginx:latest module=converter
INFO[2021-09-17T05:44:04.317992888Z] converted image 192.168.1.1/library/nginx:latest-nydus module=converter
### GETTING STARTED

#### Get Harbor

Deploy a local harbor service if you don't have one, please refer to the harbor [documentation](https://goharbor.io/docs/latest/install-config).

#### Get binaries from release page

Currently, Acceleration Service includes the following tools:

- An `acceld` daemon to work as an HTTP service to handle image conversion requests from harbor or `accelctl`.
- An `accelctl` CLI tool to manage acceleration service (`acceld`) and can do image conversion in one-time mode.

Get `accelctl` and `acceld` binaries from [acceleration-service release](https://github.com/goharbor/acceleration-service/releases/latest).

### Configuration

#### Configure Habor

1. Login to the Harbor web interface.
2. Select one project and add a new Webhook configuration with the following fields:
* Notify Type: choose HTTP
* Event Type: Enable artifact pushed
* Endpoint URL: `<acceleration service address>`/api/v1/conversions
* Auth Header: `<configured in acceleration service>`
> Note: The webhook can help to convert images automatically by acceleration service.
> Also you can trigger an image conversion task by sending an HTTP request manually or using accelctl.
3. Create a system robot account with following fields:
* Expiration time: `<by your choice>`
* Reset permissions: select Push Artifact, Pull Artifact, Create Tag

When you get the robot account `robot$<robot-name>`, please copy the secret and
generate a base64 encoded auth string like this:

```bash
$ echo '<robot-name>:<robot-secret>' | base64
```
> Note: the encoded auth string will be used in configuring acceleration service on the next step.

#### Configure Acceleration Service

1. Copy the [template config file](https://github.com/goharbor/acceleration-service/tree/main/misc/config).
2. Modify the config file.
* Change `provider.source` with your own harbor service hostname, the `auth` and
`webhook.auth_header` should also be configured as the one generated by the above step.
* Change settings in the `converter.driver` filed according to your requirements.
> Please follow the comments in the template config file.

### Convert Image with Acceleration Service

#### Convert by acceld service
1. Boot acceld daemon in config file directory
```bash
$ ./acceld --config ./config.yaml
```
2. Trigger image conversion
* Push an image to trigger webhook.
```bash
$ docker push <harbor-service-address>/library/nginx:latest
```
* Convert manually by `accelctl` CLI tool.
> Please make sure the source OCI v1 images exist in your harbor registry.
```bash
$ ./accelctl task create <harbor-service-address>/library/nginx:latest
```
Or you can create a conversion task over the HTTP API by `curl`. Please
refer to the [development document](./docs/development.md#api).
```bash
$ curl --location 'http://<acceleration-service-address>/api/v1/conversions?sync=$snyc' \
--header 'Content-Type: application/json' \
--data '{
"type": "PUSH_ARTIFACT",
"event_data": {
"resources": [
{
"resource_url": "<harbor-service-address>/dfns/alpine:latest"
}
]
}
}
'
```

#### One-time mode

One-time mode allows to do a conversion without starting the acceld service, using accelctl like this:
```

After that, we will find converted artifact in Harbor interface with name `<harbor-service-address>/nginx:latest-nydus`.

### One-time mode

One-time mode allows to do a conversion without starting the `acceld` service, using `accelctl` like this:

```
$ accelctl convert --config ./misc/config/config.yaml 192.168.1.1/library/nginx:latest
$ ./accelctl convert --config ./config.yaml 192.168.1.1/library/nginx:latest

INFO[2022-01-28T03:39:28.039029557Z] pulling image 192.168.1.1/library/nginx:latest module=converter
INFO[2022-01-28T03:39:28.075375146Z] pulled image 192.168.1.1/library/nginx:latest module=converter
Expand All @@ -114,39 +104,11 @@ INFO[2022-01-28T03:39:29.561197593Z] pushing image 192.168.1.1/library/nginx:lat
INFO[2022-01-28T03:39:29.587585066Z] pushed image 192.168.1.1/library/nginx:latest-nydus module=converter
```
## Driver

### Interface

Acceleration Service Framework provides a built-in extensible method called driver, which allows the integration of various types of accelerated image format conversions, the framework will automatically handle operations such as pulling source image and pushing converted image, the format providers need to implement the following interface in [pkg/driver](./pkg/driver):

```golang
// Driver defines image conversion interface, the following
// methods need to be implemented by image format providers.
type Driver interface {
// Convert converts the source image to target image, where
// content parameter provides necessary image utils, image
// content store and so on, where source parameter is the
// original image reference. If conversion successful, the
// converted image manifest will be returned, otherwise a
// non-nil error will be returned.
Convert(ctx context.Context, content content.Provider, source string) (*ocispec.Descriptor, error)

// Name gets the driver type name, it is used to identify
// different accelerated image formats.
Name() string

// Version gets the driver version, it is used to identify
// different accelerated image format versions with same driver.
Version() string
}
```

### Testing
### Check Converted Image
We can specify the driver name by modifying `converter.driver` in the configuration file, and modify the fields in `converter.config` to specify the driver-related configuration, see [example configuration file](./misc/config/config.estargz.yaml).
You can see the converted image and source oci image in the some repo, they have different tag suffix.
## Documentation
* [API](./docs/api.md)
* [Development](./docs/development.md)
* [Garbage Collection](./docs/garbage-collection.md)
56 changes: 45 additions & 11 deletions docs/api.md → docs/development.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# API
# Development

## API

Acceld acts as an HTTP server to serve webhook HTTP requests
from Harbor for image conversions, acceld exposes following APIs:
Expand All @@ -10,9 +12,9 @@ from Harbor for image conversions, acceld exposes following APIs:
---
<a name="create-task"></a>

## Create Task
### Create Task

### Request
#### Request

```
POST /api/v1/conversions?sync=$sync
Expand All @@ -31,7 +33,7 @@ POST /api/v1/conversions?sync=$sync

`$sync`: boolean, `true` enable waiting for api to respond until the conversion task is completed.

### Response
#### Response

```
Ok
Expand All @@ -45,15 +47,15 @@ Ok

<a name="list-task"></a>

## List Task
### List Task

### Request
#### Request

```
GET /api/v1/conversions
```

### Response
#### Response

```
[
Expand All @@ -79,15 +81,15 @@ GET /api/v1/conversions

<a name="check-healthy"></a>

## Check Healthy
### Check Healthy

### Request
#### Request

```
GET /api/v1/health
```

### Response
#### Response

```
Ok
Expand All @@ -96,4 +98,36 @@ Ok
| Status | Description |
| ------ | --------------------------- |
| 200 | Accled service is healthy |
| 500 | Accled service is unhealthy |
| 500 | Accled service is unhealthy |

## Driver

### Interface

Acceleration Service Framework provides a built-in extensible method called driver, which allows the integration of various types of accelerated image format conversions, the framework will automatically handle operations such as pulling source image and pushing converted image, the format providers need to implement the following interface in [pkg/driver](../pkg/driver):

```golang
// Driver defines image conversion interface, the following
// methods need to be implemented by image format providers.
type Driver interface {
// Convert converts the source image to target image, where
// content parameter provides necessary image utils, image
// content store and so on, where source parameter is the
// original image reference. If conversion successful, the
// converted image manifest will be returned, otherwise a
// non-nil error will be returned.
Convert(ctx context.Context, content content.Provider, source string) (*ocispec.Descriptor, error)

// Name gets the driver type name, it is used to identify
// different accelerated image formats.
Name() string

// Version gets the driver version, it is used to identify
// different accelerated image format versions with same driver.
Version() string
}
```

### Testing

We can specify the driver name by modifying `converter.driver` in the configuration file, and modify the fields in `converter.config` to specify the driver-related configuration, see [example configuration file](../misc/config/config.estargz.yaml).

0 comments on commit 49bae7d

Please sign in to comment.