-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathREADME.md
269 lines (227 loc) · 12.4 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
# EC2 Github Action Builder
## TL;DR
Jump to [examples](#example-usage)
## Overview
This is a custom GitHub action to provision and manage self-hosted runners using AWS EC2 On-Demand and/or Spot instances.
It offers multiple spot instance provisioning modes:
- **None:** (default) Strictly On-Demand instances only
- **SpotOnly**: Strictly Spot instances only
- **BestEffort**: Use a Spot instance of same class and size when price is <= On-Demand
- (Automatic fallback to On-Demand)
- **MaxPerformance**: Use the largest spot instance in the same class for <= the On-Demand price
- (Automatic fallback to On-Demand)
Supported operating system AMIs:
- Amazon Linux
- Ubuntu
- Debian
## Why?
### Cost Savings
Operating system vCPUs Per-minute rate (USD)
```text
OS vCPU GH Price/Minute EC2 Price/Minute
Linux 2 $0.008 $0.001284 (c5a.large)
Linux 4 $0.016 $0.00257 (c5a.xlarge)
Linux 8 $0.032 $0.00514 (c5a.2xlarge)
Linux 16 $0.064 $0.0114 (c5.4xlarge)
Linux 32 $0.128 $0.02054 (c5a.8xlarge)
Linux 64 $0.256 $0.041067 (c5a.16xlarge)
```
Sources:
- [EC2 On-Demand Pricing](https://aws.amazon.com/ec2/pricing/on-demand/)
- [GH Action Runner Pricing](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates)
### Customizable Machine Image
- Set custom EC2 root volume size
- Set custom storage class for root volume
- Users can provide their own custom AMI image pre-loaded with all the necessary tooling of their choice saving time and cost.
### Enhance Security
- EC2 instances run within your infrastructure
- Easier to harden runner instances using custom AMIs, Security Groups etc
- Easier monitoring and vulnerability scanning using existing tools (e.g CloudWatch, GuardDuty, AWS Inspector etc)
- Secure networking setup by eliminating any need to expose ports to external service or using Bastion hosts!
- Lower data transfer costs (e.g ECR images, S3 objects etc)
## Setup
### 1. Create GitHub Personal Access Token
1. Create a [fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token)
2. Edit the token permissions and select `Only select repositories` for `Repository access`
3. Select any repositories you wish to use with this action
4. Grant `Read and Write access` for `Administration` access level under Repository permissions
5. Add the token to GitHub Action secrets and note the secret name
### 2. Setup GitHub Secrets for IAM credentials
#### 2a. Use IAM keys
1. Add your `IAM Access Key ID` and `Secret Access Key` to GitHub Secrets and note the secret names!
2. Modify `${{ secrets.DEPLOY_AWS_ACCESS_KEY_ID }}` and `${{ secrets.DEPLOY_AWS_SECRET_ACCESS_KEY }}` in examples below to match the names of your GH secrets
#### 2b. Use OIDC
1. Configure your EC2 backend to allow a federated connection from github
2. use [configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) or similar to authenticate against your backend using OIDC. See the OIDC example for guidance on how this is done. The documentation in the `configure-aws-credendials` is very detailed.
*Note*: For information about required IAM permissions check **IAM role policy** [here](./docs/CrossAccountIAM.md)
### 3. Collect EC2 information:
- `AWS Region` (e.g `us-west-2`)
- `EC2 AMI ID` for your desired instance type in the region ([Ubuntu AMI Locator](https://cloud-images.ubuntu.com/locator/ec2/))
- **Important Note:** Only Ubuntu, Amazon Linux and Debian AMIs have been tested
- To find AMIs **_for other operating systems follow instructions_** [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/finding-an-ami.html)
- `EC2 Subnet ID` for any subnet with internet access (Can be private with NAT)
- `EC2 Security Group ID` for any security group which allows all outbound traffic (Default configuration for an empty Security Group)
Note: The security group does not require any in-bound rules. You can add in-bound rules based on your needs (e.g open SSH port 22)
<h2 id="example-usage">
Examples
</h2>
### Standard
- Modify `ec2_spot_instance_strategy` for other deployment strategies. List of all values can be found [here](action.yaml)
- Modify `github_token` value to match the name for your Personal Access Token secret name
```yaml
jobs:
start-runner:
timeout-minutes: 5 # normally it only takes 1-2 minutes
name: Start self-hosted EC2 runner
runs-on: ubuntu-latest
permissions:
actions: write
steps:
- name: Start EC2 runner
id: start-ec2-runner
uses: NextChapterSoftware/ec2-action-builder@v1.7
with:
github_token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
aws_access_key_id: ${{ secrets.DEPLOY_AWS_ACCESS_KEY_ID }}
aws_secret_access_key: ${{ secrets.DEPLOY_AWS_SECRET_ACCESS_KEY }}
aws_region: "us-west-2"
ec2_instance_type: c5.4xlarge
ec2_ami_id: ami-008fe2fc65df48dac
ec2_subnet_id: "SUBNET_ID_REDACTED"
ec2_security_group_id: "SECURITY_GROUP_ID_REDACTED"
ec2_instance_ttl: 40 # Optional (default is 60 minutes)
ec2_spot_instance_strategy: None # Other options are: SpotOnly, BestEffort, MaxPerformance
# Job that runs on the self-hosted runner
run-build:
timeout-minutes: 1
needs:
- start-runner
runs-on: ${{ github.run_id }}
steps:
- run: env
```
### Advanced
- IAM policy and role setup instructions can be found [here](docs/CrossAccountIAM.md)
- Modify `ec2_spot_instance_strategy` for other deployment strategies. List of all values can be found [here](action.yaml)
**IMPORTANT NOTE**
`An error occured: Runner version vX.YZ is deprecated and cannot receive messages.`
Error message above is usually caused by `--disableupdate` custom configuration argument used with a deprecated Runner version.
Make sure to use a runner that has not been deprecated or omit `github_action_runner_version` to use the latest available version.
```yaml
jobs:
start-runner:
timeout-minutes: 5 # normally it only takes 1-2 minutes
name: Start self-hosted EC2 runner
runs-on: ubuntu-latest
permissions:
actions: write
steps:
- name: Start EC2 runner
id: start-ec2-runner
uses: NextChapterSoftware/ec2-action-builder@v1.7
with:
aws_access_key_id: ${{ secrets.DEPLOY_AWS_ACCESS_KEY_ID }}
aws_secret_access_key: ${{ secrets.DEPLOY_AWS_SECRET_ACCESS_KEY }}
aws_iam_role_arn: "arn:aws:iam::REDACTED:role/REDACTED"
aws_region: "us-west-2"
github_token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
github_action_runner_version: v2.300.2 # Optional (default is latest release)
github_action_runner_extra_cli_args: "--disableupdate" # Extra cli args for runner startup command
github_job_start_ttl_seconds: 60 # Optional - (defaults to 0 disabling this feature)
ec2_instance_type: c5.4xlarge
ec2_ami_id: ami-008fe2fc65df48dac
ec2_root_disk_size_gb: "100" # Optional - (defaults to AMI settings)
ec2_root_disk_ebs_class: "gp2" # Optional - Only used with custom volume root size (defaults to gp2)
ec2_subnet_id: "SUBNET_ID_REDACTED"
ec2_security_group_id: "SECURITY_GROUP_ID_REDACTED"
ec2_instance_ttl: 40 # Optional - (default is 60 minutes)
ec2_spot_instance_strategy: MaxPerformance # Other options are: None, BestEffort, MaxPerformance
ec2_instance_tags: > # Required for IAM role resource permission scoping
[
{"Key": "Owner", "Value": "deploybot"}
]
# Job that runs on the self-hosted runner
run-build:
timeout-minutes: 1
needs:
- start-runner
runs-on: ${{ github.run_id }}
steps:
- run: env
```
### Use OIDC
- IAM policy and role setup instructions can be found [here](https://github.com/aws-actions/configure-aws-credentials)
- Modify `ec2_spot_instance_strategy` for other deployment strategies. List of all values can be found [here](action.yaml)
```yaml
jobs:
start-runner:
timeout-minutes: 5 # normally it only takes 1-2 minutes
name: Start self-hosted EC2 runner
runs-on: ubuntu-latest
permissions:
actions: write
contents: read
id-token: write
steps:
- name: Configure AWS credentials
id: creds # name of step, to allow access to outputs
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: "AWS_REGION"
role-to-assume: "arn:aws:iam::REDACTED:role/REDACTED"
output-credentials: true # output the credentials
- name: Start EC2 runner
id: start-ec2-runner
uses: NextChapterSoftware/ec2-action-builder@v1.7
with:
github_token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
aws_access_key_id: ${{ steps.creds.outputs.aws-access-key-id }} # generated by configure-aws-credentials
aws_secret_access_key: ${{ steps.creds.outputs.aws-secret-access-key }} # generated by configure-aws-credentials
aws_session_token: ${{ steps.creds.outputs.aws-session-token }} # generated by configure-aws-credentials
aws_region: "AWS_REGION"
ec2_subnet_id: "SUBNET_ID_REDACTED"
ec2_security_group_id: "SECURITY_GROUP_ID_REDACTED"
ec2_instance_type: t4g.large
ec2_ami_id: ami-0c29a2c5cf69b5a9c
ec2_instance_ttl: 40 # Optional (default is 60 minutes)
ec2_spot_instance_strategy: BestEffort # Other options are: None, SpotOnly, BestEffort, MaxPerformance
ec2_instance_tags: > # Required for IAM role resource permission scoping
[
{"Key": "Owner", "Value": "deploybot"}
]
# Job that runs on the self-hosted runner
run-build:
timeout-minutes: 1
needs:
- start-runner
runs-on: ${{ github.run_id }}
steps:
- run: env
```
## How it all works under the hood
### General instance launch flow
- Your GitHub personal token is used to obtain a Runner Registration token
- If no explicit runner version has been provided, it will retrieve the latest version number
- It then uses all the provided info to compile an EC2 user-data script which does the following:
- Set a max TTL on the EC2 instance on startup
- Create a shutdown script which is executed when jobs end
- Downloads GitHub Action Runner bundle
- Unpack Action Runner bundle
- Configure Runner agent as an **ephemeral** agent
- EC2 instance is launched with the user-data script from previous step
- Once EC2 boot has completed, user-data script is executed
- Runner binary registers itself with GitHub API using the current job ID
- Once the Runner is registered, control is transferred to the next job (this is your build job)
- Upon a job completion (failure/success), Shutdown script is triggered to kill the instance with a 1 minute delay
### Spot instance provisioning
- Script looks up On-Demand price for the supplied instance type
- It will then look up EC2 Spot instance prices using AWS API
- Depending on the mode
- SpotOnly: It will try to launch a spot instance with On-Demand price as the max price cut-off
- BestEffort: It will try to launch a spot instance but falls back to On-Demand if prices are too high!
- MaxPerformance: It will try to get the largest spot instance in class for the On-Demand price of the supplied instance type. It falls back to On-Demand if prices are too high!
## Other EC2 Considerations
- Each instance is named as "{repo}-{jobID}"
- Default EC2 TTL is 60 minutes
- Other EC2 tags are `github_job_id` and `github_ref`
- Spot instances might be taken away by AWS without any prior notice