Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cuspatial devcontainers #960

Merged
merged 26 commits into from
Mar 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
86fd57b
add devcontainer and initial scripts
trxcllnt Feb 10, 2023
81608ba
mount .aws and .config
trxcllnt Feb 14, 2023
dd0af56
use rapids-build-utils feature
trxcllnt Feb 22, 2023
bd8ae14
use feature from rapidsai/devcontainers
trxcllnt Feb 22, 2023
2cbc9ac
initialize the CUDA language in cuspatial-python scikit-build
trxcllnt Feb 22, 2023
5da1230
make devcontainer launcher always copy into a tmpdir
trxcllnt Feb 23, 2023
6bc7039
still mount settings in isolated devcontainers
trxcllnt Feb 24, 2023
561d26f
keep bind mounts consistent between host and container
trxcllnt Feb 24, 2023
978ddd8
make isolated images still use conda cache, make both conda and pip e…
trxcllnt Feb 27, 2023
207515e
generate correct dependencies for requirements.txt
trxcllnt Feb 27, 2023
5a2d108
rename devcontainer dirs
trxcllnt Feb 27, 2023
425d831
change symlinks
trxcllnt Feb 28, 2023
b833ffa
update main devcontainer symlink
trxcllnt Feb 28, 2023
908df32
swap launch.sh argument order
trxcllnt Feb 28, 2023
2c13e2c
replace shell with xdg-open
trxcllnt Feb 28, 2023
7921070
ensure .conda/pkgs exists
trxcllnt Feb 28, 2023
5cc687a
use '-isystem <dir>' instead of 'isystem=<dir>' to work around clangd…
trxcllnt Feb 28, 2023
0a5a216
use libc++ for clangd intellisense
trxcllnt Feb 28, 2023
e38e708
use LLVM 16
trxcllnt Feb 28, 2023
c6dd6c1
ignore attributes_not_allowed error from cudf headers
trxcllnt Feb 28, 2023
6903031
update pip versions in dependencies.yaml
trxcllnt Mar 1, 2023
0a19a7b
fix typo
trxcllnt Mar 6, 2023
473a1c9
use fancy sed in update-version.sh
trxcllnt Mar 8, 2023
465bb4c
copy in clangd config from llvm feature
trxcllnt Mar 15, 2023
827996e
add initial .devcontainer README.md
trxcllnt Mar 15, 2023
629cbc2
Merge branch 'branch-23.04' of github.com:rapidsai/cuspatial into fea…
trxcllnt Mar 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# syntax=docker/dockerfile:1.5

ARG CUDA=11.8
ARG LLVM=15
ARG RAPIDS=23.04
ARG DISTRO=ubuntu22.04
ARG REPO=rapidsai/devcontainers

ARG PYTHON_PACKAGE_MANAGER=conda

FROM ${REPO}:${RAPIDS}-cpp-llvm${LLVM}-cuda${CUDA}-${DISTRO} as pip-base

FROM ${REPO}:${RAPIDS}-cpp-llvm${LLVM}-cuda${CUDA}-mambaforge-${DISTRO} as conda-base

FROM ${PYTHON_PACKAGE_MANAGER}-base

ARG PYTHON_PACKAGE_MANAGER
ENV PYTHON_PACKAGE_MANAGER="${PYTHON_PACKAGE_MANAGER}"

USER coder

COPY --chown=coder:coder opt/cuspatial /opt/cuspatial

RUN /bin/bash -c 'mkdir -m 0755 -p ~/.{aws,cache,conda,config,local}'

WORKDIR /home/coder/

ENV PYTHONDONTWRITEBYTECODE="1"

ENV SCCACHE_REGION="us-east-2"
ENV SCCACHE_BUCKET="rapids-sccache-devs"
ENV VAULT_HOST="https://vault.ops.k8s.rapids.ai"
trxcllnt marked this conversation as resolved.
Show resolved Hide resolved

ENV HISTFILE="/home/coder/.cache/._bash_history"
103 changes: 103 additions & 0 deletions .devcontainer/README.md
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this!

Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# cuSpatial Development Containers

This directory contains [devcontainer configurations](https://containers.dev/implementors/json_reference/) for using VSCode to [develop in a container](https://code.visualstudio.com/docs/devcontainers/containers) via the `Remote Containers` [extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) or [GitHub Codespaces](https://github.com/codespaces).

This container is a turnkey development environment for building and testing the cuSpatial C++ and Python libraries.

## Table of Contents

* [Prerequisites](#prerequisites)
* [Host bind mounts](#host-bind-mounts)
* [Launch a Dev Container](#launch-a-dev-container)
* [via VSCode](#via-vscode)
* [via `launch.sh`](#via-launchsh)
* [Single mode](#single-mode)
* [Unified mode](#unified-mode)
* [Isolated mode](#isolated-mode)

## Prerequisites

* [VSCode](https://code.visualstudio.com/download)
* [VSCode Remote Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)

## Host bind mounts

By default, the following directories are bind-mounted into the devcontainer:

* `${repo}:/home/coder/cuspatial`
* `${repo}/../.aws:/home/coder/.aws`
* `${repo}/../.local:/home/coder/.local`
* `${repo}/../.cache:/home/coder/.cache`
* `${repo}/../.conda:/home/coder/.conda`
* `${repo}/../.config:/home/coder/.config`

This ensures caches, configurations, dependencies, and your commits are persisted on the host across container runs.

## Launch a Dev Container

A devcontainer can be launched directly by VSCode, or via a custom `launch.sh` shell script.

### via VSCode

To launch a devcontainer from VSCode, open the cuSpatial repo and select the "Reopen in Container" button in the bottom right:<br/><img src="https://user-images.githubusercontent.com/178183/221771999-97ab29d5-e718-4e5f-b32f-2cdd51bba25c.png"/>

Alternatively, open the VSCode command palette (typically `cmd/ctrl + shift + P`) and run the "Rebuild and Reopen in Container" command.

### via `launch.sh`

Use the `.devcontainer/launch.sh` script to start a new instance of the development container and launch a fresh VSCode window connected to it.

VSCode extends its [single-window-per-folder](https://github.com/microsoft/vscode/issues/2686) process model to devcontainers. Opening the same devcontainer in separate windows doesn't create two separate containers -- instead you have two VSCode windows each connected to the same running container.

`launch.sh` takes two arguments, a `mode` and a `package manager`.

* The `mode` argument determines how the devcontainer interacts with the files on the host.
* The `package manager` argument can be either `conda`, or `pip`. This determines whether the devcontainer uses `conda` or `pip` to install the dependencies (the default is `conda`). `pip` is experimental/not working for normal dev, and is currently meant to aid in pip packaging work.

#### Single mode

`.devcontainer/launch.sh single` launches the devcontainer with the [default bind mounts](#host-bind-mounts). RMM and cuDF are installed via the package manager.

Example:
```bash
# Launch a devcontainer that only mounts cuspatial and installs dependencies via conda
$ .devcontainer/launch.sh single conda

# or installs dependencies via pip
$ .devcontainer/launch.sh single pip
```

#### Unified mode

`.devcontainer/launch.sh unified` launches the devcontainer with the [default bind mounts](#host-bind-mounts), as well as additional `rmm` and `cudf` bind mounts (assumes RMM and cuDF are siblings to the cuspatial repository):

* `${repo}/../.rmm:/home/coder/rmm`
* `${repo}/../.cudf:/home/coder/cudf`

In this mode, RMM and cuDF will not be installed, but the devcontainer will install the dependencies necessary to build all three.

Example:
```bash
# Launch a devcontainer that mounts rmm, cudf, and cuspatial from the host and installs dependencies via conda
$ .devcontainer/launch.sh unified conda

# or installs dependencies via pip
$ .devcontainer/launch.sh unified pip
```

#### Isolated mode

`.devcontainer/launch.sh isolated` launches the devcontainer without the deps/repo bind mounts, and instead contains a unique copy of the `cuspatial` source in the container's file system.

Use this mode to launch multiple isolated development containers that can be checked out to separate branches of `cuspatial`.

**Be sure to push any commits you want to persist. Once this container is removed, any unpushed changes will be lost!**

Examples:
```bash
# Launch a devcontainer that is isolated from changes on the host and installs dependencies via conda
$ .devcontainer/launch.sh isolated conda

# or installs dependencies via pip
$ .devcontainer/launch.sh isolated pip
```
61 changes: 61 additions & 0 deletions .devcontainer/conda/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"shutdownAction": "none",

"build": {
"context": "${localWorkspaceFolder}/.devcontainer",
"dockerfile": "${localWorkspaceFolder}/.devcontainer/Dockerfile",
"args": {
"CUDA": "11.8",
"LLVM": "16",
"PYTHON_PACKAGE_MANAGER": "conda"
}
},
"hostRequirements": {
"gpu": true
},

"features": {
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:latest": {}
},

"overrideFeatureInstallOrder": [
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils"
],

"initializeCommand": [
"/bin/bash", "-c", "mkdir -m 0755 -p ${localWorkspaceFolder}/../.{aws,cache,config,conda/pkgs,conda/${localWorkspaceFolderBasename}-single-envs}"
],

"containerEnv": {
"DEFAULT_CONDA_ENV": "rapids"
},

"workspaceFolder": "/home/coder",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind,consistency=consistent",
"mounts": [
"source=${localWorkspaceFolder}/../.aws,target=/home/coder/.aws,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.cache,target=/home/coder/.cache,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.config,target=/home/coder/.config,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.conda/pkgs,target=/home/coder/.conda/pkgs,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.conda/${localWorkspaceFolderBasename}-single-envs,target=/home/coder/.conda/envs,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/.devcontainer/opt/${localWorkspaceFolderBasename},target=/opt/${localWorkspaceFolderBasename},type=bind,consistency=consistent"
],

"customizations": {
"vscode": {
"extensions": [
"mutantdino.resourcemonitor",
"tamasfe.even-better-toml"
],
"settings": {
"files.trimFinalNewlines": true,
"files.insertFinalNewline": true,
"files.trimTrailingWhitespace": true,
"files.watcherExclude": {
"**/target/**": true
},
"python.linting.flake8Enabled": true
}
}
}
}
63 changes: 63 additions & 0 deletions .devcontainer/conda/isolated/.devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"shutdownAction": "none",

"build": {
"context": "${localWorkspaceFolder}/.devcontainer",
"dockerfile": "${localWorkspaceFolder}/.devcontainer/Dockerfile",
"args": {
"CUDA": "11.8",
"LLVM": "16",
"PYTHON_PACKAGE_MANAGER": "conda"
}
},
"hostRequirements": {
"gpu": true
},

"features": {
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:latest": {}
},

"overrideFeatureInstallOrder": [
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils"
],

"initializeCommand": [
"/bin/bash", "-c", "mkdir -m 0755 -p ${localWorkspaceFolder}/../.{aws,cache,config,conda/pkgs}"
],

"updateContentCommand": [
"/bin/bash", "-c", "cp -ar /workspaces/${localWorkspaceFolderBasename} /home/coder/${localWorkspaceFolderBasename}"
],

"containerEnv": {
"DEFAULT_CONDA_ENV": "rapids"
},

"workspaceFolder": "/home/coder",
"workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},type=bind,consistency=consistent",
"mounts": [
"source=${localWorkspaceFolder}/../.aws,target=/home/coder/.aws,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.cache,target=/home/coder/.cache,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.config,target=/home/coder/.config,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.conda/pkgs,target=/home/coder/.conda/pkgs,type=bind,consistency=consistent"
],

"customizations": {
"vscode": {
"extensions": [
"mutantdino.resourcemonitor",
"tamasfe.even-better-toml"
],
"settings": {
"files.trimFinalNewlines": true,
"files.insertFinalNewline": true,
"files.trimTrailingWhitespace": true,
"files.watcherExclude": {
"**/target/**": true
},
"python.linting.flake8Enabled": true
}
}
}
}
1 change: 1 addition & 0 deletions .devcontainer/conda/single/.devcontainer/devcontainer.json
63 changes: 63 additions & 0 deletions .devcontainer/conda/unified/.devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"shutdownAction": "none",

"build": {
"context": "${localWorkspaceFolder}/.devcontainer",
"dockerfile": "${localWorkspaceFolder}/.devcontainer/Dockerfile",
"args": {
"CUDA": "11.8",
"LLVM": "16",
"PYTHON_PACKAGE_MANAGER": "conda"
}
},
"hostRequirements": {
"gpu": true
},

"features": {
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:latest": {}
},

"overrideFeatureInstallOrder": [
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils"
],

"initializeCommand": [
"/bin/bash", "-c", "mkdir -m 0755 -p ${localWorkspaceFolder}/../.{aws,cache,config,conda/pkgs,conda/${localWorkspaceFolderBasename}-unified-envs}"
],

"containerEnv": {
"DEFAULT_CONDA_ENV": "rapids"
},

"workspaceFolder": "/home/coder",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind,consistency=consistent",
"mounts": [
"source=${localWorkspaceFolder}/../rmm,target=/home/coder/rmm,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../cudf,target=/home/coder/cudf,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.aws,target=/home/coder/.aws,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.cache,target=/home/coder/.cache,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.config,target=/home/coder/.config,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.conda/pkgs,target=/home/coder/.conda/pkgs,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.conda/${localWorkspaceFolderBasename}-unified-envs,target=/home/coder/.conda/envs,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/.devcontainer/opt/${localWorkspaceFolderBasename},target=/opt/${localWorkspaceFolderBasename},type=bind,consistency=consistent"
],

"customizations": {
"vscode": {
"extensions": [
"mutantdino.resourcemonitor",
"tamasfe.even-better-toml"
],
"settings": {
"files.trimFinalNewlines": true,
"files.insertFinalNewline": true,
"files.trimTrailingWhitespace": true,
"files.watcherExclude": {
"**/target/**": true
},
"python.linting.flake8Enabled": true
}
}
}
}
61 changes: 61 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"shutdownAction": "none",

"build": {
"context": "${localWorkspaceFolder}/.devcontainer",
"dockerfile": "${localWorkspaceFolder}/.devcontainer/Dockerfile",
"args": {
"CUDA": "11.8",
"LLVM": "16",
"PYTHON_PACKAGE_MANAGER": "conda"
}
},
"hostRequirements": {
"gpu": true
},

"features": {
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:latest": {}
},

"overrideFeatureInstallOrder": [
"ghcr.io/rapidsai/devcontainers/features/rapids-build-utils"
],

"initializeCommand": [
"/bin/bash", "-c", "mkdir -m 0755 -p ${localWorkspaceFolder}/../.{aws,cache,config,conda/pkgs,conda/${localWorkspaceFolderBasename}-single-envs}"
],

"containerEnv": {
"DEFAULT_CONDA_ENV": "rapids"
},

"workspaceFolder": "/home/coder",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind,consistency=consistent",
"mounts": [
"source=${localWorkspaceFolder}/../.aws,target=/home/coder/.aws,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.cache,target=/home/coder/.cache,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.config,target=/home/coder/.config,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.conda/pkgs,target=/home/coder/.conda/pkgs,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/../.conda/${localWorkspaceFolderBasename}-single-envs,target=/home/coder/.conda/envs,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/.devcontainer/opt/${localWorkspaceFolderBasename},target=/opt/${localWorkspaceFolderBasename},type=bind,consistency=consistent"
],

"customizations": {
"vscode": {
"extensions": [
"mutantdino.resourcemonitor",
"tamasfe.even-better-toml"
],
"settings": {
"files.trimFinalNewlines": true,
"files.insertFinalNewline": true,
"files.trimTrailingWhitespace": true,
"files.watcherExclude": {
"**/target/**": true
},
"python.linting.flake8Enabled": true
}
}
}
}
Loading