The easiest way to run microservices, big data, and containers in production.
Like traditional operating systems, DC/OS is system software that manages computer hardware and software resources and provides common services for computer programs.
Unlike traditional operating systems, DC/OS spans multiple machines within a network, aggregating their resources to maximize utilization by distributed applications.
To learn more, see the DC/OS Overview.
- Learn More - https://dcos.io/
- Find the Docs - https://dcos.io/docs/
- Install - https://dcos.io/install/
- Get Started - https://dcos.io/get-started/
- Get Help - http://chat.dcos.io/
- Join the Discussion - https://groups.google.com/a/dcos.io/d/forum/users
- Report an Issue - https://dcosjira.atlassian.net
- Contribute - https://dcos.io/contribute/
DC/OS itself is composed of many individual components precisely configured to work together in concert.
This repo contains the release and package building tools necessary to produce installers for various on-premises and cloud platforms.
| Directory | Contents |
|---|---|
| docker | Locally defined docker containers packages are built in |
| docs | Documentation |
| dcos_installer | Backend for Web, SSH, and some bits of the Advanced installer. Code is being cleaned up |
| gen | Python library for rendering yaml config files for various platforms into packages, with utilities to do things like make "late binding" config set by CloudFormation |
| gen/installer | Code to take a build and transform it into a particular platform installer (Bash / command line, AWS, Azure, etc.) |
| packages | Packages which make up DC/OS (Mesos, Marathon, AdminRouter, etc). These packages are built by pkgpanda, and combined into a "bootstrap" tarball for deployment. |
| pkgpanda | DC/OS baseline/host package management system. Tools for building, deploying, upgrading, and bundling packages together which live on the root filesystem of a machine / underneath Mesos. |
| pytest | Misc. tests. Should be moved to live next to the appropriate code |
| release | Release tools for DC/OS. (Building releases, building installers for releases, promoting between channels) |
| ssh | AsyncIO based parallel ssh library used by the installer |
| test_util | various scripts, utilities to help with integration testing |
All code in this repository is Python 3
- Linux distribution:
- Docker doesn't have all the features needed on OS X or Windows
-
tarneeds to be GNU tar for the set of flags used - tox
- git 1.8.5+
- Docker
- Install Instructions for various distributions. Docker needs to be configured so your user can run docker containers. The command
docker run alpine /bin/echo 'Hello, World!'when run at a new terminal as your user should just print"Hello, World!". If it says something like "Unable to find image 'alpine:latest' locally" then re-run and the message should go away. - Python 3.5
- Arch Linux:
sudo pacman -S python- Fedora 23 Workstation: Already installed by default / no steps - Ubuntu 16.04 LTS:- pyenv-installer
- Python dependencies:
sudo apt-get install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils liblzma-dev - Install Python 3.5.2:
pyenv install 3.5.2 - Create DC/OS virtualenv:
pyenv virtualenv 3.5.2 dcos - Activate environment:
pyenv activate dcos
- Over 10GB of free disk space
- Optional pxz (speeds up package and bootstrap compression)
- ArchLinux: pxz-git in the AUR. The pxz package corrupts tarballs fairly frequently.
- Fedora 23:
sudo dnf install pxz
tox
Tox is used to run the codebase unit tests, as well as coding standard checks. The config is in tox.ini.
./build_local.sh
That will run a simple local build, and output the resulting DC/OS installers to $HOME/dcos-artifacts. You can run the created `dcos_generate_config.sh like so:
NOTE: Building a release from scratch the first time on a modern dev machine (4 cores / 8 hyper threads, SSD, reasonable interent bandwidth) takes about 1 hour.
$ $HOME/dcos-artifacts/testing/`whoami`/dcos_generate_config.sh
If you look inside of the bash script build_local.sh there are the commands with descriptions of each.
The general flow is to:
- Check the environment is reasonable
- Write a
releasetool configuration if one doesn't exist - Setup a python virtualenv where we can install the DC/OS python tools to in order to run them
- Install the DC/OS python tools to the virtualenv
- Build the release using the
releasetool
These steps can all be done by hand and customized / tweaked like standard python projects. You can hand create a virtualenvironment, and then do an editable pip install (pip install -e) to have a "live" working environment (as you change code you can run the tool and see the results).
This release tool always loads the config in dcos-release.config.yaml in the current directory.
The config is YAML. Inside it has two main sections. storage which contains a dictionary of different storage providers which the built artifacts should be sent to, and options which sets general DC/OS build configuration options.
Config values can either be specified directly, or you may use $ prefixed environment variables (the env variable must set the whole value).
All the available storage providers are in release/storage. The configuration is a dictionary of a reference name for the storage provider (local, aws, my_azure), to the configuration.
Each storage provider (ex: aws.py) is an available kind prefix. The dictionary factories defines the suffix for a particular kind. For instance kind: aws_s3 would map to the S3StorageProvider.
The configuration options for a storage provider are the storage provider's constructor parameters.
Sample config storage that will save to my home directory (/home/cmaloney):
storage:
local:
kind: local_path
path: /home/cmaloney/dcos-artifactsSample config that will store to a local archive path as wll as AWS S3. Environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY would need to be set to use the config (And something like a CI system could provide them so they don't have to be committed to a code repository).
storage:
aws:
kind: aws_s3
bucket: downloads.dcos.io
object_prefix: dcos
download_url: https://downloads.dcos.io/dcos/
access_key_id: $AWS_ACCESS_KEY_ID
secret_access_key: $AWS_SECRET_ACCESS_KEY
region_name: us-west-2
local:
kind: local_path
path: /mnt/big_artifact_store/dcos/Before a pull request can be merged into master, the following checks are required:
- teamcity/create-release-pr: in the CI system, build_teamcity is triggered and developers should use build_local.sh (see above)
- teamcity/code-quality: simply run
toxin the top-level dir to run all syntax checks as well as pytest (unit-tests). See tox.ini for more details - integration-test/*: runs integration_test.py in the network of a DC/OS cluster
- /vagrant-bash: Tests the on-prem bash provider by using dcos-vagrant. Invoke this test through run-all
- /deploy-vpc-cli: runs ccm-deploy-test with USE_INSTALLER_API=false. A Virtual Private Cloud of centos nodes is spun up by CCM (Mesosphere's Cloud Cluster Manager) and the installer (dcos_generate_config.sh) is used via the CLI options to deploy DC/OS. Finally, the same integration_test.py is run
- /deploy-vpc-api: the same as /deploy-vpc-cli (see above) except uses USE_INSTALLER_API=true, which causes the installer to be started with the
--weboption and then controlled entirely by the HTTP API
Lots of docs are still being written. If you have immediate questions please ask the DC/OS Community. Someone else probably has exactly the same question.
- Add getting started on common distros / dependencies
- Add overview of what is in here, how it works
- Add general theory of stuff that goes in here.
- PR (guidelines, testing)
- How to make different sorts of common changes