Skip to content

Latest commit

 

History

History
336 lines (246 loc) · 9.61 KB

run.md

File metadata and controls

336 lines (246 loc) · 9.61 KB

Running Mayastor

Hard Requirements

Mayastor supports the following Instruction Set Architectures (ISA):

  • x86_64 (Nehalem or later)
  • aarch64

Your system will need several control groups configured.

# /etc/nixos/configuration.nix
boot.kernelParams = [ "cgroup_enable=cpuset" "cgroup_memory=1" "cgroup_enable=memory" ];

It will also need at least 1024 2 MB Hugepages configured.

Learn more about hugepages: parts 1, 2, 3, 4, 5.

In NixOS:

# /etc/nixos/configuration.nix
boot.kernelParams = [ "hugepagez=2M" "hugepages=4096" ];
boot.kernel.sysctl = {
  "vm.hugetlb_shm_group" = "6969";
};
systemd.mounts = [
  # disable mounting hugepages by systemd,
  # it doesn't know about 1G pagesize
  { where = "/dev/hugepages";
      enable = false;
  }
  { where = "/dev/hugepages/hugepages-2MB";
      enable  = true;
      what  = "hugetlbfs";
      type  = "hugetlbfs";
      options = "pagesize=2M,gid=6969,mode=0775";
      requiredBy  = [ "basic.target" ];
  }
  { where = "/dev/hugepages/hugepages-1G";
      enable  = true;
      what  = "hugetlbfs";
      type  = "hugetlbfs";
      options = "pagesize=1G,gid=6969,mode=0775";
      requiredBy  = [ "basic.target" ];
  }
];
users = {
  users.${MAYASTOR_USER} = {
    # ...
    extraGroups = [ /* ... */ "hugepages" ];
  };
  groups = {
    hugepages = {
      gid = 6969; # Any ID works.
    };
  };
};

If you changed any of these, reboot after.

Optional Prerequisites

In order to use the full feature set of Mayastor, some or all of the following can be met:

  • A Linux Kernel 5.1+ (with io-uring support)

  • The following kernel modules loaded:

    • nbd: Network Block Device support (for testing only)
    • nvme_core: NVMe Core support
    • nvme_fabrics: NVMe over Fabric support
    • nvme_tcp: NVMe over TCP support
    • nvme_rdma: NVMe (rDMA) support
    • nvme_loop: NVMe Loop Device support (for testing only)

    To load these on NixOS:

    # /etc/nixos/configuration.nix
boot.kernelModules = [
  "nbd" "xfs" "nvme_tcp" "nvme_rdma"
];

    To load these on non-NixOS machines:

    modprobe nbd nvme_tcp nvme_rdma nvme_loop

  • For Asymmetric Namespace Access (ANA) support (early preview), the following kernel build configuration enabled:

    • CONFIG_NVME_MULTIPATH: enables support for multipath access to NVMe subsystems

    This is usually already enabled in distributions kernels, at least for RHEL/CentOS 8.2, Ubuntu 20.04 LTS, and SUSE Linux Enterprise 15.2.

    On some distributions such as RHEL 8, the feature must be enabled manually:

    # /etc/modprobe.d/nvme-multipath
options nvme_core multipath=1

    followed by reloading the nvme-core module or rebooting.

    On recent versions of NixOS this is already enabled by default, otherwise you may build it as such:

    # /etc/nixos/configuration.nix
boot.kernelPackages = pkgs.linuxPackages;
boot.kernelPatches = [ {
  name = "nvme-multipath";
  patch = null;
  extraConfig = ''
    NVME_MULTIPATH y
    '';
} ];

    followed by:

    sudo nixos-rebuild boot

  • An NVMe device. (Typically via PCI-E through a standard slot or M.2 port)

  • A version of nix configured as in the build guide.

Running binaries directly

Like in the build guide's Iterative Build section invoke nix shell. This time, don't pass --derivation, this will cause nix to evaluate the output, instead of the derivation.

nix shell -f . mayastor
io-engine-client --version
mayastor --version

Running mayastor should proceed without issue:

❯ mayastor
[2021-01-28T14:40:30.659032358-08:00  INFO mayastor:mayastor.rs:46] Starting Mayastor ..
[2021-01-28T14:40:30.659087313-08:00  INFO mayastor:mayastor.rs:47] kernel io_uring support: yes
[2021-01-28T14:40:30.659108924-08:00  INFO mayastor:mayastor.rs:51] free_pages: 4096 nr_pages: 4096
[2021-01-28T14:40:30.659132238-08:00  INFO mayastor::subsys::config:mod.rs:361] Applying Mayastor configuration settings
# ...
[2021-01-28T14:40:30.902100398-08:00  INFO mayastor:mayastor.rs:59] Mayastor started 🚀 ...
# ...

Running Docker images directly

After building the images, load them:

for FILE in artifacts/docker/*.tar; do
 echo "Loading ${FILE}..."
 docker load --quiet --input ${FILE}
done

Notice the generated tags. The build guide shows how to retag these to latest if required. If you forget, check docker images.

io-engine-client and mayastor-csi:

docker run --interactive --rm openebs/mayastor-io-engine-client:${TAG}
docker run --interactive --rm openebs/mayastor-io-engine:${TAG}

mayastor requires some special parameters:

docker run \
  --rm \
  --interactive \
  --privileged \
  --volume /dev:/dev:rw \
  --volume /dev/shm:/dev/shm:rw \
  --volume /dev/hugepages:/dev/hugepages:rw \
  --network host \
  openebs/mayastor-io-engine:${TAG}

Why these parameters?

  • --privileged to allow controlling memory policies.

    TODO: We can use control groups for this!

  • -v /dev:/dev:rw is needed to get access to any raw device you might want to consume as local storage and huge pages

  • -v /dev/shm:/dev/shm:rw is needed as for a circular buffer that can trace any IO operations as they happen

  • --network host to bypass virtual networks which might impact latency.

The moac container should not be directly launched.

Running as a systemd service

TODO: Mayastor currently depends on Mozilla's Rust Overlay, so these instructions don't work, yet!

On NixOS:

# /etc/nixos/configuration.nix
nixpkgs.overlays = [
  (import "${(builtins.fetchGit {
    url = "https://github.com/openebs/Mayastor.git";
    ref = "master";
    rev = "a9fc77f2ae30e909244556fc797451931dab3dd5";
  }).outPath}/nix/mayastor-overlay.nix")
];

systemd.services.mayastor = {
  wantedBy = [ "multi-user.target" ];
  after = [ "network.target" ];
  description = "A cloud native declarative data plane.";
  serviceConfig = {
    Type = "forking";
    User = "nobody";
    ExecStart = "${pkgs.mayastor}/bin/mayastor";
    AmbientCapabilities = "CAP_SETPCAP CAP_SYS_ADMIN CAP_IPC_LOCK CAP_SYS_NICE";
  };
};

On a non-NixOS system, create the following, and point ExecStart at your mayastor bundle:

# /etc/systemd/user/mayastor.service
[Unit]
After=network.target
Description=A cloud native declarative data plane.

[Service]
AmbientCapabilities=CAP_SETPCAP CAP_SYS_ADMIN CAP_IPC_LOCK CAP_SYS_NICE
ExecStart=/usr/bin/mayastor
Type=simple
User=nobody

Running as a NixOS Service

TODO: This is not supported yet! This is an aspiration:

# /etc/nixos/configuration.nix
services.mayastor.enable = true;
services.mayastor.grpc-bind = "0.0.0.0:10124";
services.mayastor.config = /etc/mayastor/config.yml;
services.mayastor.huge-dir = /dev/hugepages/;
# Etc...

Running in a VM

TODO: We're still writing this! Sorry! Let us know if you want us to prioritize this!

Running on a scratch Kubernetes cluster

First, get your k8s cluster working, our team likes to use the scripts in terraform/ for this. That will use lxd and libvirtd. See terraform/README for detailed instructions.

Otherwise, you can follow the production guide on your favourite.

Our testing showed some common Kubernetes development environments are not sufficient for Mayastor development.

Here are the ones known to not work by default:

Running on a real Kubernetes cluster

We have a production deployment manual & user manual prepared. Please refer to that for production Mayastor deployment and operation instructions.