Skip to content

Commit

Permalink
node-exporter as DaemonSet
Browse files Browse the repository at this point in the history
node-exporter as DaemonSet

node-exporter as DaemonSet

node-exporter as DaemonSet

node-exporter as DaemonSet

Removed 'Set Kubernetes credentials' task in Prometheus Role. Repository Rebased

Update core/src/epicli/data/common/ansible/playbooks/roles/node_exporter/tasks/install-node-exporter-as-system-service.yml

Co-authored-by: Michał Opala <sk4zuzu@gmail.com>

Update core/src/epicli/data/common/ansible/playbooks/roles/node_exporter/tasks/install-node-exporter-as-system-service.yml

Co-authored-by: Michał Opala <sk4zuzu@gmail.com>

Update core/src/epicli/data/common/ansible/playbooks/roles/prometheus/tasks/configure-k8s-apps-monitoring.yml

Co-authored-by: Michał Opala <sk4zuzu@gmail.com>

Fix kubeconfig generation logic (#1712)

* postflight: small improvements

* prometheus: be extra paranoid about kubeconfig

* kubernetes_node: drop unneeded copy/remove kubeconfig cycle

* kubernetes_master: remove unneeded remote kubeconfig generation

* kubernetes_master: cleanup

* prometheus: remove unneeded copy-kubeconfig call

* kubernetes_master: remove kubeconfig-related garbage + fix order of operations

* kubernetes_master: add noop task to make it easier to borrow "master" vars

* kubernetes_master: simplify vars handling

* Update 0.8 changelog

* Revert "Update 0.8 changelog"

This reverts commit fd05e03.

separate repository machine

Kubeconfig environment added

Kubeconfig environment added to playbook

Kubeconfig environment added to playbook
  • Loading branch information
Robert Pudlowski committed Oct 2, 2020
1 parent 983f68e commit c880da4
Show file tree
Hide file tree
Showing 35 changed files with 412 additions and 217 deletions.
6 changes: 6 additions & 0 deletions CHANGELOG-0.8.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
- [#1618](https://github.com/epiphany-platform/epiphany/issues/1618) - Add kubectl and Helm to epicli and devcontainer images
- [#1225](https://github.com/epiphany-platform/epiphany/issues/1225) - Add OS_PATCHING.md with information about patching RHEL OS
- [#1656](https://github.com/epiphany-platform/epiphany/issues/1656) - Run Helm tasks from Epiphany container
- [#1640](https://github.com/epiphany-platform/epiphany/issues/1640) - Added separate machine for repository and changed helm to use localhost address

### Updated

Expand All @@ -33,3 +34,8 @@
- [#1659](https://github.com/epiphany-platform/epiphany/issues/1659) - epicli upgrade fails on Ubuntu on downgrading kubernetes-cni package
- [#1681](https://github.com/epiphany-platform/epiphany/issues/1681) - Node exporter does not work after the upgrade
- [#1705](https://github.com/epiphany-platform/epiphany/issues/1705) - [RHEL/CentOS] epicli fails on downloading requirements - Docker CE repo not available

### Breaking changes

- Repository machine was introduced (ref #1640)
- Change cluster configuration manifest in order to be compatible with changes in #1640 [example] (https://github.com/epiphany-platform/epiphany/blob/develop/core/src/epicli/data/common/defaults/epiphany-cluster.yml)
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

Reference for actual cluster component versions can be found [here](docs/home/COMPONENTS.md)

- [CHANGELOG-0.8.x](./CHANGELOG-0.8.md)
- [CHANGELOG-0.7.x](./CHANGELOG-0.7.md)
- [CHANGELOG-0.6.x](./CHANGELOG-0.6.md)
- [CHANGELOG-0.5.x](./CHANGELOG-0.5.md)
Expand Down
1 change: 1 addition & 0 deletions core/src/epicli/cli/engine/ansible/AnsibleVarsGenerator.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ def write_role_manifest_vars(self, ansible_dir, role, kind):
def populate_group_vars(self, ansible_dir):
main_vars = ObjDict()
main_vars['admin_user'] = self.cluster_model.specification.admin_user
main_vars['k8s_as_cloud_service'] = self.cluster_model.specification.cloud.k8s_as_cloud_service
main_vars['validate_certs'] = Config().validate_certs
main_vars['offline_requirements'] = Config().offline_requirements
main_vars['wait_for_pods'] = Config().wait_for_pods
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ specification:
name: operations # YOUR-ADMIN-USERNAME
key_path: /user/.ssh/epiphany-operations/id_rsa # YOUR-SSH-KEY-PATH
components:
repository:
count: 1
machines:
- default-repository
kubernetes_master:
count: 1
machines:
Expand Down Expand Up @@ -45,6 +49,13 @@ specification:
---
kind: infrastructure/machine
provider: any
name: default-repository
specification:
hostname: repository # YOUR-MACHINE-HOSTNAME
ip: 192.168.100.112 # YOUR-MACHINE-IP
---
kind: infrastructure/machine
provider: any
name: default-k8s-master1
specification:
hostname: master1 # YOUR-MACHINE-HOSTNAME
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,14 @@ specification:
name: operations # YOUR-ADMIN-USERNAME
key_path: /user/.ssh/epiphany-operations/id_rsa # YOUR-SSH-KEY-PATH
cloud:
k8s_as_cloud_service: False
use_public_ips: False # When not using public IPs you have to provide connectivity via private IPs (VPN)
credentials:
key: XXXX-XXXX-XXXX
secret: XXXXXXXXXXXXXXXX
components:
repository:
count: 1
kubernetes_master:
count: 1
kubernetes_node:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -845,3 +845,48 @@ specification:
destination_port_range: "0"
source_address_prefix: "0.0.0.0/0"
destination_address_prefix: "0.0.0.0/0"
---
kind: infrastructure/virtual-machine
title: "Virtual Machine Infra"
provider: aws
name: repository-machine
specification:
size: t2.medium
security:
rules:
- name: ssh
description: Allow ssh traffic
direction: Inbound
protocol: Tcp
destination_port_range: "22"
source_address_prefix: "0.0.0.0/0"
destination_address_prefix: "0.0.0.0/0"
- name: repository
description: Allow repository traffic
direction: Inbound
protocol: Tcp
destination_port_range: "80"
source_address_prefix: "10.1.0.0/20"
destination_address_prefix: "0.0.0.0/0"
- name: node_exporter
description: Allow node_exporter traffic
direction: Inbound
protocol: Tcp
destination_port_range: "9100"
source_address_prefix: "10.1.0.0/20"
destination_address_prefix: "0.0.0.0/0"
- name: image_registry
description: Allow image registry traffic
direction: Inbound
protocol: Tcp
destination_port_range: "5000"
source_address_prefix: "10.1.0.0/20"
destination_address_prefix: "0.0.0.0/0"
- name: out
description: Allow out
direction: Egress
protocol: "all"
destination_port_range: "0"
source_address_prefix: "0.0.0.0/0"
destination_address_prefix: "0.0.0.0/0"

Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,11 @@ specification:
name: operations # YOUR-ADMIN-USERNAME
key_path: /user/.ssh/epiphany-operations/id_rsa # YOUR-SSH-KEY-PATH
cloud:
k8s_as_cloud_service: False
use_public_ips: False # When not using public IPs you have to provide connectivity via private IPs (VPN)
components:
repository:
count: 1
kubernetes_master:
count: 1
kubernetes_node:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -782,3 +782,62 @@ specification:
# destination_port_range: "0"
# source_address_prefix: "10.1.4.0/24"
# destination_address_prefix: "0.0.0.0/0"
---
kind: infrastructure/virtual-machine
title: "Virtual Machine Infra"
provider: azure
name: repository-machine
specification:
size: Standard_DS1_v2
security:
rules:
- name: ssh
description: Allow SSH
priority: 100
direction: Inbound
access: Allow
protocol: Tcp
source_port_range: "*"
destination_port_range: "22"
source_address_prefix: "0.0.0.0/0"
destination_address_prefix: "0.0.0.0/0"
- name: node_exporter
description: Allow node_exporter traffic
priority: 200
direction: Inbound
access: Allow
protocol: Tcp
source_port_range: "*"
destination_port_range: "9100"
source_address_prefix: "10.1.0.0/20"
destination_address_prefix: "0.0.0.0/0"
- name: repository
description: Allow repository traffic
priority: 205
direction: Inbound
access: Allow
protocol: Tcp
source_port_range: "*"
destination_port_range: "80"
source_address_prefix: "10.1.0.0/20"
destination_address_prefix: "0.0.0.0/0"
- name: image_registry
description: Allow image registry traffic
priority: 206
direction: Inbound
access: Allow
protocol: Tcp
source_port_range: "*"
destination_port_range: "5000"
source_address_prefix: "10.1.0.0/20"
destination_address_prefix: "0.0.0.0/0"
- name: out
description: Allow out
priority: 101
direction: Outbound
access: Allow
protocol: "*"
source_port_range: "*"
destination_port_range: "0"
source_address_prefix: "0.0.0.0/0"
destination_address_prefix: "0.0.0.0/0"
21 changes: 16 additions & 5 deletions core/src/epicli/data/common/ansible/playbooks/helm.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
---
- hosts: helm
become: true
become_method: sudo
become_flags: --login
- hosts: 127.0.0.1
gather_facts: false
connection: local
become: false
roles:
- helm
- role: helm
when:
- groups.helm is defined
- groups.helm | length > 0
- groups.repository is defined
- groups.repository | length > 0
environment:
# The "inventory_dir" fact cannot be used here.
# Documentation (https://docs.ansible.com/ansible/latest/inventory/implicit_localhost.html) states:
# - The inventory_file and inventory_dir magic variables are not available for the implicit localhost as they are dependent on each inventory host.
# TODO: https://github.com/epiphany-platform/epiphany/issues/1650
KUBECONFIG: "{{ vault_location }}/../kubeconfig"
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,8 @@
- import_role:
name: kubernetes_common
tasks_from: main
- import_role:
name: kubernetes_master
tasks_from: copy-kubeconfig
vars_files:
- roles/kubernetes_master/vars/main.yml
environment:
KUBECONFIG: "{{ kubeconfig.remote }}"

Expand All @@ -41,4 +40,6 @@
post_tasks:
- import_role:
name: kubernetes_master
tasks_from: remove-kubeconfig
tasks_from: copy-kubeconfig
environment:
KUBECONFIG: "{{ kubeconfig.remote }}"
13 changes: 2 additions & 11 deletions core/src/epicli/data/common/ansible/playbooks/kubernetes_node.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,8 @@
- import_role:
name: kubernetes_common
tasks_from: main
- import_role:
name: kubernetes_master
tasks_from: copy-kubeconfig
vars_files:
- roles/kubernetes_node/vars/main.yml
environment:
KUBECONFIG: "{{ kubeconfig.remote }}"

Expand All @@ -28,11 +27,3 @@
- import_role:
name: kubernetes_promote
tasks_from: update-kubelet

- hosts: kubernetes_node
become: true
become_method: sudo
post_tasks:
- import_role:
name: kubernetes_master
tasks_from: remove-kubeconfig
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,5 @@
become_method: sudo
roles:
- node_exporter
environment:
KUBECONFIG: "{{ kubeconfig.local }}"

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
---
# This file is meant to be used by other roles

- name: Prepare configuration and upgrade/install Helm chart
vars:
# Handling "undefined", "null", "empty" and "boolean" values all at once.
Expand All @@ -9,10 +11,14 @@

always:
- name: Clean up temporary Helm chart values file
delegate_to: localhost
file:
state: absent
path: /tmp/{{ helm_chart_name }}_values.yaml


delegate_to: localhost
environment:
KUBECONFIG: "{{ vault_location }}/../kubeconfig"
block:
# IF `disable_helm_chart`
- when: disable_helm_chart_bool
Expand Down
Original file line number Diff line number Diff line change
@@ -1,48 +1,42 @@
---
- import_tasks: gather-facts.yml

- name: Check system index file existence
uri:
url: http://localhost/epirepo/helm-charts/system/index.yaml
method: HEAD
register: uri_index_test
# fail with error if status is unexpected
failed_when: uri_index_test.status not in [200, 404]

- name: Deploy the "system" Helm charts
when: inventory_hostname == target_repository_hostnames[0] # execute on the main repository host only
block:
- name: Check if Helm chart repo already exists
shell: |
helm repo list | grep {{ helm_chart_repo_name }}
shell: helm repo list | grep -w "{{ helm_chart_repo_name }}"
register: helm_repo_list
failed_when: (helm_repo_list.rc != 0 and not 'no repositories' in helm_repo_list.stderr)
or helm_repo_list.rc > 1

- name: "Check if files exist in {{ specification.apache_epirepo_path }}/helm-charts/system"
find:
paths: "{{ specification.apache_epirepo_path }}/helm-charts/system"
register: helm_charts_files_number

- name: Add Helm chart repo and install charts
when: helm_charts_files_number.matched > 1
block:
- name: Get Helm charts list from repo before update
shell: |
helm search repo {{ helm_chart_repo_name }} --output json | jq -r '.[].name'
register: helm_charts_list_before_update
- name: Get Helm charts list from repo before update
shell: helm search repo {{ helm_chart_repo_name }} --output json | jq -r '.[].name'
register: helm_charts_list_before_update
failed_when: (helm_charts_list_before_update.rc != 0 and not 'no repositories' in helm_charts_list_before_update.stderr)
or helm_charts_list_before_update.rc > 1

- name: "Add {{ helm_chart_repo_name }} Helm repository from url {{ repository_url }}"
shell: |
helm repo add {{ helm_chart_repo_name }} {{ repository_url }}/helm-charts/system
when:
- helm_chart_repo_name not in helm_repo_list.stdout
- name: "Add {{ helm_chart_repo_name }} Helm repository from url http://localhost/epirepo"
shell: helm repo add {{ helm_chart_repo_name }} http://localhost/epirepo/helm-charts/system
when: helm_chart_repo_name not in helm_repo_list.stdout

- name: Update Helm repo
shell: |
helm repo update
- name: Update Helm repo
shell: helm repo update

- name: Get Helm charts list from repo after update
shell: |
helm search repo {{ helm_chart_repo_name }} --output json | jq -r '.[].name'
register: helm_charts_list_after_update
- name: Get Helm charts list from repo after update
shell: helm search repo {{ helm_chart_repo_name }} --output json | jq -r '.[].name'
register: helm_charts_list_after_update

- name: Delete Helm charts not present in repository
shell: |
helm delete {{ chart_subname }} --purge
vars:
chart_subname: "{{ item.split('/')[1] }}"
loop: "{{ helm_charts_list_before_update.stdout_lines }}"
when: item not in helm_charts_list_after_update.stdout_lines
- name: Delete Helm charts not present in repository
shell: helm delete {{ chart_subname }} --purge
vars:
chart_subname: "{{ item.split('/')[1] }}"
loop: "{{ helm_charts_list_before_update.stdout_lines }}"
when: item not in helm_charts_list_after_update.stdout_lines
when: uri_index_test.status == 200 # whole block of code is supposed to be executed only if system charts exist
Loading

0 comments on commit c880da4

Please sign in to comment.