Skip to content

Commit

Permalink
Add changes on production environment (#2232)
Browse files Browse the repository at this point in the history
* [FIX] orb helm url (#2154)

[FIX] orb helm url (#2154)

* [FIX] chart lock url on kind environment (#2156)

[FIX] chart lock url on kind environment (#2156)

* new status for policy with no tap match (#2155)

* [ORB-203] Enable sink status update by listening to otel-collector logs. (#2135)

[ORB-203] Enable sink status update by listening to otel-collector logs. (#2135)

* [FIX] change organization to orb-community (#2157)

[FIX] change organization to orb-community (#2157)

* feat(maestro): fix message with error in mid-sentence. (#2158)

feat(maestro): fix message with error in mid-sentence. (#2158)

* fix(fleet): fix group tags editing - make it unable to delete tags (#2152)

* fix group tags editing - make it unable to delete tags

* typo(agent): fix no_tap_match status typo

* fix(fleet): partial updates for agent and endpoint unit tests

* fix(fleet): partial updates for agent and service unit tests

* Remove otel collector health_check extension unused (#2159)

Remove otel collector health_check extension unused (#2159)

* Removing logging extension on otel collector (#2160)

Removing logging extension on otel collector (#2160)

* [FIX] DeployCollector function (#2162)

[FIX] DeployCollector function (#2162)

* Update README.md (#2161)

* Update README.md

* [FIX] adding ownerid on decode GRPC sinks response (#2163)

* fix(maestro): sinks status fix on active and errored sinks

* [FIX] on logic for active and error sinks status

* symplify analyzeLogs function

* lastRemoteWrite is actually controlled by sinker

* fix message

* remove comments

* remove unnecessary

* deployment status should be internally controlled

* fix err variable

* add comments

* fix comment

* fix comments

* add error if collector was not found

* fix error

* add changes

* add changes

* fix maestro lastactivity

* fix maestro lastactivity

* fix condition for idle

* remove idle control on sinker

* deployment check

* feat(migrate): add migrate service to default and support openTelemetry in all sinks (#2150)

* feat(migrate): add migrate service, to add/remove openTelemetry, add control flag there to be able to rollback if necessary.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(sinker): remove skip, and add deprecation message.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(sinker): fix deprecation message.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(sinks): add default values for OpenTelemetry sink.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(ui): remove sinks OpenTelemetry flag.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(sinks): fix opentelemetry tag case

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(UI): remove OpenTelemetry flag.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(makefile): add kubectl rollout in makefile.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(migrate): fix identation.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(charts): testing chart migrate true.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(maestro): add otel flag enabled as true.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(maestro): re-add after testing.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* [FIX] refactoring maestro event sourcing for sink status (#2165)

[FIX] refactoring maestro event sourcing for sink status (#2165)

* Fix delete sinks on maestro (#2167)

Fix delete sinks on maestro (#2167)

* [Feat] Remove otel collector when it is idle (#2168)

* feat(maestro): update cache with creation of yaml. (#2169)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix failed to parse json (#2170)

* feat(maestro): update cache with creation of yaml.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(maestro): fix parse of json.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(maestro): add debug log.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* [FIX] maestro event sourcing to update otel collector (#2171)

[FIX] maestro event sourcing to update otel collector (#2171)

* increasing timeout for sink status because of otel (#2172)

* feat(migrate): prevent previous migration from executing in migrate up/down. (#2174)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(agent): fix context propagation for failure in otel (#2166)

* feat(agent): fix context propagation for failure in otel

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(agent): add context propagation.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinks): fix merging of the sinks tags and config. (#2176)

* fix(sinks): fix merging of the sinks tags and config.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinks): fix tests

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinker): fix not removing sinker cache when removing sink.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* [FIX] do not skip when lastActivity not found on redis (#2175)

* do not skip when lastactivity not found

* [FIX] only check last activity after error

* fix variables

* fix variables

---------

Co-authored-by: Luiz Henrique Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix state change to remove the local cache, refactored code, to only redis package have redisClient (#2177)

* fix(maestro): replace local cache for wait mechanism in getting the deployment state

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): add secondary step validation

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): focused every redis integration into redis eventStream service.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix maestro redis proper client. (#2178)

* fix(maestro): fix maestro redis proper client.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): reduce time to check for pod creation

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): added fine tuning to maestro. (#2179)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* Fine tuning maestro (#2180)

* fix(maestro): added fine tuning to maestro.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): added fine tuning to maestro.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): more fine tuning in wait periods for k8s

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): more fine-tuning in wait periods for k8s.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinks): add remote_host validation to prevent users to create sinks with invalid url

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinks): fix tests to adapt to new error

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* Sink error message (#2182)

* fix(maestro): added fine tuning to maestro.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): added fine tuning to maestro.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): more fine tuning in wait periods for k8s

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): more fine-tuning in wait periods for k8s.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinks): add remote_host validation to prevent users to create sinks with invalid url

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinks): fix tests to adapt to new error

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinks): fix error message.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro-): extracted change of status.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* Sinker ifs state (#2183)

* fix(sinker): fix if nesting

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinker): fix if nesting

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(maestro): fix error messages  (#2184)

* fix(sinker): fix if nesting

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinker): fix if nesting

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): warning as error.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix casing.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* changing scenario descriptions for dry-run execution (#2185)

* Update README.md (#2181)

* Dns and Net v2 metrics test (#2173)

* Update README.md

* updating kind readme

* testing metrics of dns v2

* testing metrics of dns v2

* metrics net handler v2

* net config

* fix(maestro): attempt of different approach on idle and active (#2186)

* fix(sinker): fix if nesting

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): refactored maestro monitor logic, and add logs to have traces on otel-sinker.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* inserting tag on test scenarios with error status on sink (#2187)

* fix(maestro): fix maestro reading events. (#2188)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix maestro reading redis and groups (#2189)

* fix(maestro): fix maestro reading redis and groups

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): change approach on apply/delete kubernetes clusters

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): re-added maestro stream. (#2190)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix monitor logic (#2191)

* fix(maestroStream): remove duplicate event publishing

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): enhanced logic from monitor.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix iteration to be more performatic and less error prone. (#2192)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* Fix iteration (#2193)

* fix(maestro): fix iteration to be more performatic and less error prone.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix nil reference error

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* Orb 694 tests otel migration (#2194)

* scenarios to make easier test otel migration

* fix(maestro): fix new nil reference error (#2195)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix extract id. (#2196)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): clean up code and create wait mechanism for apply deployments (#2197)

* fix(maestro): fix extract id.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): clean up code and enhanced verification of collector apply.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix method calls.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix method calls.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): change from pod, to deployment and added logs for matching items (#2198)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* Small changes on README (#2199)

* small changes

* Update README.md

* fix(redis): Add more logs (#2200)

* fix(makefile): clean up docker-compose.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(redis): add debug logs

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(makefile): remove sleep, just retry.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* [FIX] update kind manifests to 1.0.44 (#2203)

* fix kind

* fix kind

* [fix] kind

* tests to validate private image (#2202)

* Revert "fix(redis): add debug logs" (#2205)

* Revert "fix(redis): add debug logs"

This reverts commit 44f4b98.

* fix(redis): re-add fix on iteration

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(redis): fixing how maestro and sinker handles errors in reading and handling event errors (#2206)

* fix(redis): fix breaking out of reading routine because of failure.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(redis): fix breaking out of reading routine because of failure.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(redis): clean up.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix acking messages on error. (#2207)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* add prepare-helm first on run (#2209)

* fix(maestro): new configuration handling and creation on kubernetes. (#2211)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): reverting back to file management due to throttling of … (#2212)

* fix(maestro): reverting back to file management due to throttling of k8s api.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix reversion

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): remove idle validations. (#2214)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* [ORB-673] Set default path for orb-agent.db (#2213)

[ORB-673] Set default path for orb-agent.db (#2213)

* WIP: hotfix general css and dataset list table spacings (#2208)

* fix(maestro): rollback to complete deployment json (#2216)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* [ORB-669] - Add more changes on README (#2215)

* add more changes on README

* fix gh organization to orb-community

* fix(maestro): remove only changing status in active. to go back from errors like 429. (#2217)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): fix sink deletion propagation to collector and creation event status (#2218)

* fix(maestro): remove only changing status in active. to go back from errors like 429.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): alternative fix to delete collectors and create collectors correctly.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>
Co-authored-by: Everton H. Taques <97463920+etaques@users.noreply.github.com>

* rollback the changes on db file (#2219)

* fix(sinker): fix active sink not updating (#2220)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinker): add activity in metric push in sinker, on change from idle/unknown to active (#2221)

* fix(sinker): fix active sink not updating

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* Revert "fix(sinker): fix active sink not updating"

This reverts commit 5470945.

* fix(maestro): alternative fix on sink creation.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(maestro): updating timestamp.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>
Co-authored-by: Everton H. Taques <97463920+etaques@users.noreply.github.com>

* [FIX] set last activity on redis if sink already is active (#2222)

* fix active

* disable go routines to test

* Update streams.go

* add logs

* fix(maestro): add subgroup to sinker and fix old timestamped logs interfering with status (#2224)

* fix(maestro): ignore old messages, set as active.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinker): added different group for sinker when otel is enabled.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinker): added different group for sinker when otel is enabled.

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix(sinker): don't fetch logs older than 5 minutes. (#2225)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* fix install kubectl (#2226)

* fix install kubectl

* Update Makefile

* fix(maestro): fix deletion. (#2227)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* feat(maestro): fix subscription (#2229)

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>

* [ORB-689] sinks status synchronization (#2231)

* Update monitor.go

* Update kubecontrol.go

* change to 10 minutes to be idle

* set to change if idle be active if have activity

* back to active after idle

* Update kubecontrol.go

* increase idle time to 15 minutes

* fix idle

* fix sinker activity

* add changes

* changes

* add changes

* add changes

* add changes

* add changes

* add changes

* add changes

* add changes

* add changes

* add changes

* add changes

* add changes

* increasing timeout for test sink status (#2233)

---------

Signed-off-by: Luiz Pegoraro <luiz.pegoraro@encora.com>
Co-authored-by: manrodrigues <manrodriguessilva@gmail.com>
Co-authored-by: Luiz Henrique Pegoraro <luiz.pegoraro@encora.com>
Co-authored-by: Mariana Cavalcante <mariana.cavalcante@encora.com>
Co-authored-by: Guilhermo Pazuch <1490938+gpazuch@users.noreply.github.com>
  • Loading branch information
5 people authored Feb 20, 2023
1 parent fe7d5a6 commit 9e8bcba
Show file tree
Hide file tree
Showing 250 changed files with 3,000 additions and 4,192 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/go-develop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ jobs:
if: ${{ needs.prebuild.outputs.docs == 'true' && github.event_name != 'pull_request' }}
with:
token: ${{ secrets.GH_ORB_ACCESS_TOKEN }}
repository: ns1labs/orb-website
repository: orb-community/orb-website
event-type: build-docs
client-payload: '{"branch_name": "main"}'

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/go-production.yml
Original file line number Diff line number Diff line change
Expand Up @@ -479,7 +479,7 @@ jobs:
with:
branch: production
header: ":first_place_medal: :orb: *Deployed new features in orb.live* :orb: :rocket: :tada:"
github_repo: ns1labs/orb
github_repo: orb-community/orb
slack_channel: C041B9204CF # orb netdev slack channel ID
slack_api_token: ${{ secrets.SLACK_APP_TOKEN }}

Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,6 @@ python-test/behave_orb
test_agent_name*
docker/otel-collector-config.yaml
!docker/otel-collector-config.yaml.tpl

kind/*
!kind/README.md
13 changes: 7 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -135,9 +135,9 @@ install-helm:

install-kubectl:
cd /tmp && \
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
chmod a+x /kubectl && \
sudo mv ./kubectl /user/local/bin/kubectl
curl -LO "https://dl.k8s.io/release/v1.22.1/bin/linux/amd64/kubectl" && \
chmod a+x ./kubectl && \
sudo mv ./kubectl /usr/local/bin/kubectl

install-docker:
cd /tmp
Expand All @@ -156,15 +156,15 @@ install-k9s:
prepare-helm:
cd ./kind/ && \
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts && \
helm repo add ns1labs-orb https://ns1labs.github.io/orb-helm/ && \
helm repo add orb-community https://orb-community.github.io/orb-helm/ && \
helm dependency build

kind-create-all: kind-create-cluster kind-install-orb

kind-upgrade-all: kind-load-images kind-upgrade-orb

kind-create-cluster:
kind create cluster --image kindest/node:v1.23.0 --config=./kind/config.yaml
kind create cluster --image kindest/node:v1.22.15 --config=./kind/config.yaml

kind-delete-cluster:
kind delete cluster
Expand All @@ -189,6 +189,7 @@ kind-install-orb:

kind-upgrade-orb:
helm upgrade -n orb kind-orb ./kind
kubectl rollout restart deployment -n orb

kind-delete-orb:
kubectl delete -f ./kind/nginx.yaml
Expand All @@ -200,7 +201,7 @@ kind-delete-orb:

#

run: kind-create-all
run: prepare-helm kind-create-all

stop: kind-delete-orb kind-delete-cluster

Expand Down
33 changes: 14 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
<img src="docs/images/ORB-logo-black@3x.png" alt="Orb" width="500"/>
<img src="docs/images/OrbUIPreview.png" alt="Orb UI Preview" width="500"/>
<img src="https://user-images.githubusercontent.com/97463920/218170067-16a95078-6709-4828-b137-9791376b972e.png" alt="Orb UI Preview" width="500"/>

[![Total alerts](https://img.shields.io/lgtm/alerts/g/ns1labs/orb.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/ns1labs/orb/alerts/)
[![Go Report Card](https://goreportcard.com/badge/github.com/ns1labs/orb)](https://goreportcard.com/report/github.com/ns1labs/orb)
[![CodeCov](https://codecov.io/gh/ns1labs/orb/branch/develop/graph/badge.svg)](https://app.codecov.io/gh/ns1labs/orb/tree/develop)
[![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/ns1labs/orb.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/ns1labs/orb/context:javascript)

**Orb** is a new kind of observability platform that helps operators, developers, and end users understand their
networks, distributed applications, and traffic flows in real time.
[![Go Report Card](https://goreportcard.com/badge/github.com/ns1labs/orb)](https://goreportcard.com/report/github.com/orb-community/orb)
[![CodeCov](https://codecov.io/gh/ns1labs/orb/branch/develop/graph/badge.svg)](https://app.codecov.io/gh/orb-community/orb/tree/develop)

**Orb** is a modern network observability platform built to provide critical visibility into increasingly complex and distributed networks. It can analyze network traffic, run synthetic network probes, and connect the resulting telemetry directly to your existing observability stacks with OpenTelemetry. Orb differentiates from other solutions by pushing analysis close to the traffic sources (reducing inactionable metrics and processing costs), and allows for dynamic reconfiguration of remote agents in real time.

Ready to dive in? See [orb.community](https://orb.community) for [installation instructions](https://orb.community/documentation/install/).

Expand All @@ -25,7 +23,7 @@ resulting in light-weight, actionable metrics.
Based on the [pktvisor observability agent](https://pktvisor.dev), Orb's goal is to push analysis to the edge, where
high resolution data can be analysed in real time without the need to send raw data to a central location for batch
processing.
[Current analysis](https://github.com/ns1labs/pktvisor/wiki/Current-Metrics) focuses on L2-L3 Network, DNS, and DHCP
[Current analysis](https://github.com/orb-community/pktvisor/wiki/Current-Metrics) focuses on L2-L3 Network, DNS, and DHCP
with more analyzers in the works.

## Realtime Agent Orchestration
Expand All @@ -36,9 +34,7 @@ observability [policies](https://orb.community/about/#policies) designed to prec
are grouped and addressed based on [tags](https://orb.community/about/#agent-group).

## Flexible Integration With Modern Observability Stacks

Orb was built to integrate with modern observability stacks, supporting [Prometheus](https://prometheus.io/) natively
and designed to support arbitrary [sinks](https://orb.community/about/#sinks) in the future. Collection and sinking of the
Orb was built to integrate with modern observability stacks, supporting [OpenTelemetry](https://opentelemetry.io/) natively. Collection and sinking of the
metrics from the agents is included; there is no need to run additional data collection pipelines for Orb metrics.

## Portal and REST API Included
Expand All @@ -55,18 +51,17 @@ avoiding vendor lock-in.

***

# Backed by NS1
# Backed by NetBox Labs

**Orb** was born at [NS1 Labs](https://ns1.com/labs), where we're committed to
**Orb** was born at [NetBox Labs](https://netboxlabs.com/), where we're committed to
making [open source, dynamic edge observability a reality](https://ns1.com/blog/orb-a-new-paradigm-for-dynamic-edge-observability)
.

***

* [Installation Instructions](https://orb.community/documentation/install/)
* [View our Wiki](https://github.com/ns1labs/orb/wiki) for technical and architectural information
* [File an issue](https://github.com/ns1labs/orb/issues/new)
* Follow our [public work board](https://github.com/ns1labs/orb/projects/2)
* Start a [Discussion](https://github.com/ns1labs/orb/discussions)
* [Join us on Slack](https://join.slack.com/t/ns1labs/shared_invite/zt-qqsm5cb4-9fsq1xa~R3h~nX6W0sJzmA)
* Send mail to [info@pktvisor.dev](mailto:info@pktvisor.dev)
* [View our Wiki](https://github.com/orb-community/orb/wiki) for technical and architectural information
* [File an issue](https://github.com/orb-community/orb/issues/new)
* Start a [Discussion](https://github.com/orb-community/orb/discussions)
* [Join us on Slack](https://netdev.chat/)
* Send mail to [orb@netboxlabs.com](mailto:orb@netboxlabs.com)
9 changes: 6 additions & 3 deletions agent/backend/pktvisor/scrape.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,9 @@ func (p *pktvisorBackend) scrapeMetrics(period uint) (map[string]interface{}, er
return metrics, nil
}

func (p *pktvisorBackend) createOtlpMqttExporter(ctx context.Context) (component.MetricsExporter, error) {
func (p *pktvisorBackend) createOtlpMqttExporter(ctx context.Context, cancelFunc context.CancelFunc) (component.MetricsExporter, error) {

bridgeService := otel.NewBridgeService(&p.policyRepo, p.agentTags)
bridgeService := otel.NewBridgeService(ctx, &p.policyRepo, p.agentTags)
if p.mqttClient != nil {
cfg := otlpmqttexporter.CreateConfigClient(p.mqttClient, p.otlpMetricsTopic, p.pktvisorVersion, bridgeService)
set := otlpmqttexporter.CreateDefaultSettings(p.logger)
Expand Down Expand Up @@ -168,7 +168,7 @@ func (p *pktvisorBackend) scrapeOpenTelemetry(ctx context.Context) {
if p.mqttClient != nil {
if !ok {
var errStartExp error
p.exporter[policyID], errStartExp = p.createOtlpMqttExporter(exeCtx)
p.exporter[policyID], errStartExp = p.createOtlpMqttExporter(exeCtx, execCancelF)
if errStartExp != nil {
p.logger.Error("failed to create a exporter", zap.Error(err))
return
Expand Down Expand Up @@ -204,6 +204,9 @@ func (p *pktvisorBackend) scrapeOpenTelemetry(ctx context.Context) {
}
}
select {
case <-exeCtx.Done():
ctx.Done()
p.cancelFunc()
case <-ctx.Done():
err := p.exporter[policyID].Shutdown(exeCtx)
if err != nil {
Expand Down
27 changes: 18 additions & 9 deletions agent/otel/bridgeservice.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
package otel

import (
"context"
"github.com/ns1labs/orb/agent/policies"
"strings"
)

type AgentBridgeService interface {
RetrieveAgentInfoByPolicyName(policyName string) (*AgentDataPerPolicy, error)
NotifyAgentDisconnection(ctx context.Context, err error)
}

type AgentDataPerPolicy struct {
Expand All @@ -15,21 +17,23 @@ type AgentDataPerPolicy struct {
AgentTags map[string]string
}

var _ AgentBridgeService = (*bridgeService)(nil)
var _ AgentBridgeService = (*BridgeService)(nil)

type bridgeService struct {
policyRepo policies.PolicyRepo
AgentTags map[string]string
type BridgeService struct {
bridgeContext context.Context
policyRepo policies.PolicyRepo
AgentTags map[string]string
}

func NewBridgeService(policyRepo *policies.PolicyRepo, agentTags map[string]string) *bridgeService {
return &bridgeService{
policyRepo: *policyRepo,
AgentTags: agentTags,
func NewBridgeService(ctx context.Context, policyRepo *policies.PolicyRepo, agentTags map[string]string) *BridgeService {
return &BridgeService{
bridgeContext: ctx,
policyRepo: *policyRepo,
AgentTags: agentTags,
}
}

func (b *bridgeService) RetrieveAgentInfoByPolicyName(policyName string) (*AgentDataPerPolicy, error) {
func (b *BridgeService) RetrieveAgentInfoByPolicyName(policyName string) (*AgentDataPerPolicy, error) {
pData, err := b.policyRepo.GetByName(policyName)
if err != nil {
return nil, err
Expand All @@ -40,3 +44,8 @@ func (b *bridgeService) RetrieveAgentInfoByPolicyName(policyName string) (*Agent
AgentTags: b.AgentTags,
}, nil
}

func (b *BridgeService) NotifyAgentDisconnection(ctx context.Context, err error) {
ctx.Done()
b.bridgeContext.Done()
}
5 changes: 3 additions & 2 deletions agent/otel/otlpmqttexporter/otlp.go
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ func (e *exporter) pushMetrics(ctx context.Context, md pmetric.Metrics) error {
e.logger.Info("request metrics count per policyID", zap.String("policyID", e.policyID), zap.Int("metric_count", md.MetricCount()))
err = e.export(ctx, e.config.MetricsTopic, request)
if err != nil {
defer ctx.Done()
ctx.Done()
return err
}
return err
Expand All @@ -240,11 +240,12 @@ func (e *exporter) pushLogs(_ context.Context, _ plog.Logs) error {
return fmt.Errorf("not implemented")
}

func (e *exporter) export(_ context.Context, metricsTopic string, request []byte) error {
func (e *exporter) export(ctx context.Context, metricsTopic string, request []byte) error {
compressedPayload := e.compressBrotli(request)
c := *e.config.Client
if token := c.Publish(metricsTopic, 1, false, compressedPayload); token.Wait() && token.Error() != nil {
e.logger.Error("error sending metrics RPC", zap.String("topic", metricsTopic), zap.Error(token.Error()))
e.config.OrbAgentService.NotifyAgentDisconnection(ctx, token.Error())
return token.Error()
}
e.logger.Info("scraped and published metrics", zap.String("topic", metricsTopic), zap.Int("payload_size_b", len(request)), zap.Int("compressed_payload_size_b", len(compressedPayload)))
Expand Down
2 changes: 1 addition & 1 deletion agent/policies/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ var policyStateMap = [...]string{
"running",
"failed_to_apply",
"offline",
"NoTapMatch",
"no_tap_match",
}

var policyStateRevMap = map[string]PolicyState{
Expand Down
62 changes: 51 additions & 11 deletions cmd/maestro/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,6 @@ package main
import (
"context"
"fmt"
sinksgrpc "github.com/ns1labs/orb/sinks/api/grpc"
"github.com/opentracing/opentracing-go"
jconfig "github.com/uber/jaeger-client-go/config"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/credentials/insecure"
"io"
"os"
"os/signal"
Expand All @@ -25,6 +19,14 @@ import (
"syscall"
"time"

sinksgrpc "github.com/ns1labs/orb/sinks/api/grpc"
"github.com/opentracing/opentracing-go"
"github.com/spf13/viper"
jconfig "github.com/uber/jaeger-client-go/config"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/credentials/insecure"

"github.com/ns1labs/orb/maestro"
"github.com/ns1labs/orb/pkg/config"
"go.uber.org/zap"
Expand All @@ -41,7 +43,8 @@ const (

func main() {

esCfg := config.LoadEsConfig(envPrefix)
streamEsCfg := loadStreamEsConfig(envPrefix)
sinkerEsCfg := loadSinkerEsConfig(envPrefix)
svcCfg := config.LoadBaseServiceConfig(envPrefix, httpPort)
jCfg := config.LoadJaegerConfig(envPrefix)
sinksGRPCCfg := config.LoadGRPCConfig("orb", "sinks")
Expand Down Expand Up @@ -71,14 +74,20 @@ func main() {
_ = logger.Sync()
}(logger)
log := logger.Sugar()
esClient := connectToRedis(esCfg.URL, esCfg.Pass, esCfg.DB, logger)
streamEsClient := connectToRedis(streamEsCfg.URL, streamEsCfg.Pass, streamEsCfg.DB, logger)
defer func(esClient *r.Client) {
err := esClient.Close()
if err != nil {
return
}
}(esClient)

}(streamEsClient)
sinkerEsClient := connectToRedis(sinkerEsCfg.URL, sinkerEsCfg.Pass, sinkerEsCfg.DB, logger)
defer func(esClient *r.Client) {
err := esClient.Close()
if err != nil {
return
}
}(sinkerEsClient)
tracer, tracerCloser := initJaeger(svcName, jCfg.URL, logger)
defer func(tracerCloser io.Closer) {
err := tracerCloser.Close()
Expand All @@ -102,7 +111,7 @@ func main() {
sinksGRPCClient := sinksgrpc.NewClient(tracer, sinksGRPCConn, sinksGRPCTimeout, logger)
otelCfg := config.LoadOtelConfig(envPrefix)

svc := maestro.NewMaestroService(logger, esClient, sinksGRPCClient, esCfg, otelCfg)
svc := maestro.NewMaestroService(logger, streamEsClient, sinkerEsClient, sinksGRPCClient, streamEsCfg, otelCfg)
errs := make(chan error, 2)

mainContext, mainCancelFunction := context.WithCancel(context.Background())
Expand Down Expand Up @@ -189,3 +198,34 @@ func connectToRedis(redisURL, redisPass, redisDB string, logger *zap.Logger) *r.
DB: db,
})
}

func loadStreamEsConfig(prefix string) config.EsConfig {
cfg := viper.New()
cfg.SetEnvPrefix(fmt.Sprintf("%s_stream_es", prefix))

cfg.SetDefault("url", "localhost:6379")
cfg.SetDefault("pass", "")
cfg.SetDefault("db", "0")
cfg.SetDefault("consumer", fmt.Sprintf("%s-es-consumer", prefix))

cfg.AllowEmptyEnv(true)
cfg.AutomaticEnv()
var esC config.EsConfig
cfg.Unmarshal(&esC)
return esC
}

func loadSinkerEsConfig(prefix string) config.EsConfig {
cfg := viper.New()
cfg.SetEnvPrefix(fmt.Sprintf("%s_sinker_es", prefix))

cfg.SetDefault("url", "localhost:6378")
cfg.SetDefault("pass", "")
cfg.SetDefault("db", "1")

cfg.AllowEmptyEnv(true)
cfg.AutomaticEnv()
var esC config.EsConfig
cfg.Unmarshal(&esC)
return esC
}
9 changes: 6 additions & 3 deletions cmd/migrate/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ func main() {
usersDbCfg := config.LoadPostgresConfig(fmt.Sprintf("%s_%s", envPrefix, postgres.DbUsers), postgres.DbUsers)
thingsDbCfg := config.LoadPostgresConfig(fmt.Sprintf("%s_%s", envPrefix, postgres.DbThings), postgres.DbThings)
sinksDbCfg := config.LoadPostgresConfig(fmt.Sprintf("%s_%s", envPrefix, postgres.DBSinks), postgres.DBSinks)
sinksEncryptionKey := config.LoadEncryptionKey(fmt.Sprintf("%s_%s", envPrefix, postgres.DBSinks))
//sinksEncryptionKey := config.LoadEncryptionKey(fmt.Sprintf("%s_%s", envPrefix, postgres.DBSinks))

dbs := make(map[string]postgres.Database)

Expand All @@ -67,8 +67,11 @@ func main() {
svc := migrate.New(
log,
dbs,
migration.NewM1KetoPolicies(log, dbs),
migration.NewM2SinksCredentials(log, sinksDB, sinksEncryptionKey),
// When generating a new migration image
// Comment the previous and keep only the necessary steps to migrate up/down
//migration.NewM1KetoPolicies(log, dbs),
//migration.NewM2SinksCredentials(log, sinksDB, sinksEncryptionKey),
migration.NewM3SinksOpenTelemetry(log, sinksDB),
)

rootCmd := &cobra.Command{
Expand Down
Loading

0 comments on commit 9e8bcba

Please sign in to comment.