Fantastic Health-Checks and Where to Find Them

[3ch023]

Hi Milo people,

Commerce team got a requirement from Ops to implement a health-check:

Please create a health check card of some sort which can validate the upstream services are working as expected and returning sane data. If any of the upstream services are not working, we should get an HTTP 500 response with a description of what went wrong (e.g. commerce service not responding, timeout, bad data, etc).

I know that Milo Localisation team had implemented one already with Adobe IO as middle layer, I am thinking of doing it the same way, but:

1. Should we consolidate all these health-checks in one place to benefit from knowledge share/tooling share?
2. Where should this place be? Is it safe to be public and we can put it in Milo? Or do we need git.corp?

Happy to hear your opinions!

[npeltier]

if we consider milo as not only the place where adobe.com website is implemented, but most of our team public activity, i think we could indeed think about a space in milo where to put them (and other io actions we can set as public).
I guess some aspects of the infra should be anonymicized and then we could have common action code, common healthcheck action code.

[mokimo]

Wouldn't all the health checks just live in newrelic? I'm not sure if nala tests run on an interval, that might be an option too

[npeltier]

part that consumes services saying i'm fine or i'm not, yes, but we apparently do need to have something to :

• aggregate endpoint responses,
• check on responses content,
• compile that into a simple ok / nok with message

[3ch023]

I am not sure, maybe that's an option as well.
Something we need is to check that the server replied with a 'valid' non-empty json, not just for 500/400 status codes. If it's possible to do it in new relic directly, I am fine with that.
will try to check with @thedoc31 since the request came from him

[3ch023]

p.s. jira https://jira.corp.adobe.com/browse/MWPW-158966

[amauch-adobe]

Yes, most health checks live in New Relic where we can hit a certain url and receive a payload of the status/health. Milo Loc provided us a health check page that printed out the payload and we would check the content for the simple ok / nok messaging and alert on it.

[thedoc31]

@mokimo You're on the right path, but that's only part of the overall solution. The health checks themselves and the logic used to determine whether or not we should launch a CSO is handled by New Relic, but they need something to call on the server side to validate that things are doing what they should and return a response. That's where the health check card/page comes in.

Example:
https://my-app.adobe.com/mas/health-check.html

New Relic --> card/page --> "I'm OK" HTTP 200 --> OK
New Relic --> card/page --> "One of my components is broken" HTTP 500 --> Nexus --> CSO

More detail could also be returned as JSON:

{ 
"health_check": [
  "upstream-service-1": "ok",
  "upstream-service-2": "ok",
  "database connection": "fail",
  ...
  ]
}

[npeltier]

fyi posted this https://github.com/orgs/adobecom/discussions/3311 that could be the place for your healthcheck