ALS GET /health returns a 200 OK when status is DOWN

[lewisdaly]

Summary:
On the ALS, GET /health returns a 200 OK when status is DOWN.

This is an issue because K8s won't try and recreate the pod since it uses the status code to determine the health of pods

Severity:
(Low, Medium, High)

Priority:
(Critical, Medium, Low)

Expected Behavior

Steps to Reproduce

1. Start the ALS with a database
2. stop the database
3. call the health check:

$ curl -v -X GET $ELB_URL/account-lookup-service-admin/health                                                                             558ms  Tue 09 Feb 2021 14:48:02
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 3.10.62.177:80...
* Connected to affbc062e827c49c0a47dc1cf8f1ed70-2138950192.eu-west-2.elb.amazonaws.com (3.10.62.177) port 80 (#0)
> GET /account-lookup-service-admin/health HTTP/1.1
> Host: affbc062e827c49c0a47dc1cf8f1ed70-2138950192.eu-west-2.elb.amazonaws.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.19.2
< Date: Tue, 09 Feb 2021 04:19:01 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 155
< Connection: keep-alive
< cache-control: no-cache
< accept-ranges: bytes
< 
* Connection #0 to host affbc062e827c49c0a47dc1cf8f1ed70-2138950192.eu-west-2.elb.amazonaws.com left intact
{"status":"DOWN","uptime":1301.536941157,"startTime":"2021-02-09T04:19:00.356Z","versionNumber":"11.1.2","services":[{"name":"datastore","status":"DOWN"}]}⏎  

Observe that the response is 200 OK, even though there is no database connected.

Specifications

• Component (if known): core services
• Version: 11.1.2
• Platform: AWS
• Subsystem: ALS, Health
• Type of testing: Manual
• Bug found/raised by: @lewisdaly

PR:

• fix: proper status code for health check account-lookup-service#396

Notes:

• Severity when opened:
• Priority when opened:

[elnyry-sam-k]

Thanks for quickly fixing this @shashi165 ; I think its worth ensuring this behavior on other core services as well (quoting service, central ledger, settlement services, as well)