Set up New Relic service

[JeanMarie-PM]

No description provided.

[asteel-gsa]

@JeanMarie-TTS How do we want to handle the New Relic Service?

Manifest - is one option that we can use.

Python Agent - Looks like another.

If we use the agent, we would likely need to have the following:

1. GSA New Relic Access, where multiple users can access and view the data, with a single user generating the License Key.
2. requirements.in modified to include newrelic, recompile the requirements.txt for deploying the agent.
3. Locally create the ini, with newrelic-admin generate-config $YOUR_LICENSE_KEY newrelic.ini
4. Determine how we will keep the secret $YOUR_LICENSE_KEY and pass it to the ini file.
5. Use the following command $ NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program gunicorn -c config.py config.wsgi (I think) in deploy-apps.yml

Do you have some insight to this methodology for installing the New Relic Service? I believe we can mock this by having a new relic account, and generating our own License Key, but the problem I am unsure about is if GSA has a New Relic provisioning service where we can request access to the dashboard, so that multiple people can view the metrics.

CC: @ChrisB-16 @mogul @tadhg-ohiggins

[mogul]

The manifest option is only for when you're using the PHP buildpack. The Python integration is about like you described.

I'm working on getting the license key sorted out... TTS has group licensing and there's an internal process that I'm pursuing to get the FAC account set up.

[asteel-gsa]

The manifest option is only for when you're using the PHP buildpack. The Python integration is about like you described.

I'm working on getting the license key sorted out... TTS has group licensing and there's an internal process that I'm pursuing to get the FAC account set up.

Gotcha, okay, so we will just leave it in backlog and plan for it when the license key and account is setup.

[mogul]

For Step 4 (actually 3), we provide the license key to the app via its environment using a user-provided service.

For Step 3 (actually 4), we generate the config file before the start command is invoked by calling the utility in the .profile file.

[asteel-gsa]

Status of preliminary testing with the python agent for new relic:

NOTE: this was done locally, on the docker containers, to ensure data was infact being submitted, and understand the workflow for the agent installation.

References: pypi new relic & NR Docs

• Added newrelic to requirements.in and did a pip-compile to update the requirements.txt
• Spun up the local instance with docker compose build and docker compose up
• Because newrelic-admin generate-config command requires pwd which is a UNIX native, I had to use the backend-web terminal to generate my newrelic.ini and copy over to my workspace.
• This has not been commited into code, because the license key is hardcoded at present, to establish a baseline functionality
• Inside the Dockerfile before COPY src/ I added ENTRYPOINT ["newrelic-admin", "run-program"] to allow the newrelic package to run
• Inside manage.py I added (this) as Dockerfile calls RUN npm run build && python manage.py collectstatic just to be safe, though I am unsure if this is a necessary step, and will likely need to be modified for the actual agent when running on CG

import newrelic.agent
newrelic.agent.initialize('./newrelic.ini')

• Finally, I rebuilt the containers, and let it sit for a while, which ultimately, returned the following in the output of docker compose up:

backend-web-1          | 2023-04-27 13:33:36,459 (82/Thread-1 (process_request_thread)) newrelic.core.agent INFO - New Relic Python Agent (8.8.0)
backend-web-1          | INFO "GET / HTTP/1.1" 200 18736
backend-web-1          | INFO "GET / HTTP/1.1" 200 18736
backend-web-1          | 2023-04-27 13:33:37,844 (82/NR-Activate-Session/GSA TTS Test) newrelic.core.agent_protocol INFO - Reporting to: <Link Omitted>

• All of which, gains the following:

Testing:

• Stopped the docker containers and re-composed them, and after Clamav is successfully running, returns new relic agent running

Gracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Running 4/4
 ✔ Container backend-clamav-rest-1  Stopped                                                                                                   10.8s 
 ✔ Container backend-web-1          Stopped                                                                                                   10.5s 
 ✔ Container localstack_main        Stopped                                                                                                    0.7s 
 ✔ Container backend-db-1           Stopped                                                                                                    0.4s 
canceled

backend-clamav-rest-1  | Thu Apr 27 16:09:15 2023 -> Set stacksize to 1048576
backend-clamav-rest-1  | Clamd version: "ClamAV 0.105.2/26889/Thu Apr 27 09:25:48 2023"
backend-clamav-rest-1  | Connected to clamd on tcp://localhost:3310
backend-web-1          | 2023-04-27 14:09:35,919 (85/Thread-1 (process_request_thread)) newrelic.core.agent INFO - New Relic Python Agent (8.8.0)
backend-web-1          | INFO "GET / HTTP/1.1" 302 0
backend-web-1          | INFO "GET / HTTP/1.1" 302 0
backend-web-1          | INFO "GET /audit/ HTTP/1.1" 200 21524
backend-web-1          | INFO "GET /audit/ HTTP/1.1" 200 21524
backend-web-1          | 2023-04-27 14:09:37,101 (85/NR-Activate-Session/GSA TTS Test) newrelic.core.agent_protocol INFO - Reporting to: <Link Ommitted>

[mogul]

Now blocked on New Relic license availability

[JeanMarie-PM]

@asteel-gsa

1. It's not a good idea to modify manage.py. You should make the necessary changes in settings.py in the config folder. This should have the same effect.
2. How do I access the New Relic dashboard?
3. How does New Relic access the log info? Does it access the django logs directly? Or does it periodically read them into its own persistence layer? Or something else altogether?

[asteel-gsa]

@JeanMarie-TTS

1. It's not a good idea to modify manage.py. You should make the necessary changes in settings.py in the config folder. This should have the same effect.

Moving

import newrelic.agent
newrelic.agent.initialize('./newrelic.ini')

into config/settings.py seems to work fine locally

2. How do I access the New Relic dashboard?

As of the moment, it is not possible unless you want to be added to my account (which I am not sure how to do) or, the steps above here can be used to implement this into your local work environment. When you do docker compose up, if you have your own newrelic account and license key (free) then you can simply to New Relic and view the dashboard

3. How does New Relic access the log info? Does it access the django logs directly? Or does it periodically read them into its own persistence layer? Or something else altogether?

New relic has its own built in logging functionality, and I am attempting to get rsyslog working as well in a local dockerized instance, but is having some issues. While this did work, on a functional level, it did not successfully create the Network > syslog reporting on the dashboard

[asteel-gsa]

@mogul

I think I managed to get syslog implemented. So I was messing around with some things, and I think this is effectively what this page was doing.

So with the rsyslog, I threw it all into the dockerfile and attempted to atleast see the functionality of it.

The last lines were done with this logger "testing rsyslog agentless configuration" on the docker container of backend-web however, even without rsyslog these are the kinds of logs we are seeing:

I am unsure if that is what we are going for, but from Network > Syslogs it appears to be a different solution

added the following to Dockerfile to enable this functionality

ENTRYPOINT ["newrelic-admin", "run-program"]
RUN apt-get -y install rsyslog-gnutls ca-certificates
COPY newrelic.conf /etc/rsyslog.d/
RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf

[asteel-gsa]

Per @JeanMarie-TTS I have uploaded necessary files here

I was unable to get $NEW_RELIC_LICENSE_KEY passed into the .ini on the docker container and as such, for now, has to be manually input here for syslog and here for the license key

In order to use the dashboard locally, navigate here, create an account, grab your License Key, and replace it as necessary. Moving NEW_RELIC_LICENSE_KEY into .env and docker-compose.yml did not seem to work for me.

[mogul]

@asteel-gsa and I paired on setting up the cloud.gov side of things in #1077 and that should be good to go. Once that merges the agent will be shipping telemetry to NR from all of our spaces, which should be enough to have us ready for assessment.

We may need additional PRs to set things up for local development, or to customize settings applied per-space.

[mogul]

We've set up the agent but the license key we have isn't working. In the logs we see:

ERROR - Data collector is indicating that an incorrect license key has been supplied by the agent. Please correct any problem with the license key or report this problem to New Relic support.

2023-05-11 21:36:27,804 (188/NR-Activate-Session/Python Agent Test) newrelic.core.agent_protocol ERROR - Data collector is indicating that an incorrect license key has been supplied by the agent. Please correct any problem with the license key or report this problem to New Relic support.
2023-05-11 21:36:27,805 (188/NR-Activate-Session/Python Agent Test) newrelic.core.application DEBUG - Retrying registration of the application 'Python Agent Test' with the data collector after a further 15 seconds.

We've run the troubleshooting steps that New Relic has listed in their support docs, but even after verifying that our key is accurate, we see the following:

$ env | grep ^NEW
NEW_RELIC_LICENSE_KEY=[REDACTED]
NEW_RELIC_LOG=stdout
NEW_RELIC_APP_NAME=gsa-fac
NEW_RELIC_ENVIRONMENT=dev
$ 
$ 
$ newrelic-admin validate-config -

Running Python agent test.

Any significant errors in performing the test will be shown below. If no
errors occurred in the execution of this script and data is still not
reporting through to the UI against the application:

  Python Agent Test

after 5 minutes then check the log file:

  /tmp/python-agent-test.log

for debugging information. Supply the log file to New Relic support if
requesting help with resolving any issues with the test not reporting
data to the New Relic UI.

ERROR - Data collector is indicating that an incorrect license key has been supplied by the agent. Please correct any problem with the license key or report this problem to New Relic support.

ERROR - Data collector is indicating that an incorrect license key has been supplied by the agent. Please correct any problem with the license key or report this problem to New Relic support.

ERROR - Unable to register application for test, connection could not be established within 30.0 seconds.

I've opened a support case with New Relic.

[asteel-gsa]

With this PR the service 'gsa-fac' is now visible in new relic