OOM on API Key change

[agustiza]

Description

Running inside docker containers.

We recently had an API key rotation for the new relic agent on our non productive environments as a way to save costs by disabling monitoring and logging.

We suddenly found out that we were having constant OOMs. Troubleshooting a bit we found out that the culprit was the New Relic Agent.

Sadly I cannot share a reproducible project.

Troubleshooting or NR Diag results

Steps to Reproduce

Very little I can share but it's pretty standard setup.

## Dockerfile
RUN wget "http://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip"
RUN unzip newrelic-java.zip -d "/app"

##Bash Script that runs on entrypoint
if [ "$USE_NEW_RELIC" = "true" ]; then
        echo -n "Setting New Relic..."
        if [ -f /run/secrets/newrelic_license_key ]; then
            NEWRELIC_LICENSE_KEY=$(cat /run/secrets/newrelic_license_key)
            NEW_RELIC="-javaagent:/app/newrelic/newrelic.jar"
            NEW_RELIC=$NEW_RELIC" -Dnewrelic.environment="${ENVIRONMENT}
            NEW_RELIC=$NEW_RELIC" -Dnewrelic.config.license_key="${NEWRELIC_LICENSE_KEY}
            NEW_RELIC=$NEW_RELIC" -Dnewrelic.config.app_name="${ENVIRONMENT}_${SERVICE_NAME}
            echo "New Relic set."
        else
            echo "New Relic licence key is not set."
            exit 1
        fi
    fi

apk add openjdk8
jps #pid 547
jcmd 547 GC.class_histogram | less
/app # jstat -gcutil 547 250 10
  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00  99.68  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113654 254514.054 256021.499
  0.00   0.00 100.00  99.99  97.34  95.57 164830 1507.445 113655 254516.281 256023.726

Your Environment

Docker container using openjdk:8-jre-alpine openjdk:8-jdk-alpine
Current New Relic Agent version

[workato-integration]

https://issues.newrelic.com/browse/NR-147356

[kford-newrelic]

@agustiza can you confirm : when you did the key rotation, was the new key a valid or invalid key?

[agustiza]

I'm not 100% sure but I think it was a purposedly invalid key.

Should be around 25-26th April 2023.