Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AMLII-1148 - Adding Unit test to check GC metrics emitted by each type of Java garbage collector #500

Merged
merged 24 commits into from
Dec 22, 2023
Merged

AMLII-1148 - Adding Unit test to check GC metrics emitted by each type of Java garbage collector #500

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
ba48c6d
Added ability to change the final Docker image for misbehaving-jmx-se…
carlosroman Dec 14, 2023
e352022
wip, adding GC tests
carlosroman Dec 14, 2023
96a45bc
wip: Second test that checks with JMXFetch for sanity
carlosroman Dec 14, 2023
dc55bbb
wip: Added working Old GC test
carlosroman Dec 14, 2023
f9a8680
Added option to pass JAVA_OPTS and image to use to MisbehavingJMXServer
carlosroman Dec 15, 2023
a7fdff8
Added test for default new JVM metrics
carlosroman Dec 15, 2023
22a91de
Added UseParallelGC test using new_gc_metrics: true#
carlosroman Dec 15, 2023
ed430f9
Added UseG1GC test using new_gc_metrics: true
carlosroman Dec 15, 2023
d006fd0
Added UseZGC test using new_gc_metrics: true
carlosroman Dec 15, 2023
50c6041
Adding missing change to MisbehavingJMXServer.java
carlosroman Dec 18, 2023
a1585b0
Adding a working ZGC test
carlosroman Dec 18, 2023
5ee4f54
Fixed broken test
carlosroman Dec 18, 2023
57b4d53
Fixed it so Misbehaving server runs with correct GC now
carlosroman Dec 19, 2023
bcea1a3
Small refactor
carlosroman Dec 20, 2023
fc3a762
Simple tidy up
carlosroman Dec 20, 2023
4224dc7
Small refactor to use main metrics assert function
carlosroman Dec 20, 2023
156f305
Refactor to tidy up some metrics asserts
carlosroman Dec 20, 2023
90835f9
Got TestReconnectContainer to use helper functions
carlosroman Dec 20, 2023
2484b72
Refactor + adding UseConcMarkSweepGC test
carlosroman Dec 20, 2023
5ca1719
Small refactor to remove some magic strings
carlosroman Dec 21, 2023
e8a8007
Updated docs and comments
carlosroman Dec 21, 2023
229b018
Update tools/misbehaving-jmx-server/README.md
carlosroman Dec 21, 2023
a902ae1
Updated Dockerfile comments
carlosroman Dec 21, 2023
87a8750
Updated comments and added correct TODOs
carlosroman Dec 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,13 @@
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>1.18.0</version>
<version>1.19.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
</dependencies>
Expand Down
1 change: 0 additions & 1 deletion src/main/java/org/datadog/jmxfetch/Instance.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
import java.io.InputStream;
import java.lang.management.ManagementFactory;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
Expand Down
307 changes: 307 additions & 0 deletions src/test/java/org/datadog/jmxfetch/TestGCMetrics.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,307 @@
package org.datadog.jmxfetch;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;

import java.io.IOException;
import java.util.*;

import javax.management.MBeanServerConnection;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

import org.junit.Test;

import lombok.extern.slf4j.Slf4j;

import org.datadog.jmxfetch.reporter.ConsoleReporter;
import org.datadog.jmxfetch.util.MisbehavingJMXServer;
import org.datadog.jmxfetch.util.server.SimpleAppContainer;

@Slf4j
public class TestGCMetrics extends TestCommon {

private static final int RMI_PORT = 9090;
private static final int CONTROL_PORT = 9091;
private static final int SUPERVISOR_PORT = 9092;

private static boolean isDomainPresent(final String domain, final MBeanServerConnection mbs) {
boolean found = false;
try {
final String[] domains = mbs.getDomains();
for (String s : domains) {
if(s.equals(domain)) {
found = true;
break;
}
}
} catch (IOException e) {
log.warn("Got an exception checking if domain is present", e);
}
return found;
}

/*
Just here to make sure I've not broken anything
*/
@Test
public void testJMXDirectBasic() throws Exception {
try (final MisbehavingJMXServer server = new MisbehavingJMXServer(RMI_PORT, CONTROL_PORT,
SUPERVISOR_PORT)) {
server.start();
final String ipAddress = server.getIp();
final String remoteJmxServiceUrl = String.format(
"service:jmx:rmi:///jndi/rmi://%s:%s/jmxrmi", ipAddress, RMI_PORT);
final JMXServiceURL jmxUrl = new JMXServiceURL(remoteJmxServiceUrl);
final JMXConnector conn = JMXConnectorFactory.connect(jmxUrl);
final MBeanServerConnection mBeanServerConnection = conn.getMBeanServerConnection();
assertThat(isDomainPresent("Bohnanza", mBeanServerConnection), is(true));
}
}

@Test
public void testJMXFetchBasic() throws IOException {
try (final MisbehavingJMXServer server = new MisbehavingJMXServer(RMI_PORT, CONTROL_PORT,
SUPERVISOR_PORT)) {
server.start();
final String ipAddress = server.getIp();
this.initApplicationWithYamlLines(
"init_config:",
" is_jmx: true",
"",
"instances:",
" - name: jmxint_container",
" host: " + ipAddress,
" collect_default_jvm_metrics: false",
" max_returned_metrics: 300000",
" port: " + RMI_PORT,
" conf:",
" - include:",
" domain: Bohnanza"
);
this.app.doIteration();
final List<Map<String, Object>> metrics = ((ConsoleReporter) this.appConfig.getReporter()).getMetrics();
assertThat(metrics, hasSize(1));
}
}

@Test
public void testDefaultOldGC() throws IOException {
try (final MisbehavingJMXServer server = new MisbehavingJMXServer(RMI_PORT, CONTROL_PORT,
SUPERVISOR_PORT)) {
server.start();
final String ipAddress = server.getIp();
this.initApplicationWithYamlLines("init_config:",
" is_jmx: true",
"",
"instances:",
" - name: jmxint_container",
" host: " + ipAddress,
" collect_default_jvm_metrics: true",
" max_returned_metrics: 300000",
" port: " + RMI_PORT);
this.app.doIteration();
final List<Map<String, Object>> actualMetrics = ((ConsoleReporter) appConfig.getReporter()).getMetrics();
List<String> gcGenerations = Arrays.asList(
"G1 Old Generation",
"G1 Young Generation");
assertGCMetric(actualMetrics, "jvm.gc.cms.count", gcGenerations);
assertGCMetric(actualMetrics, "jvm.gc.parnew.time", gcGenerations);
}
}

@Test
public void testDefaultNewGCMetricsUseParallelGC() throws IOException, InterruptedException {
try (final MisbehavingJMXServer server = new MisbehavingJMXServer(
MisbehavingJMXServer.JDK_11,
"-XX:+UseParallelGC",
RMI_PORT,
CONTROL_PORT,
SUPERVISOR_PORT)) {
server.start();
final String ipAddress = server.getIp();
this.initApplicationWithYamlLines("init_config:",
" is_jmx: true",
" new_gc_metrics: true",
"",
"instances:",
" - name: jmxint_container",
" host: " + ipAddress,
" collect_default_jvm_metrics: true",
" max_returned_metrics: 300000",
" port: " + RMI_PORT);
// Run one iteration first
this.app.doIteration();
// And then pull get the metrics or else reporter does not have correct number of metrics
((ConsoleReporter) appConfig.getReporter()).getMetrics();

// Actual iteration we care about
this.app.doIteration();
final List<Map<String, Object>> actualMetrics = ((ConsoleReporter) appConfig.getReporter()).getMetrics();
assertThat(actualMetrics, hasSize(13));
final List<String> gcYoungGenerations = Collections.singletonList(
"PS Scavenge");
carlosroman marked this conversation as resolved.
Show resolved Hide resolved
assertGCMetric(actualMetrics, "jvm.gc.minor_collection_count", gcYoungGenerations);
assertGCMetric(actualMetrics, "jvm.gc.minor_collection_time", gcYoungGenerations);
final List<String> gcOldGenerations = Collections.singletonList(
"PS MarkSweep");
assertGCMetric(actualMetrics, "jvm.gc.major_collection_count", gcOldGenerations);
assertGCMetric(actualMetrics, "jvm.gc.major_collection_time", gcOldGenerations);
}
}

@Test
public void testDefaultNewGCMetricsUseG1GC() throws IOException, InterruptedException {
try (final MisbehavingJMXServer server = new MisbehavingJMXServer(
MisbehavingJMXServer.JDK_17,
"-XX:+UseG1GC",
RMI_PORT,
CONTROL_PORT,
SUPERVISOR_PORT)) {
server.start();
final String ipAddress = server.getIp();
this.initApplicationWithYamlLines("init_config:",
" is_jmx: true",
" new_gc_metrics: true",
"",
"instances:",
" - name: jmxint_container",
" host: " + ipAddress,
" collect_default_jvm_metrics: true",
" max_returned_metrics: 300000",
" port: " + RMI_PORT);
// Run one iteration first
this.app.doIteration();
// And then pull get the metrics or else reporter does not have correct number of metrics
((ConsoleReporter) appConfig.getReporter()).getMetrics();

// Actual iteration we care about
this.app.doIteration();
final List<Map<String, Object>> actualMetrics = ((ConsoleReporter) appConfig.getReporter()).getMetrics();
assertThat(actualMetrics, hasSize(13));
final List<String> gcYoungGenerations = Collections.singletonList(
"G1 Young Generation");
assertGCMetric(actualMetrics, "jvm.gc.minor_collection_count", gcYoungGenerations);
assertGCMetric(actualMetrics, "jvm.gc.minor_collection_time", gcYoungGenerations);
final List<String> gcOldGenerations = Collections.singletonList(
"G1 Old Generation");
assertGCMetric(actualMetrics, "jvm.gc.major_collection_count", gcOldGenerations);
assertGCMetric(actualMetrics, "jvm.gc.major_collection_time", gcOldGenerations);
}
}

@Test
public void testDefaultNewGCMetricsUseZGC() throws IOException {
try (final SimpleAppContainer container = new SimpleAppContainer(
"eclipse-temurin:17",
"-XX:+UseZGC -Xmx128M -Xms128M",
RMI_PORT
)){
container.start();
final String ipAddress = container.getIp();
this.initApplicationWithYamlLines("init_config:",
" is_jmx: true",
" new_gc_metrics: true",
"",
"instances:",
" - name: jmxint_container",
" host: " + ipAddress,
" collect_default_jvm_metrics: true",
" max_returned_metrics: 300000",
" port: " + RMI_PORT);
// Run one iteration first
this.app.doIteration();
// And then pull get the metrics or else reporter does not have correct number of metrics
((ConsoleReporter) appConfig.getReporter()).getMetrics();

// Actual iteration we care about
this.app.doIteration();
final List<Map<String, Object>> actualMetrics = ((ConsoleReporter) appConfig.getReporter()).getMetrics();
assertThat(actualMetrics, hasSize(13));
final List<String> zgcPause = Collections.singletonList(
"ZGC Pauses");
assertGCMetric(actualMetrics, "jvm.gc.zgc_pauses_collection_count", zgcPause);
assertGCMetric(actualMetrics, "jvm.gc.zgc_pauses_collection_time", zgcPause);
final List<String> zgcCycles = Collections.singletonList(
"ZGC Cycles");
assertGCMetric(actualMetrics, "jvm.gc.zgc_cycles_collection_count", zgcCycles);
assertGCMetric(actualMetrics, "jvm.gc.zgc_cycles_collection_time", zgcCycles);
}
}

// @Ignore("Can not force ZGC to work using MisbehavingJMXServer")
carlosroman marked this conversation as resolved.
Show resolved Hide resolved
@Test
public void testDefaultNewGCMetricsUseZGCOld() throws IOException, InterruptedException {
try (final MisbehavingJMXServer server = new MisbehavingJMXServer(
MisbehavingJMXServer.JDK_17,
"-XX:+UseZGC",
RMI_PORT,
CONTROL_PORT,
SUPERVISOR_PORT)) {
server.start();
final String ipAddress = server.getIp();
this.initApplicationWithYamlLines("init_config:",
" is_jmx: true",
" new_gc_metrics: true",
"",
"instances:",
" - name: jmxint_container",
" host: " + ipAddress,
" collect_default_jvm_metrics: true",
" max_returned_metrics: 300000",
" port: " + RMI_PORT);
// Run one iteration first
this.app.doIteration();
// And then pull get the metrics or else reporter does not have correct number of metrics
((ConsoleReporter) appConfig.getReporter()).getMetrics();

// Actual iteration we care about
this.app.doIteration();
final List<Map<String, Object>> actualMetrics = ((ConsoleReporter) appConfig.getReporter()).getMetrics();
assertThat(actualMetrics, hasSize(13));
final List<String> zgcPause = Collections.singletonList(
"ZGC Pauses");
assertGCMetric(actualMetrics, "jvm.gc.zgc_pauses_collection_count", zgcPause);
assertGCMetric(actualMetrics, "jvm.gc.zgc_pauses_collection_time", zgcPause);
final List<String> zgcCycles = Collections.singletonList(
"ZGC Cycles");
assertGCMetric(actualMetrics, "jvm.gc.zgc_cycles_collection_count", zgcCycles);
assertGCMetric(actualMetrics, "jvm.gc.zgc_cycles_collection_time", zgcCycles);
}
}

private static void assertGCMetric(final List<Map<String, Object>> actualMetrics,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this be implemented in terms of the above assertGCMetric? It looks like most of the tests use the above version

final String expectedMetric,
final List<String> gcGenerations) {
final List<Map<String, Object>> filteredMetrics = new ArrayList<>();
for (Map<String, Object> actualMetric : actualMetrics) {
final String name = (String) actualMetric.get("name");
if(expectedMetric.equals(name)) {
filteredMetrics.add(actualMetric);
}
}
assertThat(filteredMetrics, hasSize(gcGenerations.size()));
for (final String name : gcGenerations) {
log.debug("Asserting for metric '{}'", name);
scottopell marked this conversation as resolved.
Show resolved Hide resolved
boolean found = false;
for (Map<String, Object> filteredMetric : filteredMetrics) {
final Set<String> mTags = new HashSet<>(
Arrays.asList((String[]) (filteredMetric.get("tags"))));

if(mTags.contains(String.format("name:%s", name))) {
assertThat(mTags, not(empty()));
assertThat(mTags, hasSize(5));
log.debug("mTags '{}' has size: {}\n{}", name, mTags.size(), mTags);
assertThat(mTags, hasItems(
"instance:jmxint_container",
"jmx_domain:java.lang",
"type:GarbageCollector",
String.format("name:%s", name)));
found = true;
}
}
assertThat(String.format("Did not find metric '%s'", name), found, is(true));
}
}
}
1 change: 0 additions & 1 deletion src/test/java/org/datadog/jmxfetch/TestReconnectContainer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestRule;
Expand Down
Loading
Loading