Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new DDR commands for modularity__Command 1 and 2 #1987

Merged
merged 1 commit into from
Jun 12, 2018

Conversation

Ailloviee
Copy link
Contributor

@Ailloviee Ailloviee commented May 24, 2018

Added two commands:

  1. !findallmodules Outputs all the modules loaded by the runtime
  2. findmodulebyname <moduleName> find the modules corresponding to its module name
    Related issue: Add new DDR commands for modularity #1859

Refactored ObjectModel.java and CheckEngine.java to make some pointer check helper functions public static.

Made change in PointerGenerator.java as DDR wouldn't compile without the change, for detail look at: #1973

Signed-off-by: Charles_Zheng Juntian.Zheng@ibm.com

Examples:
findallmodules:

> !findallmodules
jdk.javadoc                    !j9module 0x000001305F474498
jdk.attach                     !j9module 0x000001305F478798
java.prefs                     !j9module 0x000001305F478928
java.sql.rowset                !j9module 0x000001305F4789C8
jdk.security.auth              !j9module 0x000001305F478BF8
jdk.editpad                    !j9module 0x000001305F478D88
jdk.crypto.cryptoki            !j9module 0x000001305F478E78
java.rmi                       !j9module 0x000001305F478F68
jdk.httpserver                 !j9module 0x000001305F479468
jdk.internal.jvmstat           !j9module 0x000001305F4795A8
java.datatransfer              !j9module 0x000001305F476668
jdk.scripting.nashorn          !j9module 0x000001305F4767A8
jdk.jdwp.agent                 !j9module 0x000001305F477248
openj9.cuda                    !j9module 0x000001305F477298
java.logging                   !j9module 0x000001305F477388
openj9.dataaccess              !j9module 0x000001305F477CC8
jdk.naming.rmi                 !j9module 0x000001305F477D68
jdk.crypto.ec                  !j9module 0x000001305F477E58
java.naming                    !j9module 0x000001305F477EF8
jdk.management                 !j9module 0x000001305F478448
jdk.jlink                      !j9module 0x000001305F4699A8
jdk.jconsole                   !j9module 0x000001305F469D18
jdk.accessibility              !j9module 0x000001305F469EA8
openj9.dtfj                    !j9module 0x000001305F469FE8
openj9.zosconditionhandling    !j9module 0x000001305F4710C8
jdk.localedata                 !j9module 0x000001305F471168
jdk.charsets                   !j9module 0x000001305F471398
java.smartcardio               !j9module 0x000001305F471438
openj9.jvm                     !j9module 0x000001305F471528
jdk.jstatd                     !j9module 0x000001305F4715C8
java.security.jgss             !j9module 0x000001305F471708
jdk.net                        !j9module 0x000001305F471CF8
jdk.jshell                     !j9module 0x000001305F471D98
java.management                !j9module 0x000001305F468B18
jdk.internal.ed                !j9module 0x000001305F4691A8
jdk.management.agent           !j9module 0x000001305F469298
jdk.crypto.mscapi              !j9module 0x000001305F469478
jdk.jdeps                      !j9module 0x000001305F469518
java.xml                       !j9module 0x000001305F46BCB8
jdk.internal.opt               !j9module 0x000001305F46ABA8
jdk.compiler                   !j9module 0x000001305F46ACE8
jdk.jartool                    !j9module 0x000001305F46B6E8
jdk.internal.le                !j9module 0x000001305F46B8C8
jdk.unsupported                !j9module 0x000001305F4684A8
jdk.dynalink                   !j9module 0x000001305F4685E8
java.scripting                 !j9module 0x000001305F46DD98
java.se                        !j9module 0x000001305F46DE88
java.security.sasl             !j9module 0x000001305F46DED8
java.management.rmi            !j9module 0x000001305F46E0B8
java.desktop                   !j9module 0x000001305F46E1F8
jdk.naming.dns                 !j9module 0x000001305F46CCD8
java.xml.crypto                !j9module 0x000001305F46CDC8
jdk.sctp                       !j9module 0x000001305F46D868
jdk.security.jgss              !j9module 0x000001305F46D958
jdk.jsobject                   !j9module 0x000001305F46DA48
java.instrument                !j9module 0x000001305F46DB38
java.sql                       !j9module 0x00000130550DB798
jdk.zipfs                      !j9module 0x00000130550DB8D8
openj9.gpu                     !j9module 0x00000130550DB978
openj9.sharedclasses           !j9module 0x00000130550DBA68
jdk.xml.dom                    !j9module 0x00000130550DBB58
java.compiler                  !j9module 0x00000130550DBCE8
jdk.jdi                        !j9module 0x00000130550DBF18
java.base                      !j9module 0x00000130550C7E88

findmodulebyname:

> !findmodulebyname java.base
Searching for modules named 'java.base' in VM=13053677e00
java.base			!j9module 0x00000130550C7E88
Found 1 module(s) named java.base

Also, added test cases for the two commands

@@ -274,7 +274,7 @@ private void generateClass(StructureDescriptor structure) throws IOException {
File javaFile = new File(outputDir, structure.getPointerName() + ".java");

// Do not create Pointer class for Constant only structures (this includes enums)
if (structure.getFields().size() == 0 && structure.getConstants().size() != 0 && superClassName.equals("StructurePointer")) {
if (structure.getFields().size() == 0 && structure.getConstants().size() > 1 && superClassName.equals("StructurePointer")) {
Copy link
Contributor

@tajila tajila May 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

take out these changes Keith has submitted a fix for this

}
}

/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix indentation

@tajila
Copy link
Contributor

tajila commented May 24, 2018

@Ailloviee please post an example of what the two commands output in this PR

@tajila
Copy link
Contributor

tajila commented May 24, 2018

@dmitripivkine Please take a look at this PR, it refactors some of the checkEngine and ObjectModel code

J9ModulePointer modulePtr = null;
while (poolIterator.hasNext()) {
modulePtr = poolIterator.next();
if (ObjectModel.isPointerInHeap(vm, modulePtr.moduleName())) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a comment saying why this check must be done

out.printf("%s !j9module %s\n", moduleName, hexAddress);
}
}
} catch (Exception/* CorruptDataException */ e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if CorruptDataException is the only exception that can be thrown in the try block then specify it in the catch block

@tajila
Copy link
Contributor

tajila commented May 25, 2018

@Ailloviee can you fix the ObjectModel diff, its too large

@Ailloviee
Copy link
Contributor Author

Sorry I tried to squash the commits but I did it wrong and there was merge conflicts

@dmitripivkine dmitripivkine self-requested a review May 28, 2018 22:40
Copy link
Contributor

@dmitripivkine dmitripivkine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GC DDR changes LGTM

@Ailloviee Ailloviee force-pushed the DDRCommands_v1 branch 2 times, most recently from 6ce3e24 to 794a26d Compare May 29, 2018 20:50
@Ailloviee Ailloviee changed the title Add new DDR commands for modularity Add new DDR commands for modularity__Command 1 and 2 May 30, 2018
@Ailloviee Ailloviee force-pushed the DDRCommands_v1 branch 2 times, most recently from c7dffa6 to 6cf6c22 Compare May 30, 2018 17:40
return null;
}

// TODO kmt : this doesn't belong here
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this comment, no longer applies since you've moved it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also add javadocs

* any abstract pointer
* @return If the AbstractPointer pointer is in stored in heap.
*/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this space

@@ -362,4 +369,69 @@ public static VoidPointer getElementAddress(J9IndexableObjectPointer indexableOb
{
return gcObjectModel.getElementAddress(indexableObjectPointer, elementIndex, elementSize);
}

public static GCHeapRegionDescriptor findRegionForPointer(J9JavaVMPointer javaVM, GCHeapRegionManager hrm, AbstractPointer pointer, GCHeapRegionDescriptor region)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add javadocs for this

} catch (CorruptDataException cde) {}
return null;
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add javadocs

* <p>
* @param javaVM
* J9JavaVMPointer
* pointer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatting looks off here, add the @param for each parameter

* java.prefs !j9module 0x000001305F478928
* .....(A list of every module loaded by the runtime)
*/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove space

/*
* Since both packages and modules could be in the pool,
* the iterator checks on the region and make sure that it is
* pointing to a module. (For modules, the pointer must be in heap)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try to be more specific and mention that we distinguish between module and j9package by checking if the first field is in the heap. Since the first field for j9module is an object and the first field for j9package is a J9UTF8

public static GCHeapRegionDescriptor regionForAddress(J9JavaVMPointer javaVM, GCHeapRegionManager hrm, AbstractPointer pointer)
{
try {
if(null == hrm) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be a space after the if

if (null ...


public static boolean isPointerInHeap(J9JavaVMPointer javaVM, AbstractPointer pointer)
{
if(findRegionForPointer(javaVM, null, pointer, null) != null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be space after if

if(findRegionForPointer(javaVM, null, pointer, null) != null) {
return true;
}
else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

} else {

@Ailloviee Ailloviee force-pushed the DDRCommands_v1 branch 2 times, most recently from 6f70363 to a8df173 Compare May 30, 2018 19:18
@@ -358,8 +365,104 @@ public static UDATA getHeaderSize(J9ObjectPointer object) throws CorruptDataExce
return gcObjectModel.getHeaderSize(object);
}

/**
* Returns the address of an element.
* <p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont need this


/**
* Returns the heap region of a pointer.
* <p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont need this


/**
* Returns the heap region for address of a pointer.
* <p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for the rest

@tajila
Copy link
Contributor

tajila commented May 30, 2018

@keithc-ca can you please review these changes

@keithc-ca keithc-ca self-assigned this May 31, 2018
Copy link
Contributor

@keithc-ca keithc-ca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have the new commands been tested with a core file produced by Java 8?

* J9UTF8)
*/
if (ObjectModel.isPointerInHeap(vm, modulePtr.moduleName())) {
String moduleName = J9ObjectHelper.stringValue(modulePtr.moduleName());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please pull the expression modulePtr.moduleName() out of the loop.

{
public FindModuleByNameCommand()
{
addCommand("findmodulebyname","<moduleName>","find the modules corresponding to its module name");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (formatting): space after comma

while (poolIterator.hasNext()) {
modulePtr = poolIterator.next();
if (ObjectModel.isPointerInHeap(vm, modulePtr.moduleName())
&& pattern.isMatch(J9ObjectHelper.stringValue(modulePtr.moduleName()))) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please extract the expression modulePtr.moduleName() into a local variable.

@Ailloviee
Copy link
Contributor Author

@keithc-ca So far I've only tested with core file generated by Java 10, can you please teach me how to generate core file using Java 8? -Xdump does not seem to work on Java 8

@pshipton
Copy link
Member

pshipton commented May 31, 2018

-Xdump does not seem to work on Java 8

What are you trying? -Xdump works the same on Java 8 as it does on Java 10, although Java 8 doesn't support modules. I assume Keith is asking you to check the commands fail gracefully on a Java 8 core.
"java -Xdump:system:events=vmstop" is one way, create a core on shutdown.
"java -Xdump:system:events=user" is another if you have something long running, then send the QUIT signal "kill -QUIT " to create a core file.

@tajila
Copy link
Contributor

tajila commented May 31, 2018

@Ailloviee when you are testing with Java8 make sure its a J9 or Openj9 and not hotspot

@keithc-ca
Copy link
Contributor

@pshipton is right: I want to know that you've tried the commands on a core file from Java 8, that it behaves 'gracefully', and what that graceful behavior is.

@Ailloviee
Copy link
Contributor Author

@pshipton @keithc-ca for now it fails but not gracefully, working on the version handling now

@Ailloviee Ailloviee force-pushed the DDRCommands_v1 branch 3 times, most recently from 29613cb to 6218399 Compare June 1, 2018 17:33
@AdamBrousseau
Copy link
Contributor

Jenkins signed off by check

@Ailloviee
Copy link
Contributor Author

@keithc-ca @pshipton New commands now fail gracefully on Java 8, example:

> !findallmodules
This command only works with Java version 9 or higher
The current Java version is: Java 1.8

import com.ibm.j9ddr.vm29.types.UDATA;

/**
* JavaVersionHelper helps check if the JVM version is new enough for the modularity DDR commands
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the part about "modularity DDR commands". This class should be general enough for other purposes

public final static int J2SE_18 = 0x0800;
public final static int J2SE_19 = 0x0900;
public final static int J2SE_V10 = 0x0A00;
public final static int J2SE_V11 = 0x0B00;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see this written so it doesn't need attention with every new version.
The comments in j2sever.h suggest vm.j2seVersion().bitAnd(J2SE_VERSION_MASK) will give the right answer until Java version 255. The J2SE_* constants for 8-11 are thus not necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vm.j2seVersion().bitAnd(J2SE_VERSION_MASK) returns the hex version like 0x0800, is it fine if I just print it out like that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version will be vm.j2seVersion().bitAnd(J2SE_VERSION_MASK).shiftRight(J2SE_JAVA_SPEC_VERSION_SHIFT).

We should probably arrange that those constants are available via generated DDR code. It could be that only a DDR annotation is required in j2sever.h.

UDATA javaVersion = vm.j2seVersion().bitAnd(J2SE_SERVICE_RELEASE_MASK);
if (javaVersion.lt(J2SE_19)) {
out.printf("This command only works with Java version 9 or higher\n"
+ "The current Java version is: %s\n", convertVersionToString(javaVersion.intValue()));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message should make it clearer that it's the version of the VM that created the core file being examined that is relevant (not the SDK being used to examine that core file).

return true;
}

private static String convertVersionToString(int version)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also be generalized so it won't need attention with each Java version.

if (ObjectModel.isPointerInHeap(vm, moduleNameObjPtr)) {
String moduleName = J9ObjectHelper.stringValue(moduleNameObjPtr);
String hexAddress = modulePtr.getHexAddress();
out.printf("%s !j9module %s\n", moduleName, hexAddress);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strings should not have internal tabs; a better solution would be to use %-20s (or other suitable width).

* iterator checks on the region and make sure that it is
* pointing to a module. (For modules, the first field must be
* in heap since it is an object while for j9package it is a
* J9UTF8)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Please finish the sentence with a period.

J9ObjectPointer moduleNameObjPtr = null;
PatternString pattern = new PatternString(searchModuleName);

out.println(String.format("Searching for modules named '%1$s' in VM=%2$s", searchModuleName,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why use %$1s instead of just %s? The former is only helpful when arguments are used out of order or are used repeatedly.

}
}

/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Formatting and inconsistent indenting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what the indentation issue is here..

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good now, so whatever I was commenting about has been addressed.

/* Even if hrm is null, all helpers that use it will null check it and attempt to allocate it and handle the failure */
MM_HeapRegionManagerPointer hrmPtr = MM_GCExtensionsPointer.cast(_javaVM.gcExtensions()).heapRegionManager();
_hrm = GCHeapRegionManager.fromHeapRegionManager(hrmPtr);
} catch (CorruptDataException cde) {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exception ought not to be ignored: I expect many NullPointerExceptions or similar problems if _hrm is null.

@Ailloviee Ailloviee force-pushed the DDRCommands_v1 branch 3 times, most recently from fb08b31 to 4d41e58 Compare June 4, 2018 17:54
}
}

/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good now, so whatever I was commenting about has been addressed.


String moduleName = J9ObjectHelper.stringValue(moduleNameObjPtr);
String hexAddress = modulePtr.getHexAddress();
out.printf("%-30s!j9module %s\n", moduleName, hexAddress);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be at least one space between %-30s and !j9module so a module name of 30 characters or more is still separated from the command. You should also use %n instead of \n.

if (ObjectModel.isPointerInHeap(vm, moduleNameObjPtr)) {
String moduleName = J9ObjectHelper.stringValue(moduleNameObjPtr);
String hexAddress = modulePtr.getHexAddress();
out.printf("%-30s!j9module %s\n", moduleName, hexAddress);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a space between the fields and use %n for the line-end.

out.printf("%-30s!j9module %s\n", moduleName, hexAddress);
}
}
out.println(String.format("Found %d module(s) named %s", hitCount, searchModuleName));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace %s with '%s' for consistency with earlier message.

*/
private void printUsage(PrintStream out)
{
out.println("findmodulebyname <moduleName> - find the module corresponding to its module name");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The help and this usage message should make it clear that the argument is a name pattern.

@Ailloviee Ailloviee force-pushed the DDRCommands_v1 branch 6 times, most recently from deba172 to 5568562 Compare June 7, 2018 15:24
int javaVersion = vm.j2seVersion().bitAnd(J2SE_SERVICE_RELEASE_MASK).intValue() >> J2SE_JAVA_SPEC_VERSION_SHIFT;
if (javaVersion < J2SE_19) {
out.printf("This command only works with core file created by VM with Java version 9 or higher\n"
+ "The current VM Java version is: %s\n", javaVersion);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please %n instead of \n.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace the remaining \n with %n.

J9ObjectPointer moduleNameObjPtr = null;
PatternString pattern = new PatternString(searchModuleName);

out.println(String.format("Searching for modules named '%s' in VM=%s", searchModuleName,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use printf (with %n) instead of println and format.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the other instance of this pattern.

*/
private void printUsage(PrintStream out)
{
out.println("findmodulebyname <moduleNamePattern> - find the module corresponding to its given name pattern (like 'java.base')");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The help can be improved, I suggest:

findmodulebyname - find the modules corresponding to a name pattern (e.g. 'java.*')

Notice it makes it clear that more than one module might be found and shows the form of pattern used (I assume 'java.' is the same as the Java pattern 'java..').

{
public FindModuleByNameCommand()
{
addCommand("findmodulebyname", "<moduleName>", "find the modules corresponding to its module name");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The help here should be similar to that displayed by the usage method below.

Copy link
Contributor

@keithc-ca keithc-ca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address my most recent comments.

Signed-off-by: Charles_Zheng <Juntian.Zheng@ibm.com>
@keithc-ca
Copy link
Contributor

jenkins test sanity plinux

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants