Skip to content

Commit

Permalink
[SPARK-26963][MLLIB] SizeEstimator can't make some JDK fields accessi…
Browse files Browse the repository at this point in the history
…ble in Java 9+ (#580)

## Upstream SPARK-26963 apache#23866
## What changes were proposed in this pull request?

Don't use inaccessible fields in SizeEstimator, which comes up in Java 9+

## How was this patch tested?

Manually ran tests with Java 11; it causes these tests that failed before to pass.
This ought to pass on Java 8 as there's effectively no change for Java 8.

Closes apache#23866 from srowen/SPARK-26963.

Authored-by: Sean Owen <sean.owen@databricks.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
  • Loading branch information
robert3005 authored and bulldozer-bot[bot] committed Jun 28, 2019
1 parent 5454b49 commit eca4aa8
Showing 1 changed file with 14 additions and 2 deletions.
16 changes: 14 additions & 2 deletions core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
Original file line number Diff line number Diff line change
Expand Up @@ -334,9 +334,21 @@ object SizeEstimator extends Logging {
if (fieldClass.isPrimitive) {
sizeCount(primitiveSize(fieldClass)) += 1
} else {
field.setAccessible(true) // Enable future get()'s on this field
// Note: in Java 9+ this would be better with trySetAccessible and canAccess
try {
field.setAccessible(true) // Enable future get()'s on this field
pointerFields = field :: pointerFields
} catch {
// If the field isn't accessible, we can still record the pointer size
// but can't know more about the field, so ignore it
case _: SecurityException =>
// do nothing
// Java 9+ can throw InaccessibleObjectException but the class is Java 9+-only
case re: RuntimeException
if re.getClass.getSimpleName == "InaccessibleObjectException" =>
// do nothing
}
sizeCount(pointerSize) += 1
pointerFields = field :: pointerFields
}
}
}
Expand Down

0 comments on commit eca4aa8

Please sign in to comment.