diff --git a/README.md b/README.md index a8573aacc2..57c93fec20 100644 --- a/README.md +++ b/README.md @@ -4,16 +4,18 @@ GeoSpark is listed as **Infrastructure Project** on **Apache Spark Official Third Party Project Page** ([http://spark.apache.org/third-party-projects.html](http://spark.apache.org/third-party-projects.html)) -GeoSpark is a cluster computing system for processing large-scale spatial data. GeoSpark extends Apache Spark with a set of out-of-the-box Spatial Resilient Distributed Datasets (SRDDs) that efficiently load, process, and analyze large-scale spatial data across machines. GeoSpark provides APIs for Apache Spark programmer to easily develop their spatial analysis programs with Spatial Resilient Distributed Datasets (SRDDs) which have in house support for geometrical and distance operations. +GeoSpark is a cluster computing system for processing large-scale spatial data. GeoSpark extends Apache Spark with a set of out-of-the-box Spatial Resilient Distributed Datasets (SRDDs) that efficiently load, process, and analyze large-scale spatial data across machines. GeoSpark provides APIs for Apache Spark programmer to easily develop their spatial analysis programs with Spatial Resilient Distributed Datasets (SRDDs) which have in house support for geometrical and Spatial Queries (Range, K Nearest Neighbors, Join). GeoSpark artifacts are hosted in Maven Central. You can add a Maven dependency with the following coordinates: +The following version supports Apache Spark 2.X versions: + ``` groupId: org.datasyslab artifactId: geospark -version: 0.4.0 +version: 0.5.0 ``` The following version supports Apache Spark 1.X versions: @@ -21,20 +23,29 @@ The following version supports Apache Spark 1.X versions: ``` groupId: org.datasyslab artifactId: geospark -version: 0.4.0-spark-1.x +version: 0.5.0-spark-1.x ``` - - ## Version information ([Full List](https://github.com/DataSystemsLab/GeoSpark/wiki/GeoSpark-Full-Version-Release-notes)) | Version | Summary | |:----------------: |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 0.4.0| **Major updates:** ([Example](https://github.com/DataSystemsLab/GeoSpark/blob/master/src/main/java/org/datasyslab/geospark/showcase/Example.java)) 1. Refactor constrcutor API usage. 2. Simplify Spatial Join Query API. 3. Add native support for LineStringRDD; **Functionality enhancement:** 1. Release the persist function back to users. 2. Add more exception explanations. -| 0.3.2 | Functionality enhancement: 1. [JTSplus Spatial Objects](https://github.com/jiayuasu/JTSplus) now carry the original input data. Each object stores "UserData" and provides getter and setter. 2. Add a new SpatialRDD constructor to transform a regular data RDD to a spatial partitioned SpatialRDD. | -| 0.3.1 | Bug fix: Support Apache Spark 2.X version, fix a bug which results in inaccurate results when doing join query, add more unit test cases | -| 0.3 | Major updates: Significantly shorten query time on spatial join for skewed data; Support load balanced spatial partitioning methods (also serve as the global index); Optimize code for iterative spatial data mining || +| 0.5.0| **Major updates:** We are pleased to announce the initial version of [Babylon](https://github.com/DataSystemsLab/GeoSpark/tree/master/src/main/java/org/datasyslab/babylon) a large-scale in-memory geospatial visualization system extending GeoSpark. Babylon and GeoSpark are integrated together. You can just import GeoSpark and enjoy! More detials are available here: [Babylon GeoSpatial Visualization](https://github.com/DataSystemsLab/GeoSpark/tree/master/src/main/java/org/datasyslab/babylon); +| 0.4.0| **Major updates:** ([Example](https://github.com/DataSystemsLab/GeoSpark/blob/master/src/main/java/org/datasyslab/geospark/showcase/Example.java)) 1. Refactor constrcutor API usage. 2. Simplify Spatial Join Query API. 3. Add native support for LineStringRDD; **Functionality enhancement:** 1. Release the persist function back to users. 2. Add more exception explanations.| + +##News +### Babylon Visualization Framework on GeoSpark is now available! +Babylon is a large-scale in-memory geospatial visualization system. + +Babylon provides native support for general cartographic design by extending GeoSpark to process large-scale spatial data. It can visulize Spatial RDD and Spatial Queries and render super high resolution image in parallel. + +Babylon and GeoSpark are integrated together. You just need to import GeoSpark and enjoy them! More detials are available here: [Babylon GeoSpatial Visualization](https://github.com/DataSystemsLab/GeoSpark/tree/master/src/main/java/org/datasyslab/babylon) + +### Babylon Gallery + + + ## How to get started (For Scala and Java developers) @@ -81,7 +92,7 @@ Note: GeoSpark Master branch supports Apache Spark 2.X releases and GeoSpark for ## Scala and Java API usage -Please refer [GeoSpark Scala and Java API Usage](http://www.public.asu.edu/~jiayu2/geospark/javadoc/index.html) +Please refer to [GeoSpark Scala and Java API Usage](http://www.public.asu.edu/~jiayu2/geospark/javadoc/) ## Spatial Resilient Distributed Datasets (SRDDs) @@ -91,7 +102,7 @@ GeoSpark extends RDDs to form Spatial RDDs (SRDDs) and efficiently partitions SR **Supported Spatial RDDs: PointRDD, RectangleRDD, PolygonRDD, LineStringRDD** ## Supported data format -GeoSpark supports Comma-Separated Values (**CSV**), Tab-separated values (**TSV**), Well-Known Text (**WKT**), and **GeoJSON** as the input formats. Users only need to specify input format as Splitter and the start and end offset (if necessary) of spatial fields in one row when call Constructors. +GeoSpark supports Comma-Separated Values (**CSV**), Tab-separated values (**TSV**), Well-Known Text (**WKT**), and **GeoJSON** as the input formats. Users only need to specify input format as Splitter and the start and end offset (if necessary) of spatial fields in one row when call Constructors. GeoSpark also takes **any user-supplied format mapper function** to support the desired format. ## Important features @@ -109,8 +120,19 @@ GeoSpark currently provides native support for Inside, Overlap, DatasetBoundary, ### Spatial Operation -GeoSpark so far provides spatial range query, join query and KNN query in SRDDs. +GeoSpark so far provides **Spatial Range Query**, **Spatial Join Query**, and **Spatial K Nearest Neighbors Query**. + +#Babylon Visualization Framework on GeoSpark +Babylon is a large-scale in-memory geospatial visualization system. + +Babylon provides native support for general cartographic design by extending GeoSpark to process large-scale spatial data. It can visulize Spatial RDD and Spatial Queries and render super high resolution image in parallel. + +Babylon and GeoSpark are integrated together. You just need to import GeoSpark and enjoy them! More detials are available here: [Babylon GeoSpatial Visualization](https://github.com/DataSystemsLab/GeoSpark/tree/master/src/main/java/org/datasyslab/babylon) +## Babylon Gallery + + + ## Publication diff --git a/pom.xml b/pom.xml index 5967a2063f..0893905a11 100644 --- a/pom.xml +++ b/pom.xml @@ -51,7 +51,7 @@ org.apache.spark spark-core_2.10 - 1.6.2 + 1.6.3 provided diff --git a/src/main/java/org/datasyslab/babylon/README.md b/src/main/java/org/datasyslab/babylon/README.md new file mode 100644 index 0000000000..660b8e4e6b --- /dev/null +++ b/src/main/java/org/datasyslab/babylon/README.md @@ -0,0 +1,63 @@ +# BABYLON +**Babylon is a large-scale in-memory geospatial visualization system** + +**Babylon** provides native support for general cartographic design by extending **GeoSpark** to process large-scale spatial data. +## Babylon Gallery +### Scatter Plot: USA mainland rail network + +### Heat Map: New York City Taxi Trips (with a given map background) + +### Choropleth Map + Overlay Operator: USA mainland tweets per USA county (Spatial Join Query) + + +## Main Features + +### Extensible Visualization operator + +* Support super high resolution image generation: parallel map image rendering +* Visualize Spatial RDD and Spatial Queries (Spatial Range, Spatial K Nearest Neighbors, Spatial Join) +* Customizable: Can be customized to any user-supplied colors or coloring rule +* Extensible: Can be extended to any visualization effect + +### Overlay Operator +Overlay one map layer with many other map layers! + +### Various Image filter +* Gaussian Blur +* Box Blur +* Embose +* Outline +* Sharpen +* More! + +You also can buld your new image filter by easily extending the photo filter! + +### Various Image Type +* PNG +* JPEG +* GIF + +You also can support your desired image type by easily extending the photo filter! + + +### Current Visualization effect + +* Scatter Plot +* Heat Map +* Choropleth Map +* More! + +You also can build your new self-designed effects by easily extending the visualization operator! + +### Example +Here is [a runnable single machine exmaple code](https://github.com/jiayuasu/GeoSpark/blob/master/src/main/java/org/datasyslab/babylon/showcase/Example.java). You can clone this repository and directly run it on you local machine! + +### Scala and Java API +Please refer to [Babylon Scala and Java API](http://www.public.asu.edu/~jiayu2/geospark/javadoc/latest/). +### Supported Spatial Objects and Input format + +All spatial obects and input formats supported by GeoSpark + +##Contributor +* [Jia Yu](http://www.public.asu.edu/~jiayu2/) (jiayu2@asu.edu) +* [Mohamed Sarwat](http://faculty.engineering.asu.edu/sarwat/) (msarwat@asu.edu) \ No newline at end of file diff --git a/src/main/java/org/datasyslab/babylon/core/OverlayOperator.java b/src/main/java/org/datasyslab/babylon/core/OverlayOperator.java index 08ae255161..8bccb3af5e 100644 --- a/src/main/java/org/datasyslab/babylon/core/OverlayOperator.java +++ b/src/main/java/org/datasyslab/babylon/core/OverlayOperator.java @@ -24,14 +24,14 @@ public class OverlayOperator { public BufferedImage backImage; /** The distributed back image. */ - public JavaPairRDD distributedBackImage; + public JavaPairRDD distributedBackImage; /** * Instantiates a new overlay operator. * * @param distributedBackImage the distributed back image */ - public OverlayOperator(JavaPairRDD distributedBackImage) + public OverlayOperator(JavaPairRDD distributedBackImage) { this.distributedBackImage = distributedBackImage; } @@ -52,17 +52,17 @@ public OverlayOperator(BufferedImage backImage) * @param distributedFontImage the distributed font image * @return true, if successful */ - public boolean JoinImage(JavaPairRDD distributedFontImage) + public boolean JoinImage(JavaPairRDD distributedFontImage) { - this.distributedBackImage = this.distributedBackImage.cogroup(distributedFontImage).mapToPair(new PairFunction,Iterable>>,Integer,BufferedImage>() + this.distributedBackImage = this.distributedBackImage.cogroup(distributedFontImage).mapToPair(new PairFunction,Iterable>>,Integer,ImageSerializableWrapper>() { @Override - public Tuple2 call( - Tuple2, Iterable>> imagePair) + public Tuple2 call( + Tuple2, Iterable>> imagePair) throws Exception { int imagePartitionId = imagePair._1; - Iterator backImageIterator = imagePair._2._1.iterator(); - Iterator frontImageIterator = imagePair._2._2.iterator(); + Iterator backImageIterator = imagePair._2._1.iterator(); + Iterator frontImageIterator = imagePair._2._2.iterator(); if(backImageIterator.hasNext()==false) { throw new Exception("[OverlayOperator][JoinImage] The back image iterator didn't get any image partitions."); @@ -71,8 +71,8 @@ public Tuple2 call( { throw new Exception("[OverlayOperator][JoinImage] The front image iterator didn't get any image partitions."); } - BufferedImage backImage = backImageIterator.next(); - BufferedImage frontImage = frontImageIterator.next(); + BufferedImage backImage = backImageIterator.next().image; + BufferedImage frontImage = frontImageIterator.next().image; if(backImage.getWidth()!=frontImage.getWidth()||backImage.getHeight()!=frontImage.getHeight()) { throw new Exception("[OverlayOperator][JoinImage] The two given image don't have the same width or the same height."); @@ -83,7 +83,7 @@ public Tuple2 call( Graphics graphics = combinedImage.getGraphics(); graphics.drawImage(backImage, 0, 0, null); graphics.drawImage(frontImage, 0, 0, null); - return new Tuple2(imagePartitionId,combinedImage); + return new Tuple2(imagePartitionId,new ImageSerializableWrapper(combinedImage)); } }); return true; diff --git a/src/main/java/org/datasyslab/babylon/showcase/nycheatmap.png b/src/main/java/org/datasyslab/babylon/showcase/nycheatmap.png new file mode 100644 index 0000000000..c6b77cb398 Binary files /dev/null and b/src/main/java/org/datasyslab/babylon/showcase/nycheatmap.png differ diff --git a/src/main/java/org/datasyslab/babylon/showcase/usrail.png b/src/main/java/org/datasyslab/babylon/showcase/usrail.png new file mode 100644 index 0000000000..2507f6a8cf Binary files /dev/null and b/src/main/java/org/datasyslab/babylon/showcase/usrail.png differ diff --git a/src/main/java/org/datasyslab/babylon/showcase/ustweet.png b/src/main/java/org/datasyslab/babylon/showcase/ustweet.png new file mode 100644 index 0000000000..5c5c575cd1 Binary files /dev/null and b/src/main/java/org/datasyslab/babylon/showcase/ustweet.png differ diff --git a/src/test/java/org/datasyslab/babylon/NYCTripTest.java b/src/test/java/org/datasyslab/babylon/NYCTripTest.java index c52ca54eb2..ad6f6a0e88 100644 --- a/src/test/java/org/datasyslab/babylon/NYCTripTest.java +++ b/src/test/java/org/datasyslab/babylon/NYCTripTest.java @@ -54,7 +54,7 @@ class NYCTripMapper implements FlatMapFunction { Coordinate[] coordinates; LinearRing linear; int actualEndOffset; - public Iterator call(String line) throws Exception { + public Iterable call(String line) throws Exception { List result= new ArrayList(); Geometry spatialObject = null; MultiLineString multiSpatialObjects = null; @@ -65,7 +65,7 @@ public Iterator call(String line) throws Exception { coordinatesList.add(new Coordinate(Double.parseDouble(lineSplitList.get(8))*-1.0, Double.parseDouble(lineSplitList.get(9)))); spatialObject = fact.createLineString(coordinatesList.toArray(new Coordinate[coordinatesList.size()])); result.add((LineString)spatialObject); - return result.iterator(); + return result; } } diff --git a/src/test/java/org/datasyslab/geospark/spatialRDD/LineStringRDDTest.java b/src/test/java/org/datasyslab/geospark/spatialRDD/LineStringRDDTest.java index d41ec03bc1..64919a94f5 100644 --- a/src/test/java/org/datasyslab/geospark/spatialRDD/LineStringRDDTest.java +++ b/src/test/java/org/datasyslab/geospark/spatialRDD/LineStringRDDTest.java @@ -143,6 +143,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); // System.out.println(map.size()); @@ -152,6 +153,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ } /** @@ -169,16 +171,17 @@ public void testRTreeSpatialPartitioing() throws Exception { for (Envelope d : spatialRDD.grids) { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } - //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ } /** @@ -197,15 +200,17 @@ public void testVoronoiSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; - // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); + // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ } diff --git a/src/test/java/org/datasyslab/geospark/spatialRDD/PointRDDTest.java b/src/test/java/org/datasyslab/geospark/spatialRDD/PointRDDTest.java index 627c6a110e..7ae6e998b3 100644 --- a/src/test/java/org/datasyslab/geospark/spatialRDD/PointRDDTest.java +++ b/src/test/java/org/datasyslab/geospark/spatialRDD/PointRDDTest.java @@ -153,6 +153,7 @@ public void testEqualPartitioning() throws Exception { } //System.out.println(spatialRDD.boundaryEnvelope); //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); // System.out.println(map.size()); @@ -162,6 +163,7 @@ public void testEqualPartitioning() throws Exception { Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); assert spatialRDD.countWithoutDuplicates()==spatialRDD.countWithoutDuplicatesSPRDD(); } @@ -182,6 +184,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); // System.out.println(map.size()); @@ -191,6 +194,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); assert spatialRDD.countWithoutDuplicates()==spatialRDD.countWithoutDuplicatesSPRDD(); } @@ -212,15 +216,17 @@ public void testRTreeSpatialPartitioing() throws Exception { } //System.out.println(spatialRDD.boundaryEnvelope); //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; - //System.out.println("Rtree "+entry.getKey() + " : " + String.format("%.4f", percentage)); + // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); assert spatialRDD.countWithoutDuplicates()==spatialRDD.countWithoutDuplicatesSPRDD(); } @@ -241,15 +247,17 @@ public void testVoronoiSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; - //System.out.println("Voronoi "+entry.getKey() + " : " + String.format("%.4f", percentage)); + // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); assert spatialRDD.countWithoutDuplicates()==spatialRDD.countWithoutDuplicatesSPRDD(); } diff --git a/src/test/java/org/datasyslab/geospark/spatialRDD/PolygonRDDTest.java b/src/test/java/org/datasyslab/geospark/spatialRDD/PolygonRDDTest.java index e8bd617000..509c39d547 100644 --- a/src/test/java/org/datasyslab/geospark/spatialRDD/PolygonRDDTest.java +++ b/src/test/java/org/datasyslab/geospark/spatialRDD/PolygonRDDTest.java @@ -143,6 +143,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); // System.out.println(map.size()); @@ -152,6 +153,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ } /** @@ -170,15 +172,17 @@ public void testRTreeSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ } /** @@ -197,15 +201,17 @@ public void testVoronoiSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; - // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); + // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ } diff --git a/src/test/java/org/datasyslab/geospark/spatialRDD/RectangleRDDTest.java b/src/test/java/org/datasyslab/geospark/spatialRDD/RectangleRDDTest.java index 3f4620137a..e5a42b707d 100644 --- a/src/test/java/org/datasyslab/geospark/spatialRDD/RectangleRDDTest.java +++ b/src/test/java/org/datasyslab/geospark/spatialRDD/RectangleRDDTest.java @@ -139,6 +139,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); // System.out.println(map.size()); @@ -148,7 +149,7 @@ public void testHilbertCurveSpatialPartitioing() throws Exception { Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } - //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); + */ //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); assert spatialRDD.countWithoutDuplicates()==spatialRDD.countWithoutDuplicatesSPRDD(); } @@ -168,15 +169,17 @@ public void testRTreeSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); assert spatialRDD.countWithoutDuplicates()==spatialRDD.countWithoutDuplicatesSPRDD(); } @@ -197,15 +200,17 @@ public void testVoronoiSpatialPartitioing() throws Exception { //System.out.println("PointRDD spatial partitioning grids: "+d.grid); } //todo: Move this into log4j. + /* Map map = spatialRDD.spatialPartitionedRDD.countByKey(); - //System.out.println(map.size()); + // System.out.println(map.size()); for (Entry entry : map.entrySet()) { Long number = (Long) entry.getValue(); Double percentage = number.doubleValue() / spatialRDD.totalNumberOfRecords; - // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); + // System.out.println(entry.getKey() + " : " + String.format("%.4f", percentage)); } + */ //System.out.println("Original number of records: "+spatialRDD.countWithoutDuplicates()+" Spatial partitioned records: "+spatialRDD.countWithoutDuplicatesSPRDD()); assert spatialRDD.countWithoutDuplicates()==spatialRDD.countWithoutDuplicatesSPRDD(); }