[Spark Core] SPARK-4380 Edit spilling log from MB to B #3243

shenh062326 · 2014-11-13T12:42:02Z

https://issues.apache.org/jira/browse/SPARK-4380

AmplabJenkins · 2014-11-13T12:42:09Z

Can one of the admins verify this patch?

srowen · 2014-11-13T12:55:37Z

core/src/main/scala/org/apache/spark/util/collection/Spillable.scala

@@ -105,7 +105,7 @@ private[spark] trait Spillable[C] {
   */
  @inline private def logSpillage(size: Long) {
    val threadId = Thread.currentThread().getId
-    logInfo("Thread %d spilling in-memory map of %d MB to disk (%d time%s so far)"
-        .format(threadId, size / (1024 * 1024), _spillCount, if (_spillCount > 1) "s" else ""))
+    logInfo("Thread %d spilling in-memory map of %d B to disk (%d time%s so far)"


Better idea: use Utils.bytesToString to intelligently create a string like "20KB" or "3MB" or "139B" depending on the magnitude.

Yeah, that should produce the most human readable format.

Thanks Srowen, change to Utils.bytesToString.

JoshRosen · 2014-11-13T19:20:30Z

+1; the old behavior was really annoying and made it difficult to determine whether empty blocks were actually being spilled.

andrewor14 · 2014-11-13T22:11:48Z

Ok. I think this is a change important enough to make it into 1.2. LGTM pending Sean's comment.

andrewor14 · 2014-11-13T22:11:53Z

add to whitelist

SparkQA · 2014-11-13T22:15:06Z

Test build #23330 has started for PR 3243 at commit 946351c.

This patch merges cleanly.

SparkQA · 2014-11-13T23:21:06Z

Test build #23330 has finished for PR 3243 at commit 946351c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-13T23:21:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23330/
Test FAILed.

SparkQA · 2014-11-14T01:19:54Z

Test build #23344 has started for PR 3243 at commit e9145e8.

This patch merges cleanly.

SparkQA · 2014-11-14T01:22:05Z

Test build #23344 has finished for PR 3243 at commit e9145e8.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class NettyBlockTransferService(conf: SparkConf, securityManager: SecurityManager, numCores: Int)
- public class JavaSimpleTextClassificationPipeline
- case class LabeledDocument(id: Long, text: String, label: Double)
- case class Document(id: Long, text: String)
- abstract class EdgeRDD[ED, VD](
- abstract class VertexRDD[VD](
- abstract class Estimator[M <: Model[M]] extends PipelineStage with Params
- abstract class Evaluator extends Identifiable
- abstract class Model[M <: Model[M]] extends Transformer
- abstract class PipelineStage extends Serializable with Logging
- class Pipeline extends Estimator[PipelineModel]
- abstract class Transformer extends PipelineStage with Params
- class LogisticRegression extends Estimator[LogisticRegressionModel] with LogisticRegressionParams
- class HashingTF extends UnaryTransformer[Iterable[_], Vector, HashingTF]
- class StandardScaler extends Estimator[StandardScalerModel] with StandardScalerParams
- class Tokenizer extends UnaryTransformer[String, Seq[String], Tokenizer]
- class Param[T] (
- class DoubleParam(parent: Params, name: String, doc: String, defaultValue: Option[Double] = None)
- class IntParam(parent: Params, name: String, doc: String, defaultValue: Option[Int] = None)
- class FloatParam(parent: Params, name: String, doc: String, defaultValue: Option[Float] = None)
- class LongParam(parent: Params, name: String, doc: String, defaultValue: Option[Long] = None)
- class BooleanParam(parent: Params, name: String, doc: String, defaultValue: Option[Boolean] = None)
- case class ParamPair[T](param: Param[T], value: T)
- trait Params extends Identifiable with Serializable
- class CrossValidator extends Estimator[CrossValidatorModel] with CrossValidatorParams with Logging
- class ParamGridBuilder

AmplabJenkins · 2014-11-14T01:22:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23344/
Test FAILed.

SparkQA · 2014-11-14T01:35:09Z

Test build #23346 has started for PR 3243 at commit 4653378.

This patch merges cleanly.

SparkQA · 2014-11-14T02:58:51Z

Test build #23346 has finished for PR 3243 at commit 4653378.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-14T02:58:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23346/
Test PASSed.

andrewor14 · 2014-11-14T21:29:03Z

Ok merging into master and 1.2

https://issues.apache.org/jira/browse/SPARK-4380 Author: Hong Shen <hongshen@tencent.com> Closes #3243 from shenh062326/spark_change and squashes the following commits: 4653378 [Hong Shen] Edit spilling log from MB to B 21ee960 [Hong Shen] Edit spilling log from MB to B e9145e8 [Hong Shen] Edit spilling log from MB to B da761c2 [Hong Shen] Edit spilling log from MB to B 946351c [Hong Shen] Edit spilling log from MB to B (cherry picked from commit 0c56a03) Signed-off-by: Andrew Or <andrew@databricks.com>

This is the branch-1.1 version of #3243. Author: Andrew Or <andrew@databricks.com> Closes #3355 from andrewor14/spill-log-bytes-1.1 and squashes the following commits: 36ec152 [Andrew Or] Log more precise representation of bytes in spilling code

…42.7.4 and `mssql` to 12.8.1.jre11 ### What changes were proposed in this pull request? This PR aims to upgrade `h2` to 2.3.232, `postgresql` to 42.7.4 and `mssql` to 12.8.1.jre11. ### Why are the changes needed? 1. For `h2`, there are some issues fixed in version 2.3.232(full release notes: https://www.h2database.com/html/changelog.html): - [Issue #3945](h2database/h2database#3945): Column not found in correlated subquery, when referencing outer column from LEFT JOIN .. ON clause - [Issue #4097](h2database/h2database#4097): StackOverflowException when using multiple SELECT statements in one query (2.3.230) - [Issue #3982](h2database/h2database#3982): Potential issue when using ROUND - [Issue #3894](h2database/h2database#3894): Race condition causing stale data in query last result cache - [Issue #4075](h2database/h2database#4075): infinite loop in compact - [Issue #4091](h2database/h2database#4091): Wrong case with linked table to postgresql - [Issue #4088](h2database/h2database#4088): BadGrammarException when the same alias is used within two different CTEs 2. For `postgresql`, there are some issues fixed and improvements in version 42.7.4(full release notes: https://jdbc.postgresql.org/changelogs/2024-08-22-42.7.4-release/): - fix: PgInterval ignores case for represented interval string [PR #3344](pgjdbc/pgjdbc#3344) - perf: Avoid extra copies when receiving int4 and int2 in PGStream [PR #3295](pgjdbc/pgjdbc#3295) - fix: Add support for Infinity::numeric values in ResultSet.getObject [PR #3304](pgjdbc/pgjdbc#3304) - fix: Ensure order of results for getDouble [PR #3301](pgjdbc/pgjdbc#3301) - perf: Replace BufferedOutputStream with unsynchronized PgBufferedOutputStream, allow configuring different Java and SO_SNDBUF buffer sizes [PR #3248](pgjdbc/pgjdbc#3248) - fix: Fix SSL tests [PR #3260](pgjdbc/pgjdbc#3260) - fix: Support bytea in preferQueryMode=simple [PR #3243](pgjdbc/pgjdbc#3243) - fix: Fix [Issue #3234](pgjdbc/pgjdbc#3234) - Return -1 as update count for stored procedure calls [PR #3235](pgjdbc/pgjdbc#3235) - fix: Fix [Issue #3224](pgjdbc/pgjdbc#3224) - conversion for TIME ‘24:00’ to LocalTime breaks in binary-mode [PR #3225](pgjdbc/pgjdbc#3225) 3. For `mssql`, there are some issues fixed in 12.8.1.jre11(full release notes: https://github.com/microsoft/mssql-jdbc/releases/tag/v12.8.1): - Adjusted DESTINATION_COL_METADATA_LOCK, in SQLServerBulkCopy, so that is properly released in all cases [PR #2492](microsoft/mssql-jdbc#2492) - Reverted "Execute Stored Procedures Directly" feature, as well as subsequent changes related to the feature [PR #2493](microsoft/mssql-jdbc#2493) - Changed driver behavior to allow prepared statement objects to be reused, preventing a "multiple queries are not allowed" error [PR #2494](microsoft/mssql-jdbc#2494) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47810 from wayneguow/ug_h2. Authored-by: Wei Guo <guow93@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>

…42.7.4 and `mssql` to 12.8.1.jre11 ### What changes were proposed in this pull request? This PR aims to upgrade `h2` to 2.3.232, `postgresql` to 42.7.4 and `mssql` to 12.8.1.jre11. ### Why are the changes needed? 1. For `h2`, there are some issues fixed in version 2.3.232(full release notes: https://www.h2database.com/html/changelog.html): - [Issue apache#3945](h2database/h2database#3945): Column not found in correlated subquery, when referencing outer column from LEFT JOIN .. ON clause - [Issue apache#4097](h2database/h2database#4097): StackOverflowException when using multiple SELECT statements in one query (2.3.230) - [Issue apache#3982](h2database/h2database#3982): Potential issue when using ROUND - [Issue apache#3894](h2database/h2database#3894): Race condition causing stale data in query last result cache - [Issue apache#4075](h2database/h2database#4075): infinite loop in compact - [Issue apache#4091](h2database/h2database#4091): Wrong case with linked table to postgresql - [Issue apache#4088](h2database/h2database#4088): BadGrammarException when the same alias is used within two different CTEs 2. For `postgresql`, there are some issues fixed and improvements in version 42.7.4(full release notes: https://jdbc.postgresql.org/changelogs/2024-08-22-42.7.4-release/): - fix: PgInterval ignores case for represented interval string [PR apache#3344](pgjdbc/pgjdbc#3344) - perf: Avoid extra copies when receiving int4 and int2 in PGStream [PR apache#3295](pgjdbc/pgjdbc#3295) - fix: Add support for Infinity::numeric values in ResultSet.getObject [PR apache#3304](pgjdbc/pgjdbc#3304) - fix: Ensure order of results for getDouble [PR apache#3301](pgjdbc/pgjdbc#3301) - perf: Replace BufferedOutputStream with unsynchronized PgBufferedOutputStream, allow configuring different Java and SO_SNDBUF buffer sizes [PR apache#3248](pgjdbc/pgjdbc#3248) - fix: Fix SSL tests [PR apache#3260](pgjdbc/pgjdbc#3260) - fix: Support bytea in preferQueryMode=simple [PR apache#3243](pgjdbc/pgjdbc#3243) - fix: Fix [Issue apache#3234](pgjdbc/pgjdbc#3234) - Return -1 as update count for stored procedure calls [PR apache#3235](pgjdbc/pgjdbc#3235) - fix: Fix [Issue apache#3224](pgjdbc/pgjdbc#3224) - conversion for TIME ‘24:00’ to LocalTime breaks in binary-mode [PR apache#3225](pgjdbc/pgjdbc#3225) 3. For `mssql`, there are some issues fixed in 12.8.1.jre11(full release notes: https://github.com/microsoft/mssql-jdbc/releases/tag/v12.8.1): - Adjusted DESTINATION_COL_METADATA_LOCK, in SQLServerBulkCopy, so that is properly released in all cases [PR apache#2492](microsoft/mssql-jdbc#2492) - Reverted "Execute Stored Procedures Directly" feature, as well as subsequent changes related to the feature [PR apache#2493](microsoft/mssql-jdbc#2493) - Changed driver behavior to allow prepared statement objects to be reused, preventing a "multiple queries are not allowed" error [PR apache#2494](microsoft/mssql-jdbc#2494) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47810 from wayneguow/ug_h2. Authored-by: Wei Guo <guow93@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>

Edit spilling log from MB to B

946351c

srowen reviewed Nov 13, 2014
View reviewed changes

shenh062326 added 2 commits November 14, 2014 09:11

Edit spilling log from MB to B

da761c2

Edit spilling log from MB to B

e9145e8

shenh062326 added 2 commits November 14, 2014 09:24

Edit spilling log from MB to B

21ee960

Edit spilling log from MB to B

4653378

asfgit closed this in 0c56a03 Nov 14, 2014

andrewor14 mentioned this pull request Nov 19, 2014

[SPARK-4380] Log more precise number of bytes spilled (1.1) #3355

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Spark Core] SPARK-4380 Edit spilling log from MB to B #3243

[Spark Core] SPARK-4380 Edit spilling log from MB to B #3243

shenh062326 commented Nov 13, 2014

AmplabJenkins commented Nov 13, 2014

srowen Nov 13, 2014

andrewor14 Nov 13, 2014

shenh062326 Nov 14, 2014

JoshRosen commented Nov 13, 2014

andrewor14 commented Nov 13, 2014

andrewor14 commented Nov 13, 2014

SparkQA commented Nov 13, 2014

SparkQA commented Nov 13, 2014

AmplabJenkins commented Nov 13, 2014

SparkQA commented Nov 14, 2014

SparkQA commented Nov 14, 2014

AmplabJenkins commented Nov 14, 2014

SparkQA commented Nov 14, 2014

SparkQA commented Nov 14, 2014

AmplabJenkins commented Nov 14, 2014

andrewor14 commented Nov 14, 2014

[Spark Core] SPARK-4380 Edit spilling log from MB to B #3243

[Spark Core] SPARK-4380 Edit spilling log from MB to B #3243

Conversation

shenh062326 commented Nov 13, 2014

AmplabJenkins commented Nov 13, 2014

srowen Nov 13, 2014

Choose a reason for hiding this comment

andrewor14 Nov 13, 2014

Choose a reason for hiding this comment

shenh062326 Nov 14, 2014

Choose a reason for hiding this comment

JoshRosen commented Nov 13, 2014

andrewor14 commented Nov 13, 2014

andrewor14 commented Nov 13, 2014

SparkQA commented Nov 13, 2014

SparkQA commented Nov 13, 2014

AmplabJenkins commented Nov 13, 2014

SparkQA commented Nov 14, 2014

SparkQA commented Nov 14, 2014

AmplabJenkins commented Nov 14, 2014

SparkQA commented Nov 14, 2014

SparkQA commented Nov 14, 2014

AmplabJenkins commented Nov 14, 2014

andrewor14 commented Nov 14, 2014