[SPARK-44239][SQL][FOLLOWUP] Do not disable vector memory optimization when hugeVectorThreshold=0 to align its document

[wankunde]

What changes were proposed in this pull request?

Minor fix: Update default hugeVectorThreshold from 0 to -1 in code to match the documentation.
Keep consistent with the documentation

Why are the changes needed?

Keep code and documentation consistent

Does this PR introduce any user-facing change?

No

How was this patch tested?

Exists UT

Was this patch authored or co-authored using generative AI tooling?

No

[yaooqinn]

Also cc @cloud-fan who reviewed and merged #41782 / SPARK-44239

[dongjoon-hyun]

I have a few questions, @wankunde .

• The AS-IS PR title, Update default hugeVectorThreshold from 0 to -1 in code, could be a little misleading because the default value means the following. And, this is also a code. Do you think you can revise the PR title more intuitively?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines 506 to 512 in 2ed6c3e

	buildConf("spark.sql.inMemoryColumnarStorage.hugeVectorThreshold")
	.doc("When the required memory is larger than this, spark reserves required memory * " +
	s"${VECTORIZED_HUGE_VECTOR_RESERVE_RATIO.key} memory next time and release this column " +
	s"vector memory before reading the next batch rows. -1 means disabling the optimization.")
	.version("4.0.0")
	.bytesConf(ByteUnit.BYTE)
	.createWithDefault(-1)

• Although it seems to be difficult to verify, do you think we can add a test coverage to verify your new code and prevent a future regression at this part?

- if (hugeVectorThreshold > 0 && capacity > hugeVectorThreshold) {
+ if (hugeVectorThreshold > -1 && capacity > hugeVectorThreshold) {

[yaooqinn]

Hi, @dongjoon-hyun

could be a little misleading because the default value means the following. And, this is also a code

When setting spark.sql.inMemoryColumnarStorage.hugeVectorThreshold=0, the code does not work as the doc

[cloud-fan]

= 0 means we treat all vectors as huge vectors?

[wankunde]

Yes， but = 0 is not recommended, just keep code and documentation consistent

[cloud-fan]

then shall we update the doc instead?

[yaooqinn]

Changing the doc will result in code part change as other places test if hugeVectorThreshold < 0

[cloud-fan]

ah I see

[yaooqinn]

Merged to master. Thank you @wankunde and all