feat: plonk frontend filter common cases of duplicate constraints

[gbotrel]

TLDR;

// addConstraintExist check if we recorded a constraint in the form
// q1*xa + q2*xb + qC - xc == 0
//
// if we find one, this function returns the xc wire with the correct coefficients.
// if we don't, and no previous addition was recorded with xa and xb, add an entry in the map
// (this assumes that the caller will add a constraint just after this call if it's not found!)
//
// idea:
// 1. take (xa | (xb << 32)) as a identifier of an addition that used wires xa and xb.
// 2. look for an entry in builder.mAddConstraints for a previously added constraint that matches.
// 3. if so, check that the coefficients matches and we can re-use xc wire.
//
// limitations:
// 1. for efficiency, we just store the first addition that occurred with with xa and xb;
// so if we do 2*xa + 3*xb == c, then want to compute xa + xb == d multiple times, the compiler is
// not going to catch these duplicates.
// 2. this piece of code assumes some behavior from constraint/ package (like coeffIDs, or append-style
// constraint management)

Perf

• compilation time with scs.Builder seems to be ~15% slower (std/groth16bls12377 test)
• tradeoff seems reasonable IMO, for some circuits it reduces drastically the number of generated constraints.

Same logic with mul gates; except we are more permissive with xa and xb coefficients since we can always re-use the existing wire in that case.

• feat: check for common duplicate mul gate in plonk frontend
• feat: scs detects duplicate add constraints
• feat: update stats

[gbotrel]

• definitely need to add some specific test cases for this PR.

[ivokub]

Looks good. I'm still amazed at the savings this PR gives.

I think long-term maybe should consider two approaches:

• also store other add constraints with same wires (maybe instead of map[uint64]int use map[uint64][]int and then do linear scan?
• maybe put some fast inclusion checker in front of map (a la Bloom filter). Maybe helps with the perf?

[ivokub]

Suggested edit:

diff --git a/frontend/internal/expr/linear_expression.go b/frontend/internal/expr/linear_expression.go
index 3ef4c90c..21b3b8f1 100644
--- a/frontend/internal/expr/linear_expression.go
+++ b/frontend/internal/expr/linear_expression.go
@@ -38,17 +38,6 @@ func (t Term) WireID() int {
 }
 
 func (t Term) HashCode() uint64 {
-	// hashCode := t.Coeff[0] * 17
-	// hashCode *= 31
-	// hashCode += t.Coeff[1] * 17
-	// hashCode *= 31
-	// hashCode += t.Coeff[2] * 17
-	// hashCode *= 31
-	// hashCode += t.Coeff[3] * 17
-	// hashCode *= 31
-	// hashCode <<= 29
-	// hashCode |= uint64(t.VID) // << 2**29;
-	// return hashCode
 	return t.Coeff[0]*29 + uint64(t.VID<<12)
 }
 

[gbotrel]

Looks good. I'm still amazed at the savings this PR gives.

I think long-term maybe should consider two approaches:

• also store other add constraints with same wires (maybe instead of map[uint64]int use map[uint64][]int and then do linear scan?
• maybe put some fast inclusion checker in front of map (a la Bloom filter). Maybe helps with the perf?

1. Yeah first version I did that, but I reverted. Reason; it's an heuristic ^^. If we store a slice + init it for each addition we make, the (memory & CPU) overhead for a 50M constraint circuit is not going to be 15% compile time, but likely triple digit.
2. OK will try something -- we had a similar pattern in R1CS builder with linear expression and colliding hashcodes; (markboolean), if it helps here, it will help there.

re: savings; I think some of it may be reusing the wires in non-obvious situations. like we do a bit more than just "duplicate constraints", we detect them up to a factor (for additions) & with different coeffs (for mul).

[gbotrel]

@ivokub re bloom filter; I don't think it can beat the map[uint64]int, for plonk frontend (involves hashes)

[gbotrel]

we may improve CPU but make it worse for memory by deriving something from this idea, like... a slice of uint64 of len == nb variables.
So you access directly the index of the variable. left 32 bits == all the variable ID (combined with OR) of multiplication, same thing on the right 32 bits for additions.
probably something better than this... will keep it in mind if it becomes an issue.