lightning: risk of OOM

[buchuitoudegou]

reproduce

1. create a csv file with unclosed quote till end:

1,2,3,"xxxxx...

2. insert 10GB random bytes into the data file
3. import

result

If user wants to import a big file with unclosed quote, lightning might possibly be OOM.

[buchuitoudegou]

/component lightning
/type bug
/severity moderate

[buchuitoudegou]

root cause:

tidb/br/pkg/lightning/mydump/csv_parser.go

Lines 338 to 355 in f7c87c8

	buf = append(buf, parser.buf...)
	parser.buf = nil
	if err := parser.readBlock(); err != nil || len(parser.buf) == 0 {
	if err == nil {
	err = io.EOF
	}
	parser.pos += int64(len(buf))
	return buf, 0, errors.Trace(err)
	}
	index := IndexAnyByte(parser.buf, chars)
	if index >= 0 {
	buf = append(buf, parser.buf[:index]...)
	parser.buf = parser.buf[index:]
	parser.pos += int64(len(buf))
	return buf, parser.buf[0], nil
	}
	}
	}

the parser would try to read the block continually until it finds the quotes. But if the data files have syntax error (i.e., unterminated quotes), it won't stop.

[buchuitoudegou]

relevant: #39980