YQL value too long INSERT INTO ybdemo.ugent(id, data) VALUES (?, ?) with overlong JSON data #2116
Assignees
Labels
Comments
|
For the sample I tried, I was able to insert larger values - like 24MB JSONB -- but it had a single large field. Perhaps, in your case, there are many more fields, and the internal representation exceeds some threshold, and you are getting "YQL value too long" error. Also, agreed that this error should be surfaced upstream to the program cleanly. |
|
No, this happens with JSON with exactly one key |
|
@zopyx Please provide the used 7 MB value (or any similar) for testing purposes. Thanks! |
|
Here is my script |
|
Oops...my initial claim is wrong...the data has a growing number of keys...not a single large key...anyway...the error must be propagated..yes, an artificial test. |
|
hi @zopyx -- thx for updated input. Yes that makes more sense that for 7MB you had a different payload with a larger number of keys and somehow the internal representation was making it cross the 32MB limit. I agree that main/hi-pri issue is to cleanly catch the error and propagate to client. |
|
Attaching the detailed TS log: |
|
For one INSERT statement (in the user script) the TS gets a lot of CQL requests. It looks like an endless retry loop on the driver side. I'll try to reproduce it from JAVA code. |
|
The issue is reproduced in JAVA bu with bigger JSON string length = 69 MB (46 MB works). |
|
Java test: @Test
public void testPrepareInsertBindLongJson() throws Exception {
LOG.info("Begin test");
// Create table
String createStmt = "CREATE TABLE test_bind (id int PRIMARY KEY, data jsonb);";
session.execute(createStmt);
// Insert data into the test table. Bind by name.
String insertStmt = "INSERT INTO test_bind (id, data) VALUES (?, ?);";
PreparedStatement stmt = session.prepare(insertStmt);
for (int j = 1; j <= 2; ++j) {
StringBuilder builder = new StringBuilder();
builder.append("{\"0000000\":\"1234567890\"");
for (int i = 1; i < (j == 1 ? 5 : 3000000); ++i) {
builder.append(String.format(",\"%07d\":\"1234567890\"", i));
}
String jsonStr = builder.toString() + "}";
LOG.info("Executing INSERT for JSON data length = " + jsonStr.length());
// Insert long JSON value.
session.execute(stmt.bind()
.setInt("id", j)
.setString("data", jsonStr));
LOG.info("INSERT executed");
// Select data from the test table.
String selectStmt = String.format("SELECT * FROM test_bind WHERE id = %d;", j);
ResultSet rs = session.execute(selectStmt);
Row row = rs.one();
// Assert exactly 1 row is returned each time with expected column values.
assertNotNull(row);
assertEquals(j, row.getInt(0));
assertEquals(jsonStr, row.getJson("data"));
}
LOG.info("End test");
} |
|
Root cause: the error code mapping: (value too long) Status::InvalidArgument -> ErrorCode::INVALID_ARGUMENTS -> ErrorCode::STALE_METADATA -> Code::UNPREPARED (YCQL error code). UNPREPARED code is retryable in the driver. Fix: the fixed error code mapping is: (value too long) Status::NotSupported -> ErrorCode::INVALID_REQUEST -> Code::INVALID (non-retryable YCQL error code). |
OlegLoginov
added a commit
that referenced
this issue
Sep 27, 2019
Summary: Fix for #2116: YQL value too long INSERT INTO ybdemo.ugent(id, data) VALUES (?, ?) with overlong JSON data Originally we had the codes mapping: (value too long) Status::InvalidArgument -> ErrorCode::INVALID_ARGUMENTS -> ErrorCode::STALE_METADATA -> Code::UNPREPARED (YCQL error code). UNPREPARED code is retryable in the driver. Now the error code mapping is: (value too long) Status::NotSupported -> ErrorCode::INVALID_REQUEST -> Code::INVALID (non-retryable YCQL error code). Test Plan: ybd --java-test org.yb.cql.TestBindVariable#testPrepareInsertBindLongJson Reviewers: mihnea, neil, kannan Reviewed By: neil, kannan Subscribers: yql, kannan Differential Revision: https://phabricator.dev.yugabyte.com/D7220
|
Fixed by the commmit above. |
OlegLoginov
added a commit
that referenced
this issue
Sep 30, 2019
Summary: Fix for failed org.yb.cql.TestInsertValues#testLargeInsert Old error message: SQL Error: Invalid Arguments. YQL value too long New error message: SQL Error: Invalid Request. Value size (67108872) is longer than max value size supported (67108864) Main fix: 7c76527 Test Plan: ybd --java-test org.yb.cql.TestInsertValues#testLargeInsert Reviewers: mikhail Reviewed By: mikhail Subscribers: yql Differential Revision: https://phabricator.dev.yugabyte.com/D7308
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Inserting a JSON document larger than 7 MB or so using a prepared statement through yb-cassandra-python causes an internal loop of messages
and does not cause a propagation of the error back to the Python client/driver
https://forum.yugabyte.com/t/insert-of-large-json-data-for-cassandra-driver-very-slow/469/17?u=zopyx
The text was updated successfully, but these errors were encountered: