[Bug] NullPointerException when parsing the table_map event

[ruanhang1993]

Search before asking

• I searched in the issues and found nothing similar.

Flink version

1.15.2

Flink CDC version

2.3.0

Database and its version

mysql 8.0.18

Minimal reproduce step

This error occurs when parsing the binlog.

What did you expect to see?

The job runs normally.

What did you see instead?

Caused by: com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
    at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
    at com.ververica.cdc.connectors.mysql.debezium.task.context.MySqlErrorHandler.setProducerThrowable(MySqlErrorHandler.java:89)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1452)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:965)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
    at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
    ... 1 more
Caused by: io.debezium.DebeziumException
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1387)
    ... 5 more
Caused by: java.lang.NullPointerException
    at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.readMetadata(TableMapEventDataDeserializer.java:81)
    at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:42)
    at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:27)
    at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303)
    at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281)
    at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:259)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:952)
    ... 3 more

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[ruanhang1993]

The binlog client can not find the column type 20 in its ColumnType class, which is MYSQL_TYPE_TYPED_ARRAY.
MySQL 8.0.18+ change the MYSQL_TYPE_TYPED_ARRAY to 20, see mysql/mysql-server@9082b6a.

[ruanhang1993]

I have raised a PR to fix this in the mysql-binlog-connector-java(osheroff/mysql-binlog-connector-java#104).

[racoon945]

I have raised a PR to fix this in the mysql-binlog-connector-java(osheroff/mysql-binlog-connector-java#104).

@ruanhang1993
Hi , I have added the code you PR into the project. The NPE issue has been resolved, but a new error has emerged now.

mysql version = 8.0.18

Caused by: io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1679640836000, eventType=TABLE_MAP, serverId=1704062425, headerLength=19, dataLength=89, nextPosition=359034816, flags=0}
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1154)
	... 5 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1679640836000, eventType=TABLE_MAP, serverId=1704062425, headerLength=19, dataLength=89, nextPosition=359034816, flags=0}
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:309)
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281)
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228)
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:230)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:952)
	... 3 more
Caused by: java.io.IOException: Unsupported table metadata field type 216
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventMetadataDeserializer.deserialize(TableMapEventMetadataDeserializer.java:52)
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:59)
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:37)
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303)
	... 7 more

[racoon945]

I found that the problematic table statement appears here.

ALTER TABLE `my_table`
    ADD INDEX `k_source` ((CAST(`source` -> '$[*].code' AS char(10) ARRAY)));

We have used a new index type introduced in MySQL 8.0: multi-value index.
My current approach is to ignore the binlog of the problematic table.

[leonardBang]

@ruanhang1993 Could you take a look @racoon945 's case ?

[leonardBang]

I think we need consider @racoon945 's input, but we can create a new issue to track and fix it.
CC: @ruanhang1993

[ruanhang1993]

@racoon945 I add a test but cannot reproduce the error #2192 . Would you mind taking a look at the test?