Java and Python AVRO compatibility - stringType

[slominskir]

The Java AVRO class generator makes Java specific modifications to schemas, that then later when a Schema Registry validation is performed causes new versions of the schema to be added to the Confluent registry.

See:

• https://issues.apache.org/jira/browse/AVRO-2838
• https://issues.apache.org/jira/browse/AVRO-3077
• https://github.com/davidmc24/gradle-avro-plugin/discussions/156
• Schema Registry considers avro.java.string as part of the schema comparison confluentinc/schema-registry#868
• Issues matching on "avro.java.string" : "String" (version 6.0.0) confluentinc/schema-registry#1677
• avro-maven-plugin with stringType config causes 40403 Schema not found confluentinc/schema-registry#1352

[slominskir]

My pull request has been ignored for months: apache/avro#1235. In the meantime, Confluent has added a new flag to work around this issue:

KafkaAvroSerializerConfig.AVRO_REMOVE_JAVA_PROPS_CONFIG = true

Notes:
https://github.com/JeffersonLab/jaws/wiki/AVRO-Schema-Peculiarities#java-specific-logical-types

[slominskir]

It seems the easiest path forward at this point is to simply accept the Java AVRO compiler's demands that AVRO schemas must have Java specific types. Initial testing suggests this fixes the issue - Python scripts appear to silently ignore unrecognized logical type and inherit the logical base type string. Using the new AVRO_REMOVE_JAVA_PROPS_CONFIG option means we would need to move apps to the bleeding edge version of confluent libs, which is a whole can of worms I don't want to open at the moment.

However, I've noticed the AlarmOverrideUnion.avsc is bumped to version 2 the first time set-override.py is executed. However, executing set-activation.py does not bump AlarmActivationUnion.avsc. No Java Kafka Streams apps are even running. Using describe-version.py to dump both version 1 and 2 then running diff reveals they're identical. Maybe a separate issue? More investigation needed.

[slominskir]

I've created a separate issue for this new anomaly: #28.

I'm closing this issue for now, as the ugly fix of simply accepting that our schemas have Java specific logical types appears to be working.