does xk6-kafka support avro union type?

[luciowu]

Hello,

I am trying to use xk6-kafka with Avro format.
It is fine for me to use this example https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_no_schema_registry.js

But if I did a small modification on the code with avro union type (because in our case, we use union type) as the following

const valueSchema = JSON.stringify({
  type: "record",
  name: "Value",
  namespace: "dev.mostafa.xk6.kafka",
  fields: [
    {
      name: "name",
      type : ["null", "string"],
      default: null
    },
  ],
});

export default function () {
  for (let index = 0; index < 2; index++) {
    let messages = [
      {
        value: schemaRegistry.serialize({
          data: {
            name: "xk6-kafka",
          },
          schema: { schema: valueSchema },
          schemaType: SCHEMA_TYPE_AVRO,
        }),
      },
    ];

    writer.produce({ messages: messages });
  }

it will report error
executor=constant-vus scenario=sasl_auth source=stacktrace ERRO[0005] GoError: Failed to encode data, OriginalError: %!w(*errors.errorString=&{cannot decode textual record "dev.mostafa.xk6.kafka.Value": cannot decode textual union: expected: '{'; actual: '"' for key: "name"})

Not sure is this issue related to my code or this plugin. Could you help on it?
Thanks.

[mostafa]

Hey @luciowu,

I use srclient for management of schema and schema registry, which internally uses go-avro. As far as I've looked, go-avro supports union types. I tried to fix your script, but it seems something is lost in translation between JSON.stringify, Goja runtime and avro.NativeFromTextual. And it looks like a bug.
This is not a bug, rather an issue with your script and since you are using a union type, you need to explicitly pass the type of the data you're passing to the name field, which is string, otherwise go-avro won't understand it:

export default function () {
  for (let index = 0; index < 2; index++) {
    let messages = [
      {
        value: schemaRegistry.serialize({
          data: {
            name: { string: "xk6-kafka" }, // This is how the data should be passed to union type
          },
          schema: { schema: valueSchema },
          schemaType: SCHEMA_TYPE_AVRO,
        }),
      },
    ];

    writer.produce({ messages: messages });
  }
}

[luciowu]

Yep. you're right. I just figure out the difference in the go-avro and other programming language libraries.
Now it works.
Thank you very much.