GoError: Failed to encode data, OriginalError: cannot decode textual record

[Shabda1988]

Hi team,

Here is my code snippet

Data body was long ..so i just commented data there ...let me know if data body is required .
I am getting below error while running the k6 script .

`import {LoadJKS,
TLS_1_2,
SASL_SCRAM_SHA512,
Writer,
SchemaRegistry,
SCHEMA_TYPE_AVRO,
KEY,
VALUE,
RECORD_NAME_STRATEGY} from "k6/x/kafka"; // import kafka extension
import { check } from "k6";

const surl="https://schema-registry-url";
const topic = "topic";

const jks = LoadJKS({
path: "./keystore-sr.jks",
password: "password1",
clientCertAlias: "alias1",
clientKeyAlias: "alias1",
clientKeyPassword: "password1",
});

const jks1 = LoadJKS({
path: "./truststore.jks",
password: "password2",
serverCaAlias: "serveralias"
});

const tlsConfig = {
enableTls: true,
insecureSkipTlsVerify: false,
minVersion: TLS_1_2,
clientCertPem: jks["clientCertsPem"][0], // The first certificate in the chain
clientKeyPem: jks["clientKeyPem"],
serverCaPem: jks1["serverCaPem"],
};

const saslConfig= {
username: "demo-username",
password: "sasl-password",
algorithm: SASL_SCRAM_SHA512,
}

const writer = new Writer({
brokers: ["broker:443"],
topic: topic,
sasl: saslConfig,
tls: tlsConfig
});

const schemaRegistry = new SchemaRegistry({
url: surl,
tls: tlsConfig
});

const valueSchemaObject = schemaRegistry.getSchema({
subject: '${topic}-value',
schemaType: SCHEMA_TYPE_AVRO,
});

export default function () {
try{ {
let messages = [
{

    value: schemaRegistry.serialize({
      data: {
               //data
               },
      schema: valueSchemaObject,
      schemaType: SCHEMA_TYPE_AVRO,
    }),
    headers: {
      Authorization: "Bearer token",
    },
   
  }
];
writer.produce({ messages: messages });

} }

catch (e) {
console.error('Unhandled exception');
console.log(e);
console.log(e.stack);
console.log(JSON.stringify(e));
}
}`

{"value":{"code":2001,"message":"Failed to encode data","original_error":{}}} source=console INFO[0004] GoError: Failed to encode data, OriginalError: cannot decode textual record "com.m.Plan": cannot decode textual union: cannot decode textual map: cannot determine codec: "bookingNumber" for key: "booking" at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass.func4 (native) at file:///C:/Users/SAD121/OneDrive%20-%20Maersk%20Group/Desktop/jksfile/kafka1.js:78:40(2459) source=console INFO[0004] {"value":{"code":2001,"message":"Failed to encode data","original_error":{}}} source=console INFO[0004] 2024/04/23 17:11:00 No server's CA alias provided, skipping server's CA

[mostafa]

Hey @Shabda1988,

See if this #266 (comment) solves your problem.

[Shabda1988]

Hey @mostafa thanks for replying .

I tried the way by appending the namespace to the field in data body and after running , error shifted to other field ....other field also had same namespace but when i appended the namespace name to that field and re ran again , error came for the first field again giving the same reason

Let me know if you understood it :)

[mostafa]

@Shabda1988
I won't be able to help you if you don't provide the schema and sample data to me. Please remove any sensitive information, possibly by renaming fields in the schema and anonymizing data.

[Shabda1988]

@mostafa I cant give the whole avro but posting here a part of it with modified names .

{ "type": "record", "namespace": "com.demo", "name": "demo1", "doc": "The", "fields": [ { "name": "book", "doc": "doc", "type": [ "null", { "type": "record", "namespace": "com.demo.one", "name": "Book", "doc": "An", "fields": [ { "name": "bookNumber", "doc": "The 456", "type": [ "null", "string" ], "default": null }, { "name": "bookStatus", "doc": "Anything", "type": [ "null", { "type": "enum", "name": "BookType", "namespace": "com.demo.one.book", "symbols": [ "CAR", "BIKE", "SCHOOL" ] } ], "default": null } ] } ], "default": null } ] }

Sample data :

`{

"book": {
"bookNumber": null,
"bookStatus": "CAR",

}`

This is just a small part of a big thing.. see if it helps .

[Shabda1988]

@mostafa Let me know if its fine or is it possible we can get on some call . Schema is very big around 6k lines and same is the case with data which i am trying to post ...so its becoming difficult to understand what and where to change and why the issue is happening.

[Shabda1988]

@mostafa Can you please help with the above provided schema and sample data provided .

[mostafa]

Hey @Shabda1988,

I created a small tool to test your schema and data outside xk6-kafka and made it ready to be ingested into your script as is: https://github.com/mostafa/nested-avro-schema. I also provided an example Avro schema and data in the example directory. Feel free to use it and provide feedback.

[Shabda1988]

@mostafa Thanks a lot for creating this utility . Its very easy to use and it pinpoints where the issue is and what needs to be done for that field . It fixed my issue and now am able to post message on kafka topic . Very nice utility . Thank you for creating .

[mostafa]

@Shabda1988 Happy that I have been of help and that your issue is resolved.