[Bug]: MM2 The request included a message larger than the max message size the server will accept

[sentoa]

Bug Description

Hello,
I am trying to move data from my old Kafka cluster (not running Strimzi) to a Strimzi cluster running slidelined in the same namespace.

After a few minutes of moving data, Mirrormaker2 stops and throws the following error:
The request included a message larger than the max message size the server will accept

I tried to configure both Mirrormaker and Strimzi, however, I am unable to fit the larger message size to my Strimzi cluster.
The config files sent under the logs contains a lot of trail and error settings from many config files I found, but I am hoping someone can shed a light to what is going on and what I am missing (or we whether it may be an issue with Strimzi MM2)

Errors thrown from Mirrormaker

From the Kafka Mirrormaker pod logs

org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback
        at org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:322)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.prepareToSendRecord(WorkerSourceTask.java:128)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.sendRecords(AbstractWorkerSourceTask.java:400)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:359)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)
        at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

The Kafka Mirrormaker2 resource

    Tasks:
      Id:     0
      State:  FAILED
      Trace:  org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback
              at org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:322)
              at org.apache.kafka.connect.runtime.WorkerSourceTask.prepareToSendRecord(WorkerSourceTask.java:128)
              at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.sendRecords(AbstractWorkerSourceTask.java:400)
              at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:359)
              at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
              at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)
              at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)
              at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
              at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

Steps to reproduce

1. Apply Strimzi Operator
2. Apply Kafka Resource and let the cluster reconcile
3. Add kafkaMirrorMaker2 resource and let it run
4. I observe in logs and KOWL that data is moving
5. Data is stopped coming in and MM2 throws the max size error

Expected behavior

No response

Strimzi version

0.35.1

Kubernetes version

1.25.5

Installation method

HELM chart through Terraform

Infrastructure

AKS

Configuration files and logs

Configuration

Kafka

Spec:
  Entity Operator:
    Topic Operator:
    User Operator:
  Kafka:
    Config:
      default.replication.factor:                3
      inter.broker.protocol.version:             3.4.0
      max.request.size:                          604857600
      min.insync.replicas:                       2
      offsets.topic.replication.factor:          3
      socket.receive.buffer.bytes:               10240000
      socket.request.max.bytes:                  1048576000
      socket.send.buffer.bytes:                  10240000
      transaction.state.log.min.isr:             2
      transaction.state.log.replication.factor:  3
    Listeners:
      Name:    plain
      Port:    9092
      Tls:     false
      Type:    internal
      Name:    tls
      Port:    9093
      Tls:     true
      Type:    internal
    Replicas:  3
    Storage:
      Type:  jbod
      Volumes:
        Class:         azuredisk-non-premium
        Delete Claim:  false
        Id:            0
        Size:          500Gi
        Type:          persistent-claim
    Template:
      Kafka Container:
        Env:
          Name:   KAFKA_DESTINATION_MAX_MESSAGE_BYTES
          Value:  104857600
          Name:   KAFKA_SOURCE_MAX_MESSAGE_BYTES
          Value:  104857600
          Name:   STRIMZI_KAFKA_BROKER_MAX_MESSAGE_BYTES
          Value:  10485760
          Name:   STRIMZI_KAFKA_BROKER_MAX_REQUEST_SIZE
          Value:  10485760
          Name:   KAFKA_HEAP_OPTS
          Value:  -Xms8024M
          Name:   KAFKA_MIN_INSYNC_REPLICAS
          Value:  2
          Name:   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
          Value:  3
          Name:   KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
          Value:  2
          Name:   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
          Value:  3
    Version:      3.4.0

Mirrormaker

    Manager:         strimzi-cluster-operator
    Operation:       Update
    Subresource:     status
    Time:            2023-06-21T03:51:48Z
  Resource Version:  727997363
  UID:               456290f0-1607-4cd2-837d-2fe3e0aeecb9
Spec:
  Clusters:
    Alias:              old_kafka
    Bootstrap Servers:  kafka-svc:9092
    Alias:              strimzi_kafka
    Bootstrap Servers:  strimzi-westeurope-kafka-bootstrap:9092
    Config:
      config.storage.replication.factor:  3
      max.request.size:                   209715200
      message.max.bytes:                  209715200
      offset.storage.replication.factor:  3
      producer.buffer.memory:             1048576000
      producer.max.request.size:          209715200
      status.storage.replication.factor:  3
  Connect Cluster:                        strimzi_kafka
  Mirrors:
    Source Cluster:  old_kafka
    Source Connector:
      Config:
        max.request.size:                       209715200
        message.max.bytes:                      209715200
        offset-syncs.topic.replication.factor:  3
        producer.buffer.memory:                 1048576000
        producer.max.request.size:              209715200
        replication.factor:                     3
    Target Cluster:                             strimzi_kafka
  Replicas:                                     1
  Template:
    Connect Container:
      Env:
        Name:   KAFKA_PRODUCER_MAX_REQUEST_SIZE
        Value:  209715200
        Name:   KAFKA_MESSAGE_MAX_BYTES
        Value:  209715200
  Version:      3.4.0

Additional context

No response

[scholzj]

The error suggests you are trying to send bigger messages then you configured it for. You should probably start by sharing the configurations (the custom resources) and logs from the Kafka clusters and Mirror Maker.

[sentoa]

Kafka CRD

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta2
  kind: Kafka
  metadata:
    annotations:
      meta.helm.sh/release-name: strimzi-westeurope
      meta.helm.sh/release-namespace: test1
    creationTimestamp: "2023-06-19T07:42:30Z"
    generation: 11
    labels:
      app.kubernetes.io/managed-by: Helm
    name: strimzi-westeurope
    namespace: test1
    resourceVersion: "727184138"
    uid: fefaba6d-8564-4f28-8e24-b47949f5ec47
  spec:
    entityOperator:
      topicOperator: {}
      userOperator: {}
    kafka:
      config:
        default.replication.factor: 3
        inter.broker.protocol.version: 3.4.0
        max.request.size: "604857600"
        min.insync.replicas: 2
        offsets.topic.replication.factor: 3
        socket.receive.buffer.bytes: 10240000
        socket.request.max.bytes: 1048576000
        socket.send.buffer.bytes: 10240000
        transaction.state.log.min.isr: 2
        transaction.state.log.replication.factor: 3
      listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        tls: true
        type: internal
      replicas: 3
      storage:
        type: jbod
        volumes:
        - class: azuredisk-non-premium
          deleteClaim: false
          id: 0
          size: 500Gi
          type: persistent-claim
      template:
        kafkaContainer:
          env:
          - name: KAFKA_DESTINATION_MAX_MESSAGE_BYTES
            value: "104857600"
          - name: KAFKA_SOURCE_MAX_MESSAGE_BYTES
            value: "104857600"
          - name: STRIMZI_KAFKA_BROKER_MAX_MESSAGE_BYTES
            value: "10485760"
          - name: STRIMZI_KAFKA_BROKER_MAX_REQUEST_SIZE
            value: "10485760"
          - name: KAFKA_HEAP_OPTS
            value: -Xms1024M
          - name: KAFKA_MIN_INSYNC_REPLICAS
            value: "2"
          - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
            value: "3"
          - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
            value: "2"
          - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
            value: "3"
      version: 3.4.0
    zookeeper:
      replicas: 3
      storage:
        class: azuredisk-non-premium
        deleteClaim: false
        size: 50Gi
        type: persistent-claim
      template:
        pod:
          metadata:
            annotations:
              traffic.sidecar.istio.io/excludeInboundPorts: 2888,3888
              traffic.sidecar.istio.io/excludeOutboundPorts: 2888,3888
            labels:
              sidecar.istio.io/inject: "false"

Mirrormaker CRD

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta2
  kind: KafkaMirrorMaker2
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"KafkaMirrorMaker2","metadata":{"annotations":{},"name":"kafka-main-to-strimzi","namespace":"test1"},"spec":{"clusters":[{"alias":"old_kafka","bootstrapServers":"kafka-svc:9092"},{"alias":"strimzi_kafka","bootstrapServers":"strimzi-westeurope-kafka-bootstrap:9092","config":{"config.storage.replication.factor":3,"max.request.size":8388608,"offset.storage.replication.factor":3,"producer.buffer.memory":"104857600","producer.max.request.size":8388608,"status.storage.replication.factor":3}}],"connectCluster":"strimzi_kafka","mirrors":[{"sourceCluster":"old_kafka","sourceConnector":{"config":{"max.request.size":8388608,"offset-syncs.topic.replication.factor":3,"producer.buffer.memory":"104857600","producer.max.request.size":8388608,"replication.factor":3}},"targetCluster":"strimzi_kafka"}],"replicas":1,"version":"3.4.0"}}
    creationTimestamp: "2023-06-19T08:05:41Z"
    generation: 5
    name: kafka-main-to-strimzi
    namespace: test1
    resourceVersion: "727997363"
    uid: 456290f0-1607-4cd2-837d-2fe3e0aeecb9
  spec:
    clusters:
    - alias: old_kafka
      bootstrapServers: kafka-svc:9092
    - alias: strimzi_kafka
      bootstrapServers: strimzi-westeurope-kafka-bootstrap:9092
      config:
        config.storage.replication.factor: 3
        max.request.size: 209715200
        message.max.bytes: 209715200
        offset.storage.replication.factor: 3
        producer.buffer.memory: "1048576000"
        producer.max.request.size: 209715200
        status.storage.replication.factor: 3
    connectCluster: strimzi_kafka
    mirrors:
    - sourceCluster: old_kafka
      sourceConnector:
        config:
          max.request.size: 209715200
          message.max.bytes: 209715200
          offset-syncs.topic.replication.factor: 3
          producer.buffer.memory: "1048576000"
          producer.max.request.size: 209715200
          replication.factor: 3
      targetCluster: strimzi_kafka
    replicas: 1
    template:
      connectContainer:
        env:
        - name: KAFKA_PRODUCER_MAX_REQUEST_SIZE
          value: "209715200"
        - name: KAFKA_MESSAGE_MAX_BYTES
          value: "209715200"
    version: 3.4.0
  status:
    conditions:
    - lastTransitionTime: "2023-06-21T03:51:48.521758020Z"
      status: "True"
      type: Ready
    connectors:
    - connector:
        state: RUNNING
        worker_id: 10.1.8.138:8083
      name: old_kafka->strimzi_kafka.MirrorSourceConnector
      tasks:
      - id: 0
        state: FAILED
        trace: "org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception
          from producer send callback\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:322)\n\tat
          org.apache.kafka.connect.runtime.WorkerSourceTask.prepareToSendRecord(WorkerSourceTask.java:128)\n\tat
          org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.sendRecords(AbstractWorkerSourceTask.java:400)\n\tat
          org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:359)\n\tat
          org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)\n\tat
          org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)\n\tat
          org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)\n\tat
          org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)\n\tat
          java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat
          java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat
          java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat
          java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat
          java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.kafka.common.errors.RecordTooLargeException:
          The request included a message larger than the max message size the server
          will accept.\n"
        worker_id: 10.1.8.138:8083
      type: source
    labelSelector: strimzi.io/cluster=kafka-main-to-strimzi,strimzi.io/name=kafka-main-to-strimzi-mirrormaker2,strimzi.io/kind=KafkaMirrorMaker2
    observedGeneration: 5
    replicas: 1
    url: http://kafka-main-to-strimzi-mirrormaker2-api.test1.svc:8083
kind: List
metadata:
  resourceVersion: ""

Mirrormaker logs

2023-06-21 10:45:04,732 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-11 to 47 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,732 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-4 to 49 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,732 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-16 to 49 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,732 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-17 to 47 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,732 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-3 to 50 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-24 to 44 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-23 to 47 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-13 to 47 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-18 to 44 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-22 to 49 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-2 to 45 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-8 to 45 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-12 to 44 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-19 to 47 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-14 to 45 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-1 to 47 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-6 to 44 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-7 to 47 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:04,733 INFO [Producer clientId=mirrormaker2-cluster--offsets] Resetting the last seen epoch of partition mirrormaker2-cluster-offsets-21 to 50 since the associated topicId changed from null to hsHwI5vZSw6ba8RLfUHvvA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | mirrormaker2-cluster--offsets]
2023-06-21 10:45:06,233 ERROR [old_kafka->strimzi_kafka.MirrorSourceConnector|task-0] WorkerSourceTask{id=old_kafka->strimzi_kafka.MirrorSourceConnector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-old_kafka->strimzi_kafka.MirrorSourceConnector-0]
org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback
	at org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:322)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.prepareToSendRecord(WorkerSourceTask.java:128)
	at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.sendRecords(AbstractWorkerSourceTask.java:400)
	at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:359)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)
	at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)
	at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

[fallen-up]

@scholzj
we got the same case, but the reason is that mm2 ignores the compression in the source topic and tries to write the decompressed message to the target.
found this https://forum.confluent.io/t/mirror-maker-2-connector-trying-to-write-bigger-messages-than-origin/5167/3
but there are problems with the implementation.

maybe you have a recommendation on how to run mm2 for this case? it needs to repeat compression for source-topics.
thanks a lot

[scholzj]

My recommendation is to not send so huge messages. It is not the right way to use Kafka. If you need to share large data, you can for example offload them into object storage and send a pointer to it in a small Kafka message. But that is likely too late for you.

The advice from the link you share seems reasonable, but I do not have any examples of how to use it as I have never done it.