From b2e6231b1533bc1cdd1f2d24370291760a9754a5 Mon Sep 17 00:00:00 2001 From: Kyriakos Akriotis Date: Tue, 16 Jul 2024 15:43:11 +0200 Subject: [PATCH 1/4] added migrating-kafka-services (draft) --- .../migrating-kafka-services.md | 130 ++++++++++++++++++ sidebars.ts | 4 + 2 files changed, 134 insertions(+) create mode 100644 docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md diff --git a/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md new file mode 100644 index 000000000..7fcef17a3 --- /dev/null +++ b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md @@ -0,0 +1,130 @@ +--- +id: migrating-kafka-services +title: Migrating Kafka Services +tags: [cce, elb, hpa, prometheus, prometheus-exporter, prometheus-adapter, metrics, cloudeye, swr] +--- + +# Migrating Kafka Services + +You can migrate Kafka services to connect message producers and +consumers to a new Kafka instance and can even migrate persisted message +data to the new Kafka instance. Kafka services can be migrated in the +following two scenarios: + +- Migrating services to the cloud without downtime + + Services that have high requirements on continuity must be smoothly + migrated to the cloud because they cannot afford a long downtime. + +- Re-deploying services in the cloud + + A Kafka instance deployed within an AZ is not capable of cross-AZ + disaster recovery. For higher reliability, you can re-deploy + services to an instance that is deployed across AZs. + +## Prerequisites + +1. Configure the network environment. + + A Kafka instance can be accessed within a VPC or over a public + network. For public network access, the producer and consumer must + have public access permissions, and the following security group + rules must be configured: + + - **Inbound**: `TCP/9094`, `0.0.0.0/0` -> Access Kafka through the public network (**without** SSL encryption) + - **Inbound**: `TCP/9095`, `0.0.0.0/0` -> Access Kafka through the public network (**with** SSL encryption) + +2. Create a Kafka instance. + + The specifications of the new instance cannot be lower than the + original specifications. For details, see [Creating an Instance](https://docs.otc.t-systems.com/distributed-message-service/umn/creating_an_instance.html). + +3. Create a topic. + + Create a topic with the same configurations as the original Kafka + instance, including the topic name, number of replicas, number of + partitions, message aging time, and whether to enable synchronous + replication and flushing. For details, see [Creating a Topic](https://docs.otc.t-systems.com/cloud-eye/umn/using_the_alarm_function/creating_alarm_notification_topics/creating_a_topic.html). + +## Migration Scheme 1: Migrating the Production First + +Migrate the message production service to the new Kafka instance. After +migration, the original Kafka instance will no longer produce messages. +After all messages of the original Kafka instance are consumed, migrate +the message consumption service to the new Kafka instance to consume +messages of this instance. + +1. Change the Kafka connection address of the producer to that of the + new Kafka instance. +2. Restart the production service so that the producer can send new + messages to the new Kafka instance. +3. Check the consumption progress of each consumer group in the + original Kafka instance until all data in the original Kafka + instance is consumed. +4. Change the Kafka connection addresses of the consumers to that of + the new Kafka instance. +5. Restart the consumption service so that consumers can consume + messages from the new Kafka instance. +6. Check whether consumers consume messages properly from the new Kafka + instance. +7. The migration is completed. + +This is a common migration scheme. It is simple and easy to control on +the service side. During the migration, the message sequence is ensured, +so this scheme is **suitable for scenarios with strict requirements on +the message sequence**. However, latency may occur because there is a +period when you have to wait for all data to be consumed. + +## Migration Scheme 2: Migrating the Production Later + +Use multiple consumers for the consumption service. Some consume +messages from the original Kafka instance, and others consume messages +from the new Kafka instances. Then, migrate the production service to +the new Kafka instance so that all messages can be consumed in time. + +1. Start new consumer clients, set the Kafka connection addresses to + that of the new Kafka instance, and consume data from the new Kafka + instance. + + :::note + Original consumer clients must continue running. Messages are + consumed from both the original and new Kafka instances. + ::: + +2. Change the Kafka connection address of the producer to that of the + new Kafka instance. + +3. Restart the producer client to migrate the production service to the + new Kafka instance. + +4. After the production service is migrated, check whether the + consumption service connected to the new Kafka instance is normal. + +5. After all data in the original Kafka is consumed, close the original + consumption clients. + +6. The migration is completed. + +In this scheme, the migration process is controlled by services. For a +certain period of time, the consumption service consumes messages from +both the original and new Kafka instances. Before the migration, message +consumption from the new Kafka instance has already started, so there is +no latency. However, early on in the migration, data is consumed from +both the original and new Kafka instances, so the messages may not be +consumed in the order that they are produced. This scheme is **suitable +for services that require low latency but do not require strict message +sequence**. + +## How Do I Migrate Persisted Data Along with Services? + +You can migrate consumed data from the original instance to a new +instance by using the open-source tool [MirrorMaker](https://github.com/miguecoll/kafka-mirror-maker). This +tool mirrors the original Kafka producer and consumer into new ones and +migrates data to the new Kafka instance. For details, see +`Using MirrorMaker to Synchronize Data Across Clusters `. + +:::note +Each Open Telekom Cloud Kafka instance stores data in three +replicas. Therefore, the storage space of the new instance should be +three times that of the original single-replica message storage. +::: \ No newline at end of file diff --git a/sidebars.ts b/sidebars.ts index 79d6721a3..90d06eb33 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -54,6 +54,10 @@ const sidebars: SidebarsConfig = { type: 'category', label: 'Distributed Message Service', items: [ + { + type: 'doc', + id: 'best-practices/application-services/distributed-message-service/migrating-kafka-services', + }, { type: 'link', label: '📚 Go to Help Center', From bb59d50e290e71a27196d32491cc4c40a7db2650 Mon Sep 17 00:00:00 2001 From: Kyriakos Akriotis Date: Tue, 16 Jul 2024 16:00:51 +0200 Subject: [PATCH 2/4] updated migrating-kafka-services (draft) --- .../migrating-kafka-services.md | 253 +++++++++++++++++- .../en-us_image_0000001348167557.png | Bin 0 -> 8268 bytes 2 files changed, 251 insertions(+), 2 deletions(-) create mode 100644 static/img/docs/best-practices/application-services/distributed-message-service/en-us_image_0000001348167557.png diff --git a/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md index 7fcef17a3..78605af0b 100644 --- a/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md +++ b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md @@ -121,10 +121,259 @@ You can migrate consumed data from the original instance to a new instance by using the open-source tool [MirrorMaker](https://github.com/miguecoll/kafka-mirror-maker). This tool mirrors the original Kafka producer and consumer into new ones and migrates data to the new Kafka instance. For details, see -`Using MirrorMaker to Synchronize Data Across Clusters `. +[Using MirrorMaker to Synchronize Data Across Clusters](#using-mirrormaker-to-synchronize-data-across-clusters). :::note Each Open Telekom Cloud Kafka instance stores data in three replicas. Therefore, the storage space of the new instance should be three times that of the original single-replica message storage. -::: \ No newline at end of file +::: + + +## Using MirrorMaker to Synchronize Data Across Clusters + +In the following scenarios, [MirrorMaker](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330) can be used to synchronize data +between different Kafka clusters to ensure the availability and +reliability of the clusters: + +- Backup and disaster recovery: An enterprise has multiple data + centers. To prevent service unavailability caused by a fault in one + data center, cluster data is synchronously backed up in multiple + data centers. +- Cluster migration: As enterprises migrate services to the cloud, + data in on-premises clusters must be synchronized with that in cloud + clusters to ensure service continuity. + +### Solution Design + +MirrorMaker can be used to mirror data from the source cluster to the +target cluster. As shown in figure below, +in essence, MirrorMaker first consumes data from the source cluster and +then produces the consumed data to the target cluster. For more +information about MirrorMaker, see [Mirroring data between +clusters](https://kafka.apache.org/documentation/?spm=a2c4g.11186623.0.0.c82870aav6G9no#basic_ops_mirror_maker). + +![**Figure 1** How MirrorMakerworks](/img/docs/best-practices/application-services/distributed-message-service/en-us_image_0000001348167557.png) + +:::warning Restrictions +- The IP addresses and port numbers of the nodes in the source cluster + cannot be the same as those of the nodes in the target cluster. + Otherwise, data will be replicated infinitely in a topic. +- Use MirrorMaker to synchronize data between at least two clusters. + If there is only one cluster, data will be replicated infinitely in + a topic. +::: + +### Installing and Configuring MirrorMaker + +1. Create an ECS that can communicate with the source and target + clusters. For details, see the [ECS + documentation](https://docs.otc.t-systems.com/elastic-cloud-server/index.html). + +2. Log in to the ECS, install JDK, and add the following contents to + **.bash\_profile** in the home directory to configure the + environment variables `JAVA\_HOME` and `PATH`. In this command, + **/opt/java/jdk1.8.0\_151** is the JDK installation path. Change it + to the path where you install JDK. + + ```bash + export JAVA_HOME=/opt/java/jdk1.8.0_151 + export PATH=$JAVA_HOME/bin:$PATH + ``` + Run the `source .bash\_profile` command for the modification to + take effect. + + :::note + Use Oracle JDK instead of ECS's default JDK (for example, OpenJDK), + because ECS's default JDK may not be suitable. Obtain Oracle JDK + 1.8.111 or later from [Oracle's official website](https://www.oracle.com/java/technologies/downloads/#java8). + ::: + +3. Download the binary software package of Kafka 3.3.1. + + ```bash + wget https://archive.apache.org/dist/kafka/3.3.1/kafka_2.12-3.3.1.tgz + ``` + +4. Decompress the binary software package. + + ```bash + tar -zxvf kafka_2.12-3.3.1.tgz + ``` + +5. Go to the binary software package directory and specify the IP + addresses and ports of the source and target clusters and other + parameters in the **connect-mirror-maker.properties** configuration + file in the **config** directory. + + ```bash title="connect-mirror-maker.properties" + # Specify two clusters. + clusters = A, B + A.bootstrap.servers = A_host1:A_port, A_host2:A_port, A_host3:A_port + B.bootstrap.servers = B_host1:B_port, B_host2:B_port, B_host3:B_port + + # Specify the data synchronization direction. The data can be synchronized unidirectionally or bidirectionally. + A->B.enabled = true + + # Specify the topics to be synchronized. Regular expressions are supported. By default, all topics are replicated, for example, foo-.*. + A->B.topics = .* + + # If the following two configurations are enabled, clusters A and B replicate data with each other. + #B->A.enabled = true + #B->A.topics = .* + + # Specify the number of replicas. If multiple topics need to be synchronized and their replica quantities are different, create topics with the same name and replica quantity before starting MirrorMaker. + replication.factor=3 + + # Specify the consumer offset synchronization direction (unidirectionally or bidirectionally). + A->B.sync.group.offsets.enabled=true + + ############################# Internal Topic Settings ############################# + # The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and + # "mm2-offset-syncs.B.internal" + # In the test environment, the value can be 1. In the production environment, it is recommended that the value be greater than 1, for example, 3. + checkpoints.topic.replication.factor=3 + heartbeats.topic.replication.factor=3 + offset-syncs.topic.replication.factor=3 + + # The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and + # "mm2-status.B.internal" + # In the test environment, the value can be 1. In the production environment, it is recommended that the value be greater than 1, for example, 3. + offset.storage.replication.factor=3 + status.storage.replication.factor=3 + config.storage.replication.factor=3 + + # customize as needed + # replication.policy.separator = _ + # sync.topic.acls.enabled = false + # emit.heartbeats.interval.seconds = 5 + ``` + +6. In the binary software package directory, start MirrorMaker to + synchronize data. + + ```bash + ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties + ``` + +7. (Optional) If a topic is created in the source cluster after + MirrorMaker has been started, and the topic data needs to be + synchronized, restart MirrorMaker. + + :::note + To periodically synchronize new topics without restarting MirrorMaker. + `refresh.topics.interval.seconds` is mandatory. Other parameters are optional. + ::: + + + +### Verifying Data Synchronization + +1. View the topic list in the target cluster to check whether there are + source topics. + + :::note + Topic names in the target cluster have a prefix (for example, + `A.`) added to the source topic name. This is a MirrorMaker 2 + configuration for preventing cyclic topic backup. + ::: + +2. Produce and consume messages in the source cluster, view the + consumption progress in the target cluster, and check whether data + has been synchronized from the source cluster to the target cluster. + + If the target cluster is a Open Telekom Cloud Kafka instance, view + the consumption progress on the *Consumer Groups* page. \ No newline at end of file diff --git a/static/img/docs/best-practices/application-services/distributed-message-service/en-us_image_0000001348167557.png b/static/img/docs/best-practices/application-services/distributed-message-service/en-us_image_0000001348167557.png new file mode 100644 index 0000000000000000000000000000000000000000..8debeba107b817ea05385417ea4526df0805c2b7 GIT binary patch literal 8268 zcmbt)WmMb2@-|Xj0xj+i#fy7doE9za?(Ps0XmO{70>uloP#lUAq>xhFi$kyih2ZY_ zhx_q8=lyc;ha`Kl^V`kL&df9WOyaaPRfzCu@X^rFh}2XS-=d*mgrUaPxY(%oQldUd z)CZoIsZ9y=KCjmzI>=XbTN9 zVVrbR8yPLnkR@a3u z9CF?&MjtX|_=bzW-#tYriloR~4YjTxZt+Ba;j;5cHP3SymC(^0JP*tUiFLB;m>3sj5l_OQe8@o?xC@@$o6A z_f=4Iau!BB#?QXP`?`Tpg+5IzCT4Fcgdu+klX%Wi|6FwT$qy}U12fgGLfd}HdvlnN zY2NwWg|VJt#>VFU(9e$pM19atCC-)CX9ap0nO?KDK-l`4+USmYQ}umr(VZFi9#b`2 zB0?@g8KWU+fAHFsR6f>~`O#k1H-rqmYqB}iLLeUn+}ndyewmK;xZMn|c}S9jux3o2 z{}AO1n@RN^mLeTOY;_0Wm_SeF5*5aSoLd5!4zNV1L|)?%(5fdfByI;DbC=622Cn-o zfxKJY~?6R3TUR^&j1(%mW<-K7`sk~$nPTLKZ_vH?=ZpzO?mg0TCzJMR3^gaj8NaB4s7rUulYyVE%shh8^tBG zW#O0gM!!r0t!W&%rO9; zy(xr|cpt8g(kfRvBtEqVd_zR7k=IzF--k<_;Piu+i?^tdeZsm~_w9$i@fS%@v*I@P ze+c}s33l%4fzZs?s&v1%YJeMa4|#-CFWx7fRYm&6vB+>I7v1G5FS?Wu=8w2adb8z8 zx$<$OWeuJ@m2XoNTi~}nG#Jg5`F%xQe|!#bGqUhI;O)JZoFa?KK*N4vl!IiARRKOyaGXT zFkmG(?%D4Y;BKbjWu&M;=@vpG86 zdUc)bT-$*!%2)cFgv6nF1z{wu^PF4)&!3!Wo%W9n{04G03^c_T3GN^CU5$l=S#3w_ zCuaLdW2SN(%_Hvt2t)9%)rqvaprE&(3L=_Q7|n~I=F_fBkriwubZQ1VX(*g zQ-L1_I?@eY96gUEF4_fDl%g&GdjEp741wZKw-C7r((Wb%Ft8dFjKhtJ60>5)Kicj9Itlwyu zH#3h$!D!C-QDCDTd8#S217?z+I0b)C9yg@pKK;EZTWsGJV?4qV0n8lR+ri6s=h19n zq@Df`$ok=ur|LNQa|i}w`PoUX@$@GB~OJvTY)bZ$>F=$()^56W{2XoKMM3QO_ zR7m{yc3`_h=jeU;BaDujL%=YT<wol4 z_NU#r#%xsdJyd3ApUV(&Fc9vmgj$gICYu;$vZ9_gYkm--=jd&24;>Zi=h|xrE$AWz zgRDisVBFmDii#ZxL&7jQY+(UE`tG%C7I^<69?dV%6RQ_g(#4&$lT=+?sbJP%Vb9Ep zH>4=77N#wmdQ7Fy4;|0LZ=aq2G(1#~a^1x+0x`CCVg+oqR7WS=yI_zRz`|v#tH0Lq zM0de0n1g7CdAVg`HGDC=epFY&2!y%uH8l`Nr+$%4X_O3ujZqu4*EcnR1^hvIpAR~y zWzRKHihGdwjK$_m=xWI}zL{$9C!pZHvNyeuHuPV+O*PRLm4Fb{y+kKd0**Gu6SQ}> zx!K+op3z8?X<&Ob`11TEhgnv+;C#t66pV4n_Ut=8wuiLaV@60a6FX8AChoFjdRfbOsL@&1p^v zjF-`8>~LX{oL#mllO78x6$myB4N$g`0=o4TrPJk;OXk6Kl`Pm`rhVVMMHw=9a`HUk z!tg_mhW9J}5V0EjxfbV>X#Q%|18Q&KNfCDmL92tMRsqZ~H&315R5Z|7(q|2 z#-2VL>xocEM-7&k9I;jyUh`%2!12+ri^KX@9tbxWr^^ZK^hSG?%j8?tw)2+N3*6@L7yxTX$t3vkUQUds`}gR z+*c$Am@DLns`MCwGyeofn0Vj0Yp~OohlP%=X4waO{uY}nX#>-qhir_{C3`9ak7j-K z9+~}UTlBfI(%N+_=^qzjxJ2`_m=f<0Z7qvLn%;I{=TDXm`FV~i;gf(t}TM}!=Yw#w$#=?kKc*b&Ncj1ZX=23xk>Eb|h1T zDwfN{7Pa79seBG5l$rsp5k)w12a_+?NVgTFP@FH;)tWI z0IvF7bQgKLyAxAcr*onRWb03}#O8_z78L(*KO85s$1vrmuj?6_ zgRhun!35gj61JE=;ETQ()dG3E3lnq8{_O3#5!d%<$1V)5XM}r2%{C7{^k6S>w}Ba$ zDFm^+3|Zr3UA-0s9fbrjwL@r^t{J(ScV|PifOFmy4A<(GGw^Dv2R#w^(8>@Q(nB4P z=$INt=bY|nWCA8b8Mz-ypu-T_o|?Bn@|A*uZWT~Bz;@eLS9At7ATL74CZqUUstf6C z^fu7e;?5P5Q0P**A-X_*2QS^o-|Q5iLzKM z%$6sfCKH+w$S!W|9kD>#k3m)mGwwBN>an`=qyrB;7rmUt6Pvg1&$2IS!W34zWYcq} z`WB1XCh*{|gMZs^cV-=CDJ(W=2VD@R9YB6ge`|o|l~v4VwY7B+2MRxERsr6R5e)`? za|%PwR^PlV$1bopy9)N_kP$)q{d;$dU-Bn=BYTc`&1AbXQj5#`O->l||lMlS?aRx5HDiB+j6$D?5czPgUgq$dg@m|HJ6Y;a z8JUL06OkJyo+elas50IqQapU@O7UPo|0GqJWHRoSoi^pa`3rQ_T=p5iv`XhF?vfCd z3nxkLc8$qc*Bm@O@b*UkydzLZR*#qeGEhHn+h=_Df(yUTC1Fud_sv_LAF5}>8-XvL z%^Bv68QOF9SG!&x#MNCrvp?QQ@|dE@peBo&njrwZHUr&f+R{=*xLc|7R7`L2F(89p z^N4pA?v0hb(INY$(G@8`Ost$nT?!lhsWXt61#&EnjNdC(najE1Q)-pu!p1B#XQh@q zrzirQ?H&29@aC%pSLlvy?}lvI9CqIMk|}jt39fD;R4&mmsr?b|R2o$`{-MS**)DD~ zIrX#FsOyh1+n8IXt#h`Rr}ih6*HF6QsIAewkmH;R@}oZ@`y==F7WEj0^d3a`GICE! zej%2d{fQy?>BFhN7xUWw#Z`0`-h_snn#g$^k_ghS1K}SG7ml&>7}x&3oFbFQ=ZXdF;Xv0oqIf>tXjm zFGOzNgh*2RRB3HckXi6HZEQ(7S9FdmjgH~6@JQ?JPQfHQN5;uI%MAa6_~LA(&0ghi zQRnRFA~UU&PVoR?IIH;g=<&;xu)7WWcZG-is;wEAqts6T#P13!Ckqxag90-c{+O)W z&mb4y8uIRnL_zpo5&Sqqm7>A zh>9ECI?ZXQte%LETe>>lzv!-si1^d7OKR=dizCm^IOF9~+3G~L&Psy34YMTMq~mt2 z`u^HmwUa}_Ez@f?;}%5Y<8Oo6N2c989G-Y8yOUBkD&4X8h`_+C{S6skibS9M0kvLm zqq2A57ZK+HvOFFcp^SeA9n$wiX8uxYrmcy`fuSjISSn3MpcdWu5kyJtMNS6arrCw_ zFa_XgU!v{Y*XLij_Lggtd+HtW;u8P*!eMOe2d! zWPLVl>~`;i0&vKI0PLW)VP$l*76kVz#_) zRI!a!h;>c%dp{coPWp~pp~av0%s05p@FX-z&4;u=EFZ~K3e&!w_Vrtl6}JS zO*aIC3Gqh+_%?s5nn_oHq-fBp@-q+{T-Up%aU9)&j**MS&DeSP|^vgD8F%s{Bsp3F0yvLh+4$~4b5!+NnuI` zw2!+O0P$Pk18;+vNZYRDL-o@XR%MVgF6}+7qn{9%D&ZYx9t%Tdax2u^{yHWnZXA;H zxwY$DmC|?!hA8WgNBe4!>YoXSvZkFe9O8ubQJAGbjn(h#-VYX#Wcd&xW(Us${Da>2 z*%+1O5gq#5Eqt|{TG4>@v~|`wJse%kjblA^H{My5!Thp{pOsXP^}s>7%uPtl@k^($ z)f0su;s$o7X0xPAl#`iB5lfH3Q7C6d){fMjw-ObZC?} z9Aj+n2{AaBzmxl%L_Ye9L@;3Dcpr{gFkSw|gWOuH~IC}Ew;u@%XZ zK-B}yvd0RkFlepw^;?7CupG5Zxu3B+%|22aB(xb|`LS0K-gEAiH`^G|Y+4ShEqHXx z9cd_!Jvn+)VD-umwSA?%Z?W?u*j^`Sx>v(SVLdcMzbTe-$xK!!S(DtQ$1)0)4&FkiT(OcBI*>~!A>NX zFXo3774^43jae;7nkk;G&Dnt!kbsVRob$tj+Ecr9l#A1lQ!SBC{-~x?UK~PQk@9vG z(H&yW*vJ39N^)_u^p_iYoxFn(395QVl1t(zbkD8(dZ@T21Ovkug(ECwV$mJ9vuQ;m5wJ&mc- zk9Y@4NiN%d=8{=6sSYs4pU~@&$4eF}oO&v$`kVFZzy9UIx^O|2bPmc^0J7Dg$qinC z`((}@>=V$E-D`S-POgz(v4w@AwNOW^|LD1W83v)W@LnZNUXifT?g5#XI+fWXj zwTAzc>hSSH2yt9h8Q}_v(g?N_}-@6RSv{ zQXv+;EgdVSB771R^K8DDvy_<8=IEM6MZJKz`;g;ult0KHpSM-ZbMxkk3GBOmGPddb zPq*~$Qqd?VQ*L#wWBG<_>E^j7kGo(Xy9|)#ArPC25QJvteb(ds(5_?&Y{UVlmjSX{ z^vd)ZH*b5Wf{?_J1~278Ir1U6jOY1A25E>N1 zs8X{Xa{p3??aDtjYXymJZnKeX9_an9f|sXPT(EZH=H0j;cYHz_zkrIr)G+|h`SjYA z`wW5!>!A^-sZ`X}pW(Mqgmfh@8>krR`PHAt(XOmV+|2gkm!E%_d0L7O+Y&d=*Cz(H zH~~F{uYyI06^%D<^!WLVecSz)%lhKi$^^14FC*T#tbni&+Wtzyc95M%_nzVn;QV15 zNb=7&kf4`Hm(gcDCUO?_iCbb07z(?M0Kl`n(Y>9IZATpR9U;OAvHQ`eBzg57az-Om zIRNqmh7G=nKL`4b#nkS``+zYqa@-bk=N|ocDsvbTqGo>cTo#-7x0Ohtl)&i|f7+m% ztMv`Wp9%tEc{oj6mr~cf>=Uug1wi{KtfUNm!%rKuRIxi?*Mi0pY=FQa$1P0~hhF1# zNZJ8T8x8CQ@vH%2c9&ZgN88$dAaVwTna+Qn zzJKmMuM%y7H=#Dnz>xS^E+VObkJ+|Q%%$S&;LqJ2ua71jr&J*&L%zkHaTl?Mei^rb zEpkiaj8<|tzOTu^P}i^m8N5eHwsa)u{?ds{Nzzd#PZNbj(ldX?d}l6Rsi3N~TH~rc zt;O#+uQiIVmh89eN%&_&`n`OT9eu=f5A`K7{o^ZHsn4jzFc^_&;ZVg7`E3lPuvcmi zlA3!fF#(|$*vJng<{XCDfz=U-MVfjou5z$ilVVoB^>B5l~bVW-lv)cZ0@~6diuSK zvngc&nec8aChH>-O}eGAm@p9e$~&fLt(?)ooe-}D`qk;C1-|_JVUQ1hjv>F}_H4xq zW8O$5W#6IVm992Q1;3W@?_4Ev5-|aw21-O2{a+ez0 z;^BuVP+c>IoLs&8ppBiR<*;Mwxw^ltva-e+hjO*4LSV7%exb^I%bu=WkSJDtuRM4i zjU)8XqD1Zr*LetNJK=gNNR@bng(JL^e@i?1T=9(oq%=N!RLF9kve{TjEWVX}0?ZHNhX8 z);i&p%KF{jp~ZvpC8ah<)FX@hKQ?hW&>~z^^@++WTuH0!w28{9t6$+V{~8%UD|1-{ zqQ7lHwKx{rQt(zgBB{&FdPg6*<(AgHP!lyJp>7S0|Nn3JPrGzB?fcaec<@dzsyocj z$}QJD%`U=nIedNyf;vWeiJ~Zbv%d#@`TqWR>)5O)_Ih2J^?#(V|A%J4i`F(Y=IW?! wD^-2{>1q2%%c92q{zU)b)8%m|xRH#;jx*l*$GKFfzXzbHDQPNJ%Ugy2AMOnV00000 literal 0 HcmV?d00001 From 3c88d28ef33fde0d07b1e6fe67e5f9c885728b9c Mon Sep 17 00:00:00 2001 From: Kyriakos Akriotis Date: Tue, 16 Jul 2024 16:02:23 +0200 Subject: [PATCH 3/4] updated migrating-kafka-services (draft) --- .../distributed-message-service/migrating-kafka-services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md index 78605af0b..b5170b497 100644 --- a/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md +++ b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md @@ -1,7 +1,7 @@ --- id: migrating-kafka-services title: Migrating Kafka Services -tags: [cce, elb, hpa, prometheus, prometheus-exporter, prometheus-adapter, metrics, cloudeye, swr] +tags: [kafka, mirror-maker, migration, dms] --- # Migrating Kafka Services From 85cf5be2ae46eceabf102b8f9d0e481dc27f370a Mon Sep 17 00:00:00 2001 From: Kyriakos Akriotis Date: Tue, 16 Jul 2024 16:09:14 +0200 Subject: [PATCH 4/4] updated migrating-kafka-services --- .../migrating-kafka-services.md | 112 +++--------------- 1 file changed, 18 insertions(+), 94 deletions(-) diff --git a/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md index b5170b497..eddc5408c 100644 --- a/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md +++ b/docs/best-practices/application-services/distributed-message-service/migrating-kafka-services.md @@ -265,100 +265,24 @@ clusters](https://kafka.apache.org/documentation/?spm=a2c4g.11186623.0.0.c82870a `refresh.topics.interval.seconds` is mandatory. Other parameters are optional. ::: - +### MirrorMaker Configuration Properties + +| Parameter | Default Value | Description | +|-----------------------------------|----------------------------------------------------------|---------------------------------------------------------------------------------| +| sync.topic.configs.enabled | true | Whether to monitor the source cluster for configuration changes. | +| sync.topic.acls.enabled | true | Whether to monitor the source cluster for ACL changes. | +| emit.heartbeats.enabled | true | Whether to let the connector send heartbeats periodically. | +| emit.heartbeats.interval.seconds | 5 seconds | Heartbeat frequency. | +| emit.checkpoints.enabled | true | Whether to let the connector periodically send the consumer offset information. | +| emit.checkpoints.interval.seconds | 5 seconds | Checkpoint frequency. | +| refresh.topics.enabled | true | Whether to let the connector periodically check for new topics. | +| refresh.topics.interval.seconds | 5 seconds | Frequency of checking for new topics in the source cluster. | +| refresh.groups.enabled | true | Whether to let the connector periodically check for new consumer groups. | +| refresh.groups.interval.seconds | 5 seconds | Frequency of checking for new consumer groups in the source cluster. | +| replication.policy.class | org.apache.kafka.connect.mirror.DefaultReplicationPolicy | Use LegacyReplicationPolicy to imitate MirrorMaker of an earlier version. | +| heartbeats.topic.retention.ms | One day | Used when heartbeat topics are created for the first time. | +| checkpoints.topic.retention.ms | One day | Used when checkpoint topics are created for the first time. | +| offset.syncs.topic.retention.ms | max long | Used when offset sync topics are created for the first time. | ### Verifying Data Synchronization