initial commit Commerce DB Sync

SAP · Nov 15, 2022 · 55f156b · 55f156b
1 parent 7c9c3c4
commit 55f156b
Show file tree

Hide file tree

Showing 216 changed files with 16,613 additions and 238 deletions.
diff --git a/DISCLAIMER.md b/DISCLAIMER.md
@@ -0,0 +1 @@
+These Project development objects are not managed or delivered or intended for future inclusion as a standard component of the SAP Software. Therefore, at Project closure, these Project development objects will not include any further support services, defect resolution, maintenance, or upgrades or in any way be within scope of SAP support obligations for licensed SAP Software. Licensee is solely responsible for supporting such objects. SAP does not assure the compatibility of such objects with future releases of SAP Software or other SAP solutions.
diff --git a/LICENSE b/LICENSE
diff --git a/README.md b/README.md
@@ -1,37 +1,87 @@
-# SAP Repository Template
-
-Default templates for SAP open source repositories, including LICENSE, .reuse/dep5, Code of Conduct, etc... All repositories on github.com/SAP will be created based on this template.
-
-## To-Do
-
-In case you are the maintainer of a new SAP open source project, these are the steps to do with the template files:
-
-- Check if the default license (Apache 2.0) also applies to your project. A license change should only be required in exceptional cases. If this is the case, please change the [license file](LICENSE).
-- Enter the correct metadata for the REUSE tool. See our [wiki page](https://wiki.wdf.sap.corp/wiki/display/ospodocs/Using+the+Reuse+Tool+of+FSFE+for+Copyright+and+License+Information) for details how to do it. You can find an initial .reuse/dep5 file to build on. Please replace the parts inside the single angle quotation marks < > by the specific information for your repository and be sure to run the REUSE tool to validate that the metadata is correct.
-- Adjust the contribution guidelines (e.g. add coding style guidelines, pull request checklists, different license if needed etc.)
-- Add information about your project to this README (name, description, requirements etc). Especially take care for the <your-project> placeholders - those ones need to be replaced with your project name. See the sections below the horizontal line and [our guidelines on our wiki page](https://wiki.wdf.sap.corp/wiki/display/ospodocs/Guidelines+for+README.md+file) what is required and recommended.
-- Remove all content in this README above and including the horizontal line ;)
-
-***
-
-# Our new open source project
-
-## About this project
-
-*Insert a short description of your project here...*
-
-## Requirements and Setup
-
-*Insert a short description what is required to get your project running...*
-
-## Support, Feedback, Contributing
-
-This project is open to feature requests/suggestions, bug reports etc. via [GitHub issues](https://github.com/SAP/<your-project>/issues). Contribution and feedback are encouraged and always welcome. For more information about how to contribute, the project structure, as well as additional contribution information, see our [Contribution Guidelines](CONTRIBUTING.md).
-
-## Code of Conduct
-
-We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone. By participating in this project, you agree to abide by its [Code of Conduct](CODE_OF_CONDUCT.md) at all times.
-
-## Licensing
-
-Copyright (20xx-)20xx SAP SE or an SAP affiliate company and <your-project> contributors. Please see our [LICENSE](LICENSE) for copyright and license information. Detailed information including third-party components and their licensing/copyright information is available [via the REUSE tool](https://api.reuse.software/info/github.com/SAP/<your-project>).
+# SAP Commerce DB Sync
+
+[![REUSE status](https://api.reuse.software/badge/github.com/SAP-samples/commerce-migration-toolkit)](https://api.reuse.software/info/github.com/SAP-samples/commerce-migration-toolkit)
+
+SAP Commerce DB Sync performs table-to-table replication in single-directionally manner between two SAP Commerce instances (onPrem to Cloud) or between SAP Commerce and an external database.
+
+SAP Commerce DB Sync is implemented as SAP Commerce extensions and it does not require any third-party ETL.
+
+There are two main use cases:
+* __Replicate data across an external database__: you can push data regularly in batch mode through a Commerce Cloud cronjob and synchronize to an external database. A typical use case is for analytics and reporting purpose when you need direct JDBC access to the database to run analytic jobs.
+* __Data migration__: paired with the self-service media process described on [this CXWorks article](https://www.sap.com/cxworks/article/2589632453/migrate_to_sap_commerce_cloud_migrate_media_with_azcopy), it allows to self-service a one-shot data migration from the on-premise SAP Commerce environment to a SAP Commerce Cloud subscription.
+
+# Getting started
+
+* [User Guide for Data Replication](docs/user/USER-GUIDE-DATA-REPLICATION.md) Go through the details about Data replication between SAP Commerce Cloud and an external database. 
+* [User Guide for Data Migration](docs/user/USER-GUIDE-DATA-MIGRATION.md) When ready to start the migration activities, follow the instructions in the User Guide to trigger the data migration.
+* [Configuration Guide](docs/configuration/CONFIGURATION-GUIDE.md) The extensions ship with a default configuration that may need to be adjusted depending on the desired behaviour. This guide explains how different features and behaviours can be configured.
+* [Security Guide](docs/security/SECURITY-GUIDE.md) A data migration typically features sensitive data and uses delicate system access. Make sure you have read the Security Guide before you proceed with any migration activities and thereby acknowledge the security recommendations stated in the guide.
+* [Performance Guide](docs/performance/PERFORMANCE-GUIDE.md) Performance is crucial for any data migration, not only for large databases but also generally to reduce the time of the cut-over window. The performance guide explains the basic concept of performance tuning and also provides benchmarks that will give you an impression of how to estimate the cutover time window.
+* [Developer Guide](docs/developer/DEVELOPER-GUIDE.md) If you want to contribute please read this guide.
+* [Troubleshooting Guide](docs/troubleshooting/TROUBLESHOOTING-GUIDE.md) A collection of common problems and how to tackle them.
+
+# Features Overview
+
+* Database Connectivity
+  * Multipe supported databases: Oracle, MySQL, HANA, MSSQL
+  * UI based connection validation
+* Schema Differences
+  * UI based schema differences detector
+  * Automated target schema adaption
+    * Table creation / removal
+    * Column creation / removal
+  * Configurable behaviour
+* Data Copy
+  * UI based copy trigger
+  * Configurable target table truncation
+  * Configurable index disabling
+  * Read/write batching with configurable sizes
+  * Copy parallelization
+  * Cluster awareness
+  * Column exclusions
+  * Table exclusions/inclusions
+  * Incremental mode (delta)
+  * Custom tables
+  * Staged approach using table prefix
+* Reporting / Audit
+  * Automated reporting for schema changes
+  * Automated reporting for copy processes
+  * Stored on blob storage
+  * Logging of all actions triggered from the UI
+
+# Compatibility
+
+  * SAP Commerce (>=1811)
+  * Tested with source databases:
+    * Azure SQL
+    * MySQL (5.7)
+    * Oracle (XE 11g)
+    * HANA (express 2.0) and HANA Cloud
+  * Tested with target databases:
+    * Azure SQL
+    * Oracle (XE 11g)
+    * HANA (express 2.0) and HANA Cloud
+
+# Performance
+
+Commerce DB Sync has been built to offer reasonable performance with large amount of data using the following design: 
+* Table to table replication using JDBC (low level)
+* Selection of tables so we do not need a full synchronization in particular for large technical table (task logs, audit logs...)
+* Multi-threaded and can manage multiple tables at the same time 
+* Using UPSERT (INSERT/UPDATE)
+* Use read replica Commerce database as a source database
+
+# Demo Video
+Here is a video that presents how to use SAP Commerce DB sync (formerly known as CMT) for data migration from onPrem to Cloud: 
+  https://sapvideoa35699dc5.hana.ondemand.com/?entry_id=1_gxduwrl3
+
+# How to Obtain Support
+
+This repository is provided "as-is"; no support is available.
+
+Find more information about SAP Commerce Cloud Setup on our [help site](https://help.sap.com/viewer/product/SAP_COMMERCE_CLOUD_PUBLIC_CLOUD/LATEST/en-US).
+
+With regards Commerce DB Sync, access to the database for customers is and will not be possible in the future and SAP does not provide any additional support on Commerce DB Sync in particular. Support can be bought as paid engagement from SAP Consulting only.
+
+# License
+Copyright (c) 2022 SAP SE or an SAP affiliate company. All rights reserved. This project is licensed under the Apache Software License, version 2.0 except as noted otherwise in the [LICENSE file](LICENSE).
diff --git a/commercedbsync/.classpath b/commercedbsync/.classpath
@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<classpath>
+	<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
+	<classpathentry kind="lib" path="web/webroot/WEB-INF/lib/displaytag-1.2.jar"/>
+	<classpathentry kind="lib" path="web/webroot/WEB-INF/lib/jstl-impl-1.2.jar"/>
+	<classpathentry exported="true" kind="lib" path="resources"/>
+	<classpathentry kind="src" output="eclipsebin/classes" path="src"/>
+	<classpathentry kind="src" output="eclipsebin/web/classes" path="web/src"/>
+	<classpathentry kind="src" output="eclipsebin/classes" path="testsrc"/>
+	<classpathentry kind="src" output="eclipsebin/classes" path="gensrc"/>
+	<classpathentry kind="src" path="/platform"/>
+	<classpathentry kind="lib" path="/platform/tomcat/lib/tomcat-dbcp.jar" sourcepath="/tmp/.org.sf.feeling.decompiler1586659330919/source/tomcat-dbcp-8.5.32-sources.jar"/>
+	<classpathentry combineaccessrules="false" kind="src" path="/commercedbsynctypesystem"/>
+	<classpathentry kind="output" path="eclipsebin/notused"/>
+</classpath>
diff --git a/commercedbsync/.springBeans b/commercedbsync/.springBeans
@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<beansProjectDescription>
+	<version>1</version>
+	<pluginVersion><![CDATA[2.2.4.RELEASE]]></pluginVersion>
+	<configSuffixes>
+		<configSuffix><![CDATA[xml]]></configSuffix>
+	</configSuffixes>
+	<enableImports><![CDATA[true]]></enableImports>
+	<configs>
+		<config>resources/commercedbsync-spring.xml</config>
+		<config>web/webroot/WEB-INF/commercedbsync-web-spring.xml</config>
+	</configs>
+	<configSets>
+	</configSets>
+</beansProjectDescription>
diff --git a/commercedbsync/buildcallbacks.xml b/commercedbsync/buildcallbacks.xml
@@ -0,0 +1,161 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!--
+  ~  Copyright: 2022 SAP SE or an SAP affiliate company and commerce-db-synccontributors.
+  ~  License: Apache-2.0
+  ~
+  -->
+
+<!--
+ All hybris buildcallbacks.xml macrodefinitions:
+ 
+ Build/Documentation
+ 
+	 before/after ant macro "clean"
+		 <macrodef name="commercedbsync_before_clean"/>
+		 <macrodef name="commercedbsync_after_clean"/>
+	 
+	 before/after ant macro "build"
+		 <macrodef name="commercedbsync_before_build"/>
+		 <macrodef name="commercedbsync_after_build"/>
+		 
+	 before/after ant macro "compile_core" - the core module of the extension
+		 <macrodef name="commercedbsync_before_compile_core">
+		 <macrodef name="commercedbsync_after_compile_core">
+	 
+	 before/after ant macro "compile_web" - the web module of the extension
+		 <macrodef name="commercedbsync_before_compile_web" />
+		 <macrodef name="commercedbsync_after_compile_web" />
+	 
+	 before/after ant macro "compile_hmc" - the hmc module of the extension
+		 <macrodef name="commercedbsync_before_compile_hmc" />
+		 <macrodef name="commercedbsync_after_compile_hmc" />
+ 
+ Preparing extension
+ 
+	 will be called in the beginning of the ant call and only once (also when using multiple 
+	 ant targets e.g. ant build yunittest)	
+		 <macrodef name="commercedbsync_only_once_prepare"/>
+ 
+ Creating ear module/production
+ 
+	 before/after ant macro "ear"
+		 <macrodef name="commercedbsync_before_ear"/>
+		 <macrodef name="commercedbsync_after_ear"/>
+
+	 before/after ant macro "production" - for hybris server only
+		 <macrodef name="commercedbsync_before_production" />
+		 <macrodef name="commercedbsync_after_production" />
+ 
+ JUnit Test
+ 
+	 before/after ant macro "yunitinit" 
+		 <macrodef name="commercedbsync_before_yunitinit" />
+		 <macrodef name="commercedbsync_after_yunitinit" />
+	 
+	 before/after ant macro "yunit"
+		 <macrodef name="commercedbsync_before_yunit" />
+		 <macrodef name="commercedbsync_after_yunit" /> 
+		 
+ Distribution package
+ 
+ 	 before/after ant macro "dist" - internal target; only for use when platform is available in source code
+		 <macrodef name="commercedbsync_after_dist"/>
+		 <macrodef name="commercedbsync_before_dist"/>
+	 
+	 before/after ant macro "dist_copy" - internal target; only for use when platform is available in source code
+		 <macrodef name="commercedbsync_before_dist_copy"/>
+		 <macrodef name="commercedbsync_after_dist_copy"/>
+		  
+ 	 With these filters you can override the default extension filters defined in platform/resources/ant/dist/filtersets.xml
+ 	  	 <patternset id="extension.commercedbsync.binary.filter">
+ 	  	  	 <patternset refid="extension.filter" />
+ 	  	  	 <exclude name="**/*-source.jar" />
+ 	  	 </patternset>
+ 	  	 <patternset id="extension.commercedbsync.source.filter">
+ 	  	  	 <exclude name="**/bin/**" />
+ 	  	 </patternset>
+     With this filter you can decide what should be excluded from development zip.
+ 	  	 <patternset id="extension.commercedbsync.devzip.filter">
+ 	  	 Include all files from extension.source.filter.
+ 	  	     <patternset refid="extension.source.filter" />
+         Exclude unwanted files.
+ 	  	  	 <exclude name="lib/exclude-me.jar" />
+ 	  	 </patternset>
+ 
+--><project name="commercedbsync_buildcallbacks">
+
+    <macrodef name="commercedbsync_after_build">
+
+        <sequential>
+
+            <patchAzureCloudUtils/>
+
+        </sequential>
+
+    </macrodef>
+
+    <macrodef name="commercedbsync_before_ear">
+
+        <sequential>
+
+            <patchAzureCloudUtils/>
+
+        </sequential>
+
+    </macrodef>
+
+    <macrodef name="commercedbsync_before_production">
+
+        <sequential>
+
+            <patchAzureCloudUtils/>
+
+        </sequential>
+
+    </macrodef>
+
+    <macrodef name="commercedbsync_before_dist">
+
+        <sequential>
+
+            <patchAzureCloudUtils/>
+
+        </sequential>
+
+    </macrodef>
+
+    <macrodef name="patchAzureCloudUtils">
+
+        <sequential>
+
+            <if>
+
+                <available file="${ext.azurecloud.path}/bin/azurecloudserver.jar"/>
+
+                <then>
+
+                    <echo>PATCHING azurecloudserver.jar to enable configurable fake tenants in AzureCloudUtils</echo>
+
+                    <jar destfile="${ext.azurecloud.path}/bin/azurecloudserver.jar" update="true">
+
+                        <fileset dir="${ext.commercedbsync.path}/classes" includes="**/AzureCloudUtils.class"/>
+
+                    </jar>
+
+                </then>
+
+                <else>
+
+                    <echo>${ext.azurecloud.path}/bin/azurecloudserver.jar doesn't exist. Cannot patch AzureCloudUtils to
+                        enable fake tenants!
+                    </echo>
+
+                </else>
+
+            </if>
+
+        </sequential>
+
+    </macrodef>
+
+</project>
diff --git a/commercedbsync/extensioninfo.xml b/commercedbsync/extensioninfo.xml
@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+
+<!--
+  ~  Copyright: 2022 SAP SE or an SAP affiliate company and commerce-db-synccontributors.
+  ~  License: Apache-2.0
+  ~
+  --><extensioninfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="extensioninfo.xsd">
+
+    <extension abstractclassprefix="Generated" classprefix="Commercedbsync" managername="CommercedbsyncManager" managersuperclass="de.hybris.platform.jalo.extension.Extension" name="commercedbsync" usemaven="true">
+
+        <coremodule generated="true" manager="de.hybris.platform.jalo.extension.GenericManager" packageroot="com.sap.cx.boosters.commercedbsync"/>
+
+
+    </extension>
+
+</extensioninfo>
diff --git a/commercedbsync/external-dependencies.xml b/commercedbsync/external-dependencies.xml
@@ -0,0 +1,52 @@
+<!--
+  ~  Copyright: 2022 SAP SE or an SAP affiliate company and commerce-db-synccontributors.
+  ~  License: Apache-2.0
+  ~
+  -->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <groupId>de.hybris.platform</groupId>
+    <artifactId>commercedbsync</artifactId>
+    <version>6.7.0.0-RC19</version>
+
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>com.google.code.gson</groupId>
+            <artifactId>gson</artifactId>
+            <version>2.8.6</version>
+        </dependency>
+        <dependency>
+            <groupId>com.google.guava</groupId>
+            <artifactId>guava</artifactId>
+            <version>28.0-jre</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.commons</groupId>
+            <artifactId>commons-dbcp2</artifactId>
+            <version>2.7.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.microsoft.azure</groupId>
+            <artifactId>azure-storage</artifactId>
+            <version>8.1.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.zaxxer</groupId>
+            <artifactId>HikariCP</artifactId>
+            <version>3.4.5</version>
+        </dependency>
+        <dependency>
+            <groupId>com.github.freva</groupId>
+            <artifactId>ascii-table</artifactId>
+            <version>1.1.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.fasterxml.jackson.datatype</groupId>
+            <artifactId>jackson-datatype-jsr310</artifactId>
+            <version>2.13.3</version>
+        </dependency>
+    </dependencies>
+</project>
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		These Project development objects are not managed or delivered or intended for future inclusion as a standard component of the SAP Software. Therefore, at Project closure, these Project development objects will not include any further support services, defect resolution, maintenance, or upgrades or in any way be within scope of SAP support obligations for licensed SAP Software. Licensee is solely responsible for supporting such objects. SAP does not assure the compatibility of such objects with future releases of SAP Software or other SAP solutions.