Migrate to bcp for bulk insert performance #6

hsyyid · 2025-01-15T21:32:40Z

No description provided.

…emove them from values and parameters

This reverts commit 90f7103.

This reverts commit 9540822.

… arounds the bug that caused fast_executemany to break.

…end.

… table and table exisits commands.

…ound it.

…are brackets.

… target table. ii) Ensure CSV is exported in default db column order.

…ema. Fixed itgit status

arilton and others added 30 commits November 19, 2024 11:38

add all properties inside brackets

9dfe892

cannot set key_properties

b901f74

only applying brackets in reserved words in table and column names. r…

b02c7e9

…emove them from values and parameters

only bracket names in insert/merge statements

83799f2

force drop table for testing

10cb388

actually force drop the table

a41f9c4

clean the code and improve against sql injection

90f7103

fix drop logic

09af618

Revert "clean the code and improve against sql injection"

6a9b761

This reverts commit 90f7103.

turn off force drop logic

9540822

alter varchar columns to max size before insert data

c886bc8

add BEGIN TRANSACTION

30260f7

execute the merge inside a transaction

bf9bb8b

check if current_type.length is None at merge_sql_types

068e3ca

Revert "turn off force drop logic"

49b0137

This reverts commit 9540822.

change drop table

8293ac5

fix

06e5318

remove drop tables logic

91e74eb

Force TDS version

0ad21fc

use pyodbc

c352f06

fix

0cf60e4

fix

d299a59

fix issue with passwords with special chars

7d6a7f0

fix issues with schemas

75415de

fix schema

a873586

Merge branch 'hgi-6830' into feature/hgi-7043

432d0bb

add default constraint copy

b858e9f

bug fix

6e752cd

turn off alter varchar logic

12f489a

update logic

ecf3f43

izkapadia-hg and others added 30 commits January 15, 2025 02:06

HGI-7050 Create physical table instead of temporary table, this works…

b57f923

… arounds the bug that caused fast_executemany to break.

HGI-7050 Create physical table instead of temporary table, this works…

8797f6b

… arounds the bug that caused fast_executemany to break.

HGI-7050 int field are now created is big int, drop tmp table at the …

f017f60

…end.

HGI-7050 Drop actual table before fisrt batch.

e771016

HGI-7050 using the engine that explictly create to run connector drop…

9331a53

… table and table exisits commands.

fix

ef1a8c7

Switch to bcp

2275da1

handle int pks

9116bf6

fix bcp path

ed4763f

fix drop

cb66637

fix

ab01083

handle nvarchar

5ff35e8

fix

5f08005

fix

8213af2

add jenkinsfile

466fde4

add log

c2be646

fix table ref

351db4e

fix

3beb506

turn off truncate

9e4c232

add trunc back

f415c6e

escape drop

0ad1c82

recreate connection every hour to ensure freshness

4d78eca

HGI-7077 case is the internal keywork, need to put square brackets ar…

46b0579

…ound it.

HGI-7077 split schema and table name, and enclose table name with squ…

6e1f642

…are brackets.

HGI-7077 disable main table drop and recreate.

dc4f6cb

fix missing dot

f31f0f4

HGI-7077 i) If new columns are selected in singer schema, add them to…

605ba62

… target table. ii) Ensure CSV is exported in default db column order.

HGI-7077 i) Field name in tables are differet from field names in sch…

2e33ebb

…ema. Fixed itgit status

HGI-7101 added a metadata.json.

97e5e0b

fix import

afa364b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate to bcp for bulk insert performance #6

Migrate to bcp for bulk insert performance #6

hsyyid commented Jan 15, 2025

Migrate to bcp for bulk insert performance #6

Are you sure you want to change the base?

Migrate to bcp for bulk insert performance #6

Conversation

hsyyid commented Jan 15, 2025