Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Database schema optimizations #4731

Open
wants to merge 108 commits into
base: main
Choose a base branch
from
Open

Database schema optimizations #4731

wants to merge 108 commits into from
+12,911 −594

Conversation

SergeyGaluzo
Copy link
Contributor

@SergeyGaluzo SergeyGaluzo commented Nov 19, 2024 •
edited
Loading

Optimizations:

  1. History/current resources split with removal of not needed indexes.
  2. Raw resources in separate table.
  3. Resource id integer mapping.
  4. Enabled writes/reads of raw resources to/from ADLS.
  5. Extended string reference id length.

Deployment is dependent on small schema change creating CurrentResource view. #4755

https://microsofthealth.visualstudio.com/Health/_workitems/edit/135369

SergeyGaluzo and others added 30 commits September 27, 2024 08:42
@SergeyGaluzo
Raw resources in the Lake and Resource Id integer map
47c370f
@SergeyGaluzo
perf tweeks
5aca029
@SergeyGaluzo
Merge history separation with separated raw resources (#4685)
de5a156 
* History separation V0

* Forgotten resource

* replace current by view

* TRUNCATE

* HardDelete

* delete history v0

* missed 64 and removed dead code from SQL query generator

* Removed history from current in merge

* Triggers

* tests and tran

* disable/enable indexes

* rollback hard delete and merge

* rollback delete invisible history

* TRUNCATE -> DELETE

* line

* leftovers

* PK check

* Cosmetic

* WHERE nanes

* Get resources with forced indexes

* Removed redundant where

* Adding feature flag for raw resource dedupping

* Enable invisible history by default

* parameters

* commit

* fixed typo

* 65 + more realistic diff

* adjusted update trigger

* right trigger

* next iteration

* special chars

* Added delete and update

* \r

* Adding script runner

* Added verification

* cosmetic

* HOLDLOCK hint

* removed incorrect comments

* RT

* lock timeout

* 180 and remove empty

* no changes in update resource search params

* blob rewriter tool

* Revert "no changes in update resource search params"

This reverts commit b2e0c38.

* comment

* Generic disable indexes and update search params

* Dummy resources

* fixes

* Dummy records based on surr id

* adjust test to filter dummy rows

* exclude history and current

* Adding PerfTest V-1

* Get asyn wrapper without type string to id function

* tool

* not exists on data copy

* packages back

* Added comments.

* Correct filtered index on resource current

* Added redundant IsHistory=0 to index

* deduping

* Removed dummy records

* pp-p

* history clause

* deduping

* testing parameters

* get resource by type and surr id range with many versions

* skip large databases

* Added calls to fhir

* added conflict

* max retries = 3

* database pings

* examples

* reverse

* start closer

* put logic

* Create

* Move 65 to 84

* Fixes after merge

* Adding RawResources table

* change capture tweeks

* Test fix

* Added MI

* inline insert

* Repeat on updates
@SergeyGaluzo
Merge resource id map
e37ffd4
@SergeyGaluzo
Fix after merge
5d8d03a
@SergeyGaluzo
Fix after merge
cc94008
@SergeyGaluzo
adjustments
d6ce380
@SergeyGaluzo
tests
2950bcf
@SergeyGaluzo
Merge from main
8e74731
@SergeyGaluzo
Merge fixes
c3ccf69
@SergeyGaluzo
Ignore NULL reference resource tyoe
b0ed1eb
@SergeyGaluzo
test fix
51a942d
@SergeyGaluzo
default number of columns = 13
38e869b
@SergeyGaluzo
commented 84 diff
bb535af
@SergeyGaluzo
set end date on enqueue
7060321
@SergeyGaluzo
rename secondary to adls
427cdfa
@SergeyGaluzo
defrag formatting
02a545f
@SergeyGaluzo
set timeout to 0 for getting frag
fbbaf20
@SergeyGaluzo
isnull on input param
3f5dcd9
@SergeyGaluzo
Merge branch 'main' into users/sergal/rawmaplakeandhistoryandresource…
e6e3fe9 
…idmap
@SergeyGaluzo
after merge from main
5569ea1
@SergeyGaluzo
Merge branch 'users/sergal/rawmaplakeandhistoryandresourceidmap' into…
2648ecd 
… users/sergal/newschemarefsep
@SergeyGaluzo
Reference separation
037061d
@SergeyGaluzo
file id
99c6620
@SergeyGaluzo
small things
0b226bf
@SergeyGaluzo
Do not write raw resource
7f286f9
@SergeyGaluzo
Added missing file id in history insert
873583e
@SergeyGaluzo
hard delete tests
ce28891
@SergeyGaluzo
change capture
848195c
@SergeyGaluzo
FK
61b1c06
github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 27, 2024
View reviewed changes
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlStoreClient.cs Fixed Show fixed Hide fixed
tools/Exporter/Program.cs Fixed Show fixed Hide fixed
tools/Exporter/Program.cs Fixed Show fixed Hide fixed
tools/Exporter/Program.cs Fixed Show fixed Hide fixed
tools/Exporter/Program.cs Fixed Show fixed Hide fixed
tools/Exporter/Program.cs Fixed Show fixed Hide fixed
github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 28, 2024
View reviewed changes
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Search/SqlServerSearchService.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlStoreClient.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlStoreClient.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlStoreClient.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlStoreClient.cs Fixed Show fixed Hide fixed
src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlStoreClient.cs Fixed Show fixed Hide fixed
github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 28, 2024
View reviewed changes
tools/Exporter/Program.cs Dismissed Show dismissed Hide dismissed
github-advanced-security[bot]
github-advanced-security bot found potential problems Jan 3, 2025
View reviewed changes
tools/PerfTester/Program.cs Dismissed Show dismissed Hide dismissed
tools/PerfTester/Program.cs Dismissed Show dismissed Hide dismissed
tools/PerfTester/Program.cs Dismissed Show dismissed Hide dismissed
tools/PerfTester/Program.cs Dismissed Show dismissed Hide dismissed
tools/PerfTester/Program.cs Dismissed Show dismissed Hide dismissed
tools/PerfTester/Program.cs Dismissed Show dismissed Hide dismissed
@microsoft microsoft deleted a comment from azure-pipelines bot Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Azure Healthcare APIs Label denotes that the issue or PR is relevant to the FHIR service in the Azure Healthcare APIs Enhancement Enhancement on existing functionality. Schema Version backward compatible
Projects
None yet
Milestone
S156
Development

Successfully merging this pull request may close these issues.
1 participant
@SergeyGaluzo