Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#12048] Update liquibase configuration #12930

Merged
merged 23 commits into from
Apr 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
21cf9c8
Update gradle config
NicolasCwy Mar 23, 2024
1f64184
Update liquibase config for v9
NicolasCwy Mar 23, 2024
8151bde
Turn off table generate for prod
NicolasCwy Mar 24, 2024
4765e3e
Add back wrongly deleted properties
NicolasCwy Mar 24, 2024
862821f
Merge branch 'master' into liquibase-configuration
NicolasCwy Mar 24, 2024
5a5ffc8
Update of changelog file
NicolasCwy Mar 24, 2024
217e9cf
Fix lint
NicolasCwy Mar 24, 2024
08bc71d
Add configuration for generating changelog
NicolasCwy Mar 26, 2024
5a76894
Add schema migration docs
NicolasCwy Mar 26, 2024
4b86ed4
Merge branch 'master' into liquibase-configuration
NicolasCwy Mar 26, 2024
4c8ac0d
Update process.md
NicolasCwy Mar 26, 2024
951f9a3
Temporary fix to build gradle
NicolasCwy Mar 26, 2024
f5067f3
Add fix to choose liquibase activity
NicolasCwy Mar 26, 2024
14f79f3
Amend documentation for schema migration
NicolasCwy Mar 27, 2024
bd665ad
Merge branch 'master' into liquibase-configuration
NicolasCwy Mar 27, 2024
21d0db5
Add connection instructions to documentation
NicolasCwy Mar 27, 2024
a31ff40
Merge branch 'master' into liquibase-configuration
NicolasCwy Apr 1, 2024
628f274
Revert naming changes and update schema process
NicolasCwy Apr 1, 2024
d952f33
Remove PR naming requirement
NicolasCwy Apr 1, 2024
60946f9
Fix changelog renaming
NicolasCwy Apr 1, 2024
2672b43
Update schema migration to only be done by release leader
NicolasCwy Apr 4, 2024
736670a
Merge branch 'master' into liquibase-configuration
NicolasCwy Apr 4, 2024
78a15df
Add section on gradle activities
NicolasCwy Apr 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,9 @@ dependencies {
exclude group: "org.apache.jmeter", module: "bom"
}

liquibaseRuntime("org.liquibase:liquibase-core:4.19.0")
liquibaseRuntime("info.picocli:picocli:4.7.1")
liquibaseRuntime("org.postgresql:postgresql:42.7.2")
liquibaseRuntime(sourceSets.main.output)
}

Expand Down Expand Up @@ -136,6 +138,10 @@ sourceSets {
}
}

if (!project.hasProperty("runList")) {
project.ext.set("runList", "main")
}

liquibase {
activities {
main {
Expand All @@ -145,7 +151,23 @@ liquibase {
username project.properties['liquibaseUsername']
password project.properties['liquibasePassword']
}
snapshot {
url project.properties['liquibaseDbUrl']
username project.properties['liquibaseUsername']
password project.properties['liquibasePassword']
snapshotFormat "json"
outputFile "liquibase-snapshot.json"
}
diffMain {
searchPath "${projectDir}"
changeLogFile "src/main/resources/db/changelog/db.changelog-new.xml"
referenceUrl project.properties['liquibaseDbUrl']
referenceUsername project.properties['liquibaseUsername']
referencePassword project.properties['liquibasePassword']
url "offline:postgresql?snapshot=liquibase-snapshot.json"
}
}
runList = project.ext.runList
}

tasks.withType(cz.habarta.typescript.generator.gradle.GenerateTask) {
Expand Down
34 changes: 34 additions & 0 deletions docs/schema-migration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
<frontmatter>
title: "Schema Migration"
</frontmatter>

# SQL Schema Migration

Teammates uses _[Liquibase]_(https://docs.liquibase.com/start/home.html), a database schema change management solution that enables developers to revise and release database changes to production. The maintainers in charge of releases (Release Leader) will be in charge of generating a _Liquibase_ changelog prior to each release to keep the production databases schema in sync with the code. Therefore this section is just for documentation purposes for contributors.

## Liquibase in Teammates
_Liquibase_ is made available using the [gradle plugin](https://github.com/liquibase/liquibase-gradle-plugin), providing _liquibase_ functions as tasks. Try `gradle tasks | grep "liquibase"` to see all the tasks available. In teammates, change logs (more in the next section) are written in _XML_.

### Liquibase connection
Amend the `liquibaseDbUrl`, `liquibaseUsername` and `liquibasePassword` in `gradle.properties` to allow the _Liquibase_ plugin to connect your database.

## Change logs, change sets and change types
A _change log_ is a file that contains a series of _change sets_ (analagous to a transaction) which applies _change types_ (actions). You can refer to this page on liquibase on the types of [change types](https://docs.liquibase.com/change-types/home.html) that can be used.

## Gradle Activities for Liquibase
Activities in Gradle are a way of specifying different variables provided by gradle to the Liquibase plugin. The argument `runList` provided by `-pRunList=<activity_name>` e.g `./gradlew liquibaseSnapshot -PrunList=snapshot` is used to specify which activity to be used for the Liquibase command. In this case the `liquibaseSnapshot` command is run using the `snapshot` activity.

Here is a brief description of the activities defined for Liquibase
1. Main: The default activity used by Liquibase commands and is used for running changelogs against a database. This is used by default if a `runList` is not defined
2. Snapshot: Used to specify output format and name for snapshots i.e JSON
3. diffMain: Specify the reference and the target database to generate changelog that contains operations to update reference database to the state of the target database. i.e the reference is the JSON file generated by the snapshot command, this can be replaced with a live database which is used as reference.

## Generating/ Updating liquibase change logs
1. Ensure `diff-main` activity in `build.gradle` is pointing to the latest release changelog `src/main/resources/db/changelog/db.changelog-<release_number>.xml`
2. Delete the `postgres-data` folder to clear any old database schemas
3. Run `git checkout <reference_branch>` and
4. Run the server using `./gradlew serverRun` to generate tables found on branch
5. Generate snapshot of database by running `./gradlew liquibaseSnapshot -PrunList=snapshot`, the snapshot will be output to `liquibase-snapshot.json`
6. Checkout your branch and repeat steps 2 and 4 to generate the tables found on your branch
7. Run `./gradlew liquibaseDiffChangeLog -PrunList=diffMain` to generate changeLog to resolve database schema differences

6 changes: 5 additions & 1 deletion src/main/java/teammates/common/util/HibernateUtil.java
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ public static void buildSessionFactory(String dbUrl, String username, String pas
.setProperty("hibernate.connection.username", username)
.setProperty("hibernate.connection.password", password)
.setProperty("hibernate.connection.url", dbUrl)
.setProperty("hibernate.hbm2ddl.auto", "update")
.setProperty("hibernate.hbm2ddl.auto", "validate")
.setProperty("show_sql", "true")
.setProperty("hibernate.current_session_context_class", "thread")
.setProperty("hibernate.hikari.minimumIdle", "10")
Expand All @@ -130,6 +130,10 @@ public static void buildSessionFactory(String dbUrl, String username, String pas
// .setProperty("hibernate.jdbc.fetch_size", "50")
.addPackage("teammates.storage.sqlentity");

if (Config.IS_DEV_SERVER) {
config.setProperty("hibernate.hbm2ddl.auto", "update");
}

for (Class<? extends BaseEntity> cls : ANNOTATED_CLASSES) {
config = config.addAnnotatedClass(cls);
}
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/db/changelog/db.changelog-root.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd
http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
<include file="src/main/resources/db/changelog/db.changelog-v9.xml" />
<include file="src/main/resources/db/changelog/db.changelog-v9.0.0.xml" />
</databaseChangeLog>
Loading
Loading