Create `water` schema views #551

Jozzey · 2023-11-27T16:40:37Z

https://eaflood.atlassian.net/browse/WATER-4188

As part of the work we have been doing on two-part tariff we are going to be creating all our new tables in a single schema public.

We have also decided that when there is a legacy table that we are still going to need for SROC. We are going to create a View of that table in the public schema and correct any issues with naming conventions, unused fields etc in the view. The first of these views that represents the old returns tables have been created in #531

In this PR we will build the migration scripts to create the Views we need for SROC in the water schema.

https://eaflood.atlassian.net/browse/WATER-4188 As part of the work we have been doing on two-part tariff we are going to be creating all our new tables in a single schema `public`. We have also decided that when there is a legacy table that we are still going to need for SROC. We are going to create a View of that table in the `public` schema and correct any issues with naming conventions, unused fields etc in the view. The first of these views that represents the old `returns` tables have been created in #531 In this PR we will build the migration scripts to create the Views we need for SROC in the `water` schema.

Cruikshanks

Think there are 1 couple of corrections in here, some suggestions for alternate names.

The key change is dropping the convention of prefixing boolean fields with is_, especially as going through this, I have seen they haven't followed the convention of applying it consistently.

db/migrations/public/20231127193216_create-bills-view.js

db/migrations/public/20231127195654_create-bill-runs-view.js

db/migrations/public/20231127202549_create-bill-run-volumes-view.js

db/migrations/public/20231127204818_create-change-reasons-view.js

db/migrations/public/20231127222209_create-licences-view.js

db/migrations/public/20231127222742_create-purposes-view.js

db/migrations/public/20231127224323_create-transactions-view.js

db/migrations/public/20231127230416_create-workflows-view.js

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Cruikshanks

Sorry Jason. This is all great. It's just I hadn't spotted that the GitHub suggestion was adding a new line to everything. Accepting them has added lots of blank lines in the middle of your migrations.

If we can just whip them out I'll jump on and approve.

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff we are going to be creating all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities that are currently sat in different schemas. The first example of this approach was done in PR #531 . This change adds the models and helpers for the views in the `water` schema that were created in a previous PR #551 > The final step will then be to refactor the existing code to use the new models and delete the old legacy ones.

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff we are going to be creating all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities that are currently sat in different schemas. The first example of this approach was done in PR #531 . This change adds the models and helpers for the views in the `water` schema that were created in PR #551 > The final step will then be to refactor the existing code to use the new models and delete the old legacy ones.

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff, we will be creating all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities currently sat in different schemas. The first example of this approach was done in PR #531 . We created the new views in [Create water schema views](#551). Then in [Create water schema models & helpers](#560) we added the new models and helpers that use them. This is the final step in the process, we are refactoring any use of the legacy models to use the new ones and deleting all the legacy-based code.

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff, we will be creating all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities currently sat in different schemas. The first example of this approach was done in PR #531 . We created the new views in [Create WATER schema views](#551). Then in [Create WATER schema models & helpers](#560) we added the new models and helpers that use them. This is the final step in the process, we are refactoring any use of the legacy models to use the new ones and deleting all the legacy-based code.

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff, we will be creating all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities currently sat in different schemas. The first example of this approach was done in PR #531 . In [Create water schema views](#551) we created the view for `water.billing_batches` and dropped a column we thought wasn't needed and renamed one we thought was wrong. We actually use `errorCode:` in our own codebase during supplementary billing if a bill run fails so it is needed. And `invoiceCount` actually refers to the number of debit (positive value) bills in a bill run, not the total number of bills. 'Invoice' rather than 'Debit' is the term used throughout the UI. So, we need to revert the name change. > Note - when it comes to views you have to drop and recreate them. Unlike tables you cannot alter columns individually diff --git a/db/migrations/public/20231215174532_alter-bill-runs-view.js b/db/migrations/public/20231215174532_alter-bill-runs-view.js new file mode 100644 index 0000000..af07d8e --- /dev/null +++ b/db/migrations/public/20231215174532_alter-bill-runs-view.js @@ -0,0 +1,68 @@ +'use strict' + +const viewName = 'bill_runs' + +exports.up = function (knex) { + return knex + .schema + .dropView(viewName) + .createView(viewName, (view) => { + view.as(knex('billing_batches').withSchema('water').select([ + 'billing_batch_id AS id', + 'region_id', + 'batch_type', + 'from_financial_year_ending', + 'to_financial_year_ending', + 'status', + 'invoice_count', + 'credit_note_count', + 'net_total', + 'bill_run_number', + 'error_code', + 'external_id', + 'is_summer AS summer', + 'source', + 'legacy_id', + 'metadata', + 'invoice_value', + 'credit_note_value', + 'transaction_file_reference', + 'scheme', + 'date_created AS created_at', + 'date_updated AS updated_at' + ])) + }) +} + +exports.down = function (knex) { + return knex + .schema + .dropView(viewName) + .createView(viewName, (view) => { + // NOTE: We have commented out unused columns from the source table + view.as(knex('billing_batches').withSchema('water').select([ + 'billing_batch_id AS id', + 'region_id', + 'batch_type', + 'from_financial_year_ending', + 'to_financial_year_ending', + 'status', + 'invoice_count AS bill_count', + 'credit_note_count', + 'net_total', + 'bill_run_number', + // 'error_code', + 'external_id', + 'is_summer AS summer', + 'source', + 'legacy_id', + 'metadata', + 'invoice_value', + 'credit_note_value', + 'transaction_file_reference', + 'scheme', + 'date_created AS created_at', + 'date_updated AS updated_at' + ])) + }) +}

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff, we will create all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities currently sat in different schemas. The first example of this approach was done in PR #531 . In [Create water schema views](#551) we created the view for `water.billing_batches` and dropped a column we thought wasn't needed and renamed one we thought was wrong. We use `errorCode:` in our codebase during supplementary billing if a bill run fails so it is needed. And `invoiceCount` refers to the number of debit (positive value) bills in a bill run, not the total number. 'Invoice' rather than 'Debit' is used throughout the UI. So, we need to revert the name change. > Note - when it comes to views you have to drop and recreate them. Unlike tables, you cannot alter columns individually

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff, we will be creating all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities currently sat in different schemas. The first example of this approach was done in PR #531 . We created the new views in [Create WATER schema views](#551). Then in [Create WATER schema models & helpers](#560) we added the new models and helpers that use them. This is the final step in the process, we are refactoring any use of the legacy models to use the new ones and deleting all the legacy-based code.

https://eaflood.atlassian.net/browse/WATER-4057 As part of the work we have been doing on two-part tariff, we will create all our new tables in the default `public` schema. We have also decided that when there is a legacy table that we are still going to need we will create a [View](https://www.postgresql.org/docs/current/sql-createview.html) of it in the `public` schema. This allows us to correct any issues with naming conventions, strip out unused fields, and join entities currently sat in different schemas. The first example of this approach was done in PR #531 . We created the new views in [Create WATER schema views](#551). Then in [Create WATER schema models & helpers](#560) we added the new models and helpers that use them. This is the final step in the process, we are refactoring any use of the legacy models to use the new ones and deleting all the legacy-based code.

https://eaflood.atlassian.net/browse/WATER-4365 > For context this came out of us working on re-implementing the SROC annual bill run using what we've learnt and components from our supplementary billing engine. As part of looking at re-implementing the SROC annual billing engine in this project our spike (WATER-4348 ) confirmed we could simplify the implementation and improve performance if we extracted the billing account details at the same time as the charge versions to be billed. When we implemented the SROC supplementary billing engine we were directly working with the legacy schemas and tables. This meant we couldn't create relationships in models that sat in different schemas. But thanks to [Create water schema views](#551) and [Create crm_v2 schema views](#556) they are hidden away from us. We can now work as if the legacy schemas are a single schema. So, this change updates the `ChargeVersionModel` and `BillingAccountModel` to add a relationship between them which we can then exploit with our [Objection.js queries](https://vincit.github.io/objection.js/).

Jozzey added the enhancement New feature or request label Nov 27, 2023

Jozzey self-assigned this Nov 27, 2023

Jozzey and others added 11 commits November 27, 2023 23:08

Create migrations

43aa0ab

Update views to re-order columns

fdbea79

Tidy up migrations

3b5c548

Remove unused columns

17c5f9d

Fix legacy migration and remove ones not required

03d16a5

Fix legacy migration and remove ones not required

e4fd1b0

Fix legacy migration and remove ones not required

2696e14

Fix legacy migration and remove one

19f695c

Fix charge version helper

1e4fba5

Fix migration

1f0673f

Merge branch 'main' into create-water-schema-views

ed6249b

Jozzey marked this pull request as ready for review November 28, 2023 15:06

Jozzey requested review from Cruikshanks and Beckyrose200 November 28, 2023 15:06

Jozzey marked this pull request as draft November 28, 2023 15:17

Rename some foreign keys

ca6584a

Jozzey marked this pull request as ready for review November 28, 2023 15:30

Cruikshanks requested changes Nov 28, 2023

View reviewed changes

Jozzey and others added 9 commits November 29, 2023 09:31

Rename foreign key

5d19e00

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

4891f71

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

a2c71cb

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

755545d

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

dadffd6

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

d4dbda0

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

949bd7b

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

f39e54e

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Rename "include in billing" flags

a216039

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Jozzey and others added 6 commits November 29, 2023 09:40

Remove is from boolean col names

abc9024

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Rename date_deleted => deleted_at

79c12e8

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Fix foreign key name

f49b466

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Remove is from boolean col names

4aa5cb4

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Rename billed_upto_date => billed_up_to_date

551bd24

Co-authored-by: Alan Cruikshanks <alan.cruikshanks@gmail.com>

Merge branch 'main' into create-water-schema-views

dd64139

Jozzey requested a review from Cruikshanks November 29, 2023 09:49

Cruikshanks requested changes Nov 29, 2023

View reviewed changes

Remove whitespace from migrations

f1401d4

Jozzey requested a review from Cruikshanks November 29, 2023 18:03

Cruikshanks approved these changes Nov 30, 2023

View reviewed changes

Merge branch 'main' into create-water-schema-views

a4d466a

Jozzey merged commit 87e06e7 into main Nov 30, 2023
6 checks passed

Jozzey deleted the create-water-schema-views branch November 30, 2023 14:42

Jozzey mentioned this pull request Dec 1, 2023

Create water schema models & helpers #560

Merged

Cruikshanks mentioned this pull request Dec 5, 2023

Refactor to use new view based WATER models #569

Merged

Cruikshanks mentioned this pull request Dec 15, 2023

Fix missing fields in bill runs view #599

Merged

Cruikshanks mentioned this pull request Feb 20, 2024

Add relationship - billing acc. <-> chg. version #750

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create `water` schema views #551

Create `water` schema views #551

Jozzey commented Nov 27, 2023

Cruikshanks left a comment

Cruikshanks left a comment

Create water schema views #551

Create water schema views #551

Conversation

Jozzey commented Nov 27, 2023

Cruikshanks left a comment

Choose a reason for hiding this comment

Cruikshanks left a comment

Choose a reason for hiding this comment

Create `water` schema views #551

Create `water` schema views #551