Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ACQ - Updated metadb derived tables #804

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 64 additions & 13 deletions sql_metadb/derived_tables/feesfines_accounts_actions.sql
Original file line number Diff line number Diff line change
@@ -1,44 +1,95 @@
--metadb:table feesfines_accounts_actions

-- Create a derived table that takes feesfines_accounts as the main table
-- join all transaction data from the feesfines_actions table
-- add patron group information from user_group table
--Creates a derived table that takes folio_feesfines.accounts as the main table
Comment on lines 1 to +2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you would, in both feesfines_accounts_actions.sql and po_notes.sql, please add a blank line after the --metadb:table directive (see https://metadb.dev/doc/#_external_sql_directives) and a blank line in between individual SQL statements below.

--Payments and credits are shown in as a negative balance for accounting purposes

DROP TABLE IF EXISTS feesfines_accounts_actions;
nassibnassar marked this conversation as resolved.
Show resolved Hide resolved

CREATE TABLE feesfines_accounts_actions AS
SELECT
fa.id AS fine_account_id,
jsonb_extract_path_text(fa.jsonb, 'amount')::numeric(12,2) AS fine_account_amount,
jsonb_extract_path_text(fa.jsonb, 'dateCreated')::timestamptz AS fine_date,
jsonb_extract_path_text(fa.jsonb, 'dateUpdated')::timestamptz AS fine_updated_date,
fa.creation_date::timestamptz AS fine_date,
jsonb_extract_path_text(fa.jsonb, 'metadata','updatedDate')::timestamptz AS fine_updated_date,
jsonb_extract_path_text(fa.jsonb, 'feeFineId')::uuid AS fee_fine_id,
jsonb_extract_path_text(fa.jsonb, 'ownerId')::uuid AS owner_id,
jsonb_extract_path_text(fa.jsonb, 'feeFineOwner') AS fee_fine_owner,
jsonb_extract_path_text(fa.jsonb, 'feeFineType') AS fee_fine_type,
jsonb_extract_path_text(fa.jsonb, 'materialTypeId')::uuid AS material_type_id,
jsonb_extract_path_text(fa.jsonb, 'materialType') AS material_type,
jsonb_extract_path_text(fa.jsonb, 'payment_status') AS payment_status,
jsonb_extract_path_text(fa.jsonb, 'paymentStatus', 'name') AS payment_status,
jsonb_extract_path_text(fa.jsonb, 'status', 'name') AS fine_status, -- open or closed
jsonb_extract_path_text(fa.jsonb, 'userId')::uuid AS account_user_id,
ff.id AS transaction_id,
jsonb_extract_path_text(ff.jsonb, 'accountId')::uuid AS account_id,
jsonb_extract_path_text(ff.jsonb, 'amountAction')::numeric(12,2) AS transaction_amount,
jsonb_extract_path_text(ff.jsonb, 'balance')::numeric(12,2) AS account_balance,
jsonb_extract_path_text(ff.jsonb, 'typeAction') AS type_action,
jsonb_extract_path_text(ff.jsonb, 'dateAction')::timestamptz AS transaction_date,
jsonb_extract_path_text(ff.jsonb, 'createdAt') AS transaction_location,
jsonb_extract_path_text(ff.jsonb, 'transactionInformation') AS transaction_information,
jsonb_extract_path_text(ff.jsonb, 'source') AS operator_id,
jsonb_extract_path_text(ff.jsonb, 'source') AS action_created_by,
jsonb_extract_path_text(ff.jsonb, 'paymentMethod') AS payment_method,
uu.id AS user_id,
uu.patron_group AS user_patron_group_id,
ug.id AS patron_group_id,
ug.group AS patron_group_name
ug.group AS patron_group_name,
CASE
WHEN jsonb_extract_path_text(ff.jsonb, 'typeAction') IN ('Paid partially','Paid fully','Waived partially','Waived fully','Credited partially','Credited fully') THEN jsonb_extract_path_text(ff.jsonb, 'amountAction')::numeric(12,2) * -1
ELSE jsonb_extract_path_text(ff.jsonb, 'amountAction')::numeric(12,2)
END AS signed_transaction_amount
FROM
folio_feesfines.accounts AS fa
LEFT JOIN folio_feesfines.feefineactions AS ff ON fa.id = jsonb_extract_path_text(ff.jsonb, 'accountId')::uuid
LEFT JOIN folio_users.users__t AS uu ON jsonb_extract_path_text(fa.jsonb, 'userId')::uuid = uu.id
LEFT JOIN folio_users.groups__t AS ug ON uu.patron_group = ug.id
ORDER BY fine_account_id, transaction_date;
ORDER BY fine_account_id, transaction_date
;

COMMENT ON COLUMN feesfines_accounts_actions.fine_account_id IS 'User fine/fee account id, UUID';

COMMENT ON COLUMN feesfines_accounts_actions.fine_account_amount IS 'Amount of the fine/fee';

COMMENT ON COLUMN feesfines_accounts_actions.fine_date IS 'Date and time the account of the fine/fee was created';

COMMENT ON COLUMN feesfines_accounts_actions.fine_updated_date IS 'Date and time the account of the fine/fee was updated';

COMMENT ON COLUMN feesfines_accounts_actions.fee_fine_id IS 'ID of the fee/fine';

COMMENT ON COLUMN feesfines_accounts_actions.owner_id IS 'ID of the account owner';

COMMENT ON COLUMN feesfines_accounts_actions.fee_fine_owner IS 'Owner of the account';

COMMENT ON COLUMN feesfines_accounts_actions.fee_fine_type IS 'Fee/fine that is up to the desecration of the user';

COMMENT ON COLUMN feesfines_accounts_actions.material_type_id IS 'ID of the material type of the item';

COMMENT ON COLUMN feesfines_accounts_actions.material_type IS 'Material type of the item';

COMMENT ON COLUMN feesfines_accounts_actions.payment_status IS 'Overall status of the payment/waive/transfer/refund/cancel';

COMMENT ON COLUMN feesfines_accounts_actions.fine_status IS 'Overall status of the fee/fine';

COMMENT ON COLUMN feesfines_accounts_actions.account_user_id IS 'ID of the user';

COMMENT ON COLUMN feesfines_accounts_actions.transaction_id IS 'Fine/fee action id, UUID';

COMMENT ON COLUMN feesfines_accounts_actions.transaction_amount IS 'Amount of activity';

COMMENT ON COLUMN feesfines_accounts_actions.account_balance IS 'Calculated amount of remaining balance based on original fee/fine and what has been paid/waived/transferred/refunded';

COMMENT ON COLUMN feesfines_accounts_actions.type_action IS 'Type of activity including the type of transaction';

COMMENT ON COLUMN feesfines_accounts_actions.transaction_date IS 'Date and time the transaction of the fine/fee was created';

COMMENT ON COLUMN feesfines_accounts_actions.transaction_location IS 'The service point where the action was created';

COMMENT ON COLUMN feesfines_accounts_actions.transaction_information IS 'Number or other transaction id related to payment';

COMMENT ON COLUMN feesfines_accounts_actions.action_created_by IS 'The source of the fee action';

COMMENT ON COLUMN feesfines_accounts_actions.payment_method IS 'Overall status of the action-setting';

COMMENT ON COLUMN feesfines_accounts_actions.user_id IS 'User UUID';

COMMENT ON COLUMN feesfines_accounts_actions.user_patron_group_id IS 'UUID for user patron group';

COMMENT ON COLUMN feesfines_accounts_actions.patron_group_name IS 'User patron group';

COMMENT ON COLUMN feesfines_accounts_actions.signed_transaction_amount IS 'Identifies the type_action value that decreases the balance of the account and adds a - to those values';
33 changes: 33 additions & 0 deletions sql_metadb/derived_tables/po_notes.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
--metadb:table po_notes
--Creates a derived table to show all notes attached to purchase orders, including general notes, renewal notes, and notes when purchase order is closed.

DROP TABLE IF EXISTS po_notes;
CREATE TABLE po_notes AS
SELECT
po.id AS po_id,
Comment on lines +6 to +7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you would, please consider using indentation to make the query easier to read. For examples, see feesfines_accounts_actions.sql above or the query within: https://github.com/metadb-project/metadb-examples/blob/main/folio/reports/missing_items.sql

(po.jsonb -> 'metadata' ->> 'createdDate')::timestamptz AS created_date,
po.jsonb ->> 'poNumber' AS po_number,
po.jsonb ->> 'workflowStatus' AS po_workflow_status,
Comment on lines +8 to +10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use jsonb_extract_path_text() for JSON extraction (here and below).

po_notes.jsonb #>> '{}' AS po_note,
po_notes.ORDINALITY AS po_note_ordinality,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

po.jsonb -> 'closeReason' ->> 'note' AS po_close_reason_note,
po.jsonb -> 'ongoing' ->> 'renewalNote' AS po_renewal_note
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not find renewalNote in any databases or under "ongoing" in the JSON schema here: https://s3.amazonaws.com/foliodocs/api/mod-orders-storage/p/purchase-order.html

Did I look in the wrong place? Are you sure this is correct?

Copy link
Contributor

@natalyapik natalyapik Dec 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is on po_line table: https://s3.amazonaws.com/foliodocs/api/mod-orders-storage/p/po-line.html#orders_storage_po_lines__id__get
and it is extracted in the transformed table.@arthur-aguilera please delete "renewal note" from this derived table.

FROM folio_orders.purchase_order AS po
CROSS JOIN LATERAL jsonb_array_elements(jsonb_extract_path(po.jsonb, 'notes')) WITH ORDINALITY AS po_notes (jsonb)
;

COMMENT ON COLUMN po_notes.po_id IS 'UUID of purchase order';

COMMENT ON COLUMN po_notes.created_date IS 'Purchase Order creation date and time';

COMMENT ON COLUMN po_notes.po_number IS 'Purchase order number';

COMMENT ON COLUMN po_notes.po_workflow_status IS 'workflow status of the purchase order';

COMMENT ON COLUMN po_notes.po_note IS 'Purchase order note';

COMMENT ON COLUMN po_notes.po_note_ordinality IS 'The ordinality of the note';

COMMENT ON COLUMN po_notes.po_close_reason_note IS 'Notes entered when a purchase order is closed';

COMMENT ON COLUMN po_notes.po_renewal_note IS 'Notes entered in the ongoing order information';
1 change: 1 addition & 0 deletions sql_metadb/derived_tables/runlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ po_lines_eresource.sql
po_lines_locations.sql
po_lines_phys_mat_type.sql
po_lines_physical.sql
po_notes.sql
po_ongoing.sql
po_prod_ids.sql
po_organization.sql
Expand Down