Skip to content

Commit

Permalink
Merge pull request #17 from fivetran/feature/netsuite2
Browse files Browse the repository at this point in the history 
Feature/netsuite2
  • Loading branch information
@fivetran-jamie
fivetran-jamie authored Jul 27, 2022
2 parents deae43c + d05dab2 commit abb33e1
Show file tree
Hide file tree
Showing 179 changed files with 3,813 additions and 338 deletions.
12 changes: 11 additions & 1 deletion .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,16 @@ jobs:
pip install -r integration_tests/requirements.txt
mkdir -p ~/.dbt
cp integration_tests/ci/sample.profiles.yml ~/.dbt/profiles.yml
- run:
name: "Run Tests - Postgres"
command: |
. venv/bin/activate
echo `pwd`
cd integration_tests
dbt deps
dbt seed --target postgres --full-refresh
dbt run --target postgres --full-refresh
dbt test --target postgres
- run:
name: "Run Tests - Redshift"
command: |
Expand Down Expand Up @@ -52,7 +62,7 @@ jobs:
echo `pwd`
cd integration_tests
dbt deps
dbt seed --target bigquery
dbt seed --target bigquery --full-refresh
dbt run --target bigquery --full-refresh
dbt test --target bigquery
- save_cache:
Expand Down
1 change: 0 additions & 1 deletion .github/pull_request_template.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
Pull Request
**Are you a current Fivetran customer?**
<!--- Please tell us your name, title and company -->

Expand Down
24 changes: 23 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,27 @@
# dbt_netsuite_source v0.4.2
# dbt_netsuite_source v0.5.0
🎉 [Netsuite2](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/article_163465025391.html) Compatibility 🎉
PRs [#15](https://github.com/fivetran/dbt_netsuite_source/pull/15) and [#17](https://github.com/fivetran/dbt_netsuite_source/pull/17) include the following update to the dbt_netsuite_source package:
## 🚨 Breaking Changes 🚨
- The declaration of passthrough variables within your root `dbt_project.yml` has changed. To allow for more flexibility and better tracking of passthrough columns, you will now want to define passthrough columns in the following format:
> This applies to all passthrough columns within the `dbt_netsuite_source` package and not just the `customers_pass_through_columns` example.
```yml
vars:
customers_pass_through_columns:
- name: "my_field_to_include" # Required: Name of the field within the source.
alias: "field_alias" # Optional: If you wish to alias the field within the staging model.
transform_sql: "cast(field_alias as string)" # Optional: If you wish to define the datatype or apply a light transformation.
```
## Features 🎉
- Addition of the `netsuite_data_model` variable. This variable may either be `netsuite` (the original Netsuite.com connector endpoint) or `netsuite2` (the new Netsuite2 connector endpoint).
- The variable is set to `netsuite` by default. If you wish to run the data models for the Netsuite2 connector, you may simply change the variable within your root dbt_project.yml to `netsuite2`.
- Postgres compatibility!
- Added identifier variables to each Netsuite.com and Netsuite2 source to enable dynamic source table-name adjustments.
- Applied schema level tests to each source table to ensure data validation.
- README updates for easier navigation and package use.

## Contributors
- [@jamesrayoub](https://github.com/jamesrayoub) [#15](https://github.com/fivetran/dbt_netsuite_source/pull/15)
# dbt_netsuite_source v0.4.2
## Features
- Removed the `_fivetran_synced` column definition from the `get_customers_columns` macro, as this is consistent with other macros in the project. Previously, this introduced a bug that caused an ambiguous column name error when passing the `_fivetran_synced` column to the `customers_pass_through_columns` variable since the column is not actually listed in the `stg_netsuite__customers` model field list. Now, when passing the `_fivetran_synced` column to the pass through variable, the ambiguous column error no longer happens. ([#16](https://github.com/fivetran/dbt_netsuite_source/pull/16))

Expand Down
217 changes: 140 additions & 77 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,28 @@
[![Apache License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
# Netsuite

This package models Netsuite data from [Fivetran's connector](https://fivetran.com/docs/applications/netsuite). It uses data in the format described by [this ERD](https://fivetran.com/docs/applications/netsuite-suiteanalytics#schemainformation).

This package enriches your Fivetran data by doing the following:
* Adds descriptions to tables and columns that are synced using Fivetran
* Adds freshness tests to source data
* Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values.
* Models staging tables, which will be used in our transform package

## Requirements
A Fivetran NetSuite Analytics connector, with the following tables synced:
<p align="center">
<a alt="License"
href="https://github.com/fivetran/dbt_netsuite_source/blob/main/LICENSE">
<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
<a alt="dbt-core">
<img src="https://img.shields.io/badge/dbt_Core™_version->=1.0.0_<2.0.0-orange.svg" /></a>
<a alt="Maintained?">
<img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
<a alt="PRs">
<img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a>
</p>

# Netsuite Source dbt Package ([Docs](https://fivetran.github.io/dbt_netsuite_source/))
# 📣 What does this dbt package do?
- Materializes [Netsuite staging tables](https://fivetran.github.io/dbt_netsuite_source/#!/overview/netsuite_source/models/?g_v=1&g_e=seeds) which leverage data in the format described by [this ERD](https://fivetran.com/docs/applications/netsuite#schemainformation). These staging tables clean, test, and prepare your Netsuite data from [Fivetran's connector](https://fivetran.com/docs/applications/netsuite) for analysis by doing the following:
- Name columns for consistency across all packages and for easier analysis
- Adds freshness tests to source data
- Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values.
- Generates a comprehensive data dictionary of your netsuite data through the [dbt docs site](https://fivetran.github.io/dbt_netsuite_source/).
- These tables are designed to work simultaneously with our [Netsuite transformation package](https://github.com/fivetran/dbt_netsuite).

# 🎯 How do I use the dbt package?
## Step 1: Prerequisites
To use this dbt package, you must have At least either one Fivetran **Netsuite** (netsuite.com) or **Netsuite2** (netsuite2) connector syncing the respective tables to your destination:
### Netsuite.com
- accounts
- accounting_periods
- accounting_books
Expand All @@ -29,90 +41,141 @@ A Fivetran NetSuite Analytics connector, with the following tables synced:
- subsidiaries
- vendors
- vendor_types
## Models

This package contains staging models, designed to work simultaneously with our [netsuite transformation package](https://github.com/fivetran/dbt_netsuite). The staging models:
* Remove any rows that are soft-deleted
* Name columns consistently across all packages:
* Boolean fields are prefixed with `is_` or `has_`
* Timestamps are appended with `_at`
* Columns are renamed to fit consistency of other fields. For example, the accounts accountnumber field is renamed to account_number.


## Installation Instructions
Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.

Include in your `packages.yml`

### Netsuite2
- account
- accounttype
- accountingbooksubsidiary
- accountingperiodfiscalcalendar
- accountingperiod
- accountingbook
- consolidatedexchangerate
- currency
- customer
- classification
- department
- entity
- entityaddress
- item
- job
- location
- locationmainaddress
- transactionaccountingline
- transactionline
- transaction
- subsidiary
- vendor
- vendorcategory

### Database Compatibility
This package is compatible with either a **BigQuery**, **Snowflake**, **Redshift**, or **PostgreSQL** destination.
## Step 2: Install the package
Include the following netsuite_source package version in your `packages.yml` file.
> TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
```yaml
packages:
- package: fivetran/netsuite_source
version: [">=0.4.0", "<0.5.0"]
version: [">=0.5.0", "<0.6.0"]
```
## Configuration
By default, this package will run using your target database and the `netsuite` schema. If this is not where your netsuite data is (perhaps your netsuite schema is `netsuite_fivetran`), add the following configuration to your `dbt_project.yml` file:

## Step 3: Define Netsuite.com or Netsuite2 Source
As of April 2022 Fivetran made available a new Netsuite connector which leverages the Netsuite2 endpoint opposed to the original Netsuite.com endpoint. This package is designed to run for either or, not both. By default the `netsuite_data_model` variable for this package is set to the original `netsuite` value which runs the netsuite.com version of the package. If you would like to run the package on Netsuite2 data, you may adjust the `netsuite_data_model` variable to run the `netsuite2` version of the package.
```yml
# dbt_project.yml
...
vars:
netsuite_source:
netsuite_database: your_database_name
netsuite_schema: your_schema_name
netsuite_data_model: netsuite2 #netsuite by default
```
### Passthrough Columns
This package includes all source columns defined in the macros folder. To add additional columns to this package, do so using our pass-through column variables. This is extremely useful if you'd like to include custom fields to the package.

```yml
# dbt_project.yml
## Step 4: Define database and schema variables
By default, this package runs using your destination and the `netsuite` schema. If this is not where your Netsuite data is (for example, if your netsuite schema is named `netsuite_fivetran`), add the following configuration to your root `dbt_project.yml` file:

...
```yml
vars:
netsuite_source:
accounts_pass_through_columns: ['new_custom_field', 'we_can_account_for_that']
classes_pass_through_columns: ['class_is_in_session', 'pass_through_additional_fields_here']
departments_pass_through_columns: ['department_custom_fields']
transactions_pass_through_columns: ['transactions_can_be_custom','pass_this_transaction_field_on']
transaction_lines_pass_through_columns: ['transaction_lines_field']
customers_pass_through_columns: ['customers_field']
locations_pass_through_columns: ['this_new_location','lets_also_add_this_location_field']
subsidiaries_pass_through_columns: ['subsidiaries_field']
consolidated_exchange_rates_pass_through_columns: ['this_exchange_rate','that_exchange_rate']
netsuite_database: your_destination_name
netsuite_schema: your_schema_name
```

### Changing the Build Schema
By default this package will build the Netsuite staging models within a schema titled (<target_schema> + `_stg_netsuite`) in your target database. If this is not where you would like your Netsuite staging data to be written to, add the following configuration to your `dbt_project.yml` file:
## (Optional) Step 5: Additional configurations
### Passing Through Additional Fields
This package includes all source columns defined in the macros folder. You can add more columns using our pass-through column variables. These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:

```yml
# dbt_project.yml
vars:
accounts_pass_through_columns:
- name: "new_custom_field"
alias: "custom_field"
classes_pass_through_columns:
- name: "this_field"
departments_pass_through_columns:
- name: "unique_string_field"
alias: "field_id"
transform_sql: "cast(field_id as string)"
transactions_pass_through_columns:
- name: "that_field"
transaction_lines_pass_through_columns:
- name: "other_id"
alias: "another_id"
transform_sql: "cast(another_id as int64)"
customers_pass_through_columns:
- name: "customer_custom_field"
alias: "customer_field"
locations_pass_through_columns:
- name: "location_custom_field"
subsidiaries_pass_through_columns:
- name: "sub_field"
alias: "subsidiary_field"
consolidated_exchange_rates_pass_through_columns:
- name: "consolidate_this_field"
```
### Change the build schema
By default, this package builds the Netsuite staging models within a schema titled (`<target_schema>` + `_netsuite_source`) in your destination. If this is not where you would like your netsuite staging data to be written to, add the following configuration to your root `dbt_project.yml` file:

...
```yml
models:
netsuite_source:
+schema: my_new_schema_name # leave blank for just the target_schema
```

### Change the source table references
If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
> IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_netsuite_source/blob/main/dbt_project.yml) variable declarations to see the expected names.

```yml
vars:
# For all Netsuite source tables
netsuite_<default_source_table_name>_identifier: your_table_name
# For all Netsuite2 source tables
netsuite2_<default_source_table_name>_identifier: your_table_name
```
### Override the data models variable
This package is designed to run **either** the Netsuite.com or Netsuite2 data models. However, for documentation purposes, an additional variable `netsuite_data_model_override` was created to allow for both data model types to be run at the same time by setting the variable value to `netsuite`. This is only to ensure the [dbt docs](https://fivetran.github.io/dbt_netsuite_source/) (which is hosted on this repository) is generated for both model types. While this variable is provided, we recommend you do not adjust the variable and instead change the `netsuite_data_model` variable to fit your configuration needs.

## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™

Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt). Learn how to set up your project for orchestration through Fivetran in our [Transformations for dbt Core™ setup guides](https://fivetran.com/docs/transformations/dbt#setupguide).

# 🔍 Does this package have dependencies?
This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the following packages, refer to the [dbt hub](https://hub.getdbt.com/) site.
> IMPORTANT: If you have any of these dependent packages in your own `packages.yml` file, we highly recommend that you remove them from your root `packages.yml` to avoid package version conflicts.
```yml
packages:
- package: fivetran/fivetran_utils
version: [">=0.3.0", "<0.4.0"]
- package: dbt-labs/dbt_utils
version: [">=0.8.0", "<0.9.0"]
```

# 🙌 How is this package maintained and can I contribute?
## Package Maintenance
The Fivetran team maintaining this package _only_ maintains the latest version of the package. We highly recommend that you stay consistent with the [latest version](https://hub.getdbt.com/fivetran/netsuite_source/latest/) of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_netsuite_source/blob/main/CHANGELOG.md) and release notes for more information on changes across versions.

## Contributions
Additional contributions to this package are very welcome! Please create issues
or open PRs against `main`. Check out
[this post](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657)
on the best workflow for contributing to a package.


## Database support
This package has been tested on BigQuery, Snowflake and Redshift.

## Resources:
- Provide [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next
- Have questions, feedback, or need help? Book a time during our office hours [here](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at solutions@fivetran.com
- Find all of Fivetran's pre-built dbt packages in our [dbt hub](https://hub.getdbt.com/fivetran/)
- Learn how to orchestrate your models with [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt)
- Learn more about Fivetran overall [in our docs](https://fivetran.com/docs)
- Check out [Fivetran's blog](https://fivetran.com/blog)
- Learn more about dbt [in the dbt docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [chat](http://slack.getdbt.com/) on Slack for live discussions and support
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the dbt blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions!

We highly encourage and welcome contributions to this package. Check out [this dbt Discourse article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) to learn how to contribute to a dbt package!

# 🏪 Are there any resources available?
- If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_netsuite_source/issues/new/choose) section to find the right avenue of support for you.
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
- Have questions or want to just say hi? Book a time during our office hours [on Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at solutions@fivetran.com.
Loading

0 comments on commit abb33e1

Please sign in to comment.