Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update get_tables_by_pattern_sql.sql #563

Closed
wants to merge 1 commit into from

Conversation

silvioluiz
Copy link

@silvioluiz silvioluiz commented Apr 27, 2022

Create specific athena macro because ilike is not a valid function

This is a:

  • bug fix with no breaking changes — please ensure the base branch is next/patch
  • new functionality — please ensure the base branch is next/minor
  • a breaking change — please ensure the base branch is next/major

Description & motivation

When I trying to use dbt codegen to generate source yaml's, I get a query error:

dbt run-operation generate_source --args '{"schema_name": "ktech_datalake_raw", "table_pattern": "course_course"}' 16:17:53 Running with dbt=1.0.5 Failed to execute query. Traceback (most recent call last): File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pyathena/common.py", line 307, in _execute query_id = retry_api_call( File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pyathena/util.py", line 84, in retry_api_call return retry(func, *args, **kwargs) File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__ do = self.iter(retry_state=retry_state) File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter return fut.result() File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result raise self._exception File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__ result = fn(*args, **kwargs) File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/site-packages/botocore/client.py", line 388, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/silvio/.pyenv/versions/3.8.6/lib/python3.8/site-packages/botocore/client.py", line 708, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: line 18:28: mismatched input 'ilike'. Expecting: '%', '*', '+', '-', '.', '/', 'AT', '[', '||', <expression> Failed to execute query.

This errors happens because the macro get_tables_by_pattern_sql uses like function (not supported on Athena)

Checklist

  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
    • Athena
  • I followed guidelines to ensure that my changes will work on "non-core" adapters by:
    • dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
    • using the limit_zero() macro in place of the literal string: limit 0
    • using dbt_utils.type_* macros instead of explicit datatypes (e.g. dbt_utils.type_timestamp() instead of TIMESTAMP
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)
  • I have added an entry to CHANGELOG.md

Create  specific athena macro because ilike is not a valid function
@joellabes
Copy link
Contributor

@silvioluiz thanks for this! right now, this package is just for the core databases that dbt Labs supports; https://github.com/lalalilo/athena-utils is the place to go for custom overrides of macros in dbt utils.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants