[ADAP-500] [Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames #419

kochalex · 2023-04-27T19:40:34Z

Is this a new bug in dbt-redshift?

I believe this is a new bug in dbt-redshift
I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When upgrading to dbt 1.5, I get an 'index out of range error' when attempting to use things that connect to the database.

This is related to using a non-standard hostname for redshift - in my case I am opening a SSH tunnel to Redshift and using localhost as the host.

Expected Behavior

It works with the same configuration as prior versions of DBT or allows a region configuration to side-step this issue.

Steps To Reproduce

Start a SSH tunnel to redshift (the bug would also manifest if you have a cname to a redshift cluster) and localhost or the cname for the 'host' in your profiles.yml.
Attempt to run a model or do something that connects to redshift

Relevant log output

❯ dbt compile
19:37:05  Running with dbt=1.5.0
19:37:05  Found 111 models, 356 tests, 6 snapshots, 0 analyses, 746 macros, 0 operations, 1 seed file, 135 sources, 0 exposures, 0 metrics, 0 groups
19:37:05
19:37:05  Encountered an error:
list index out of range
19:37:05  Traceback (most recent call last):
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 190, in exception_handler
    yield
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/sql/connections.py", line 80, in add_query
    cursor = connection.handle.cursor()
             ^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/contracts/connection.py", line 94, in handle
    self._handle.resolve(self)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/contracts/connection.py", line 118, in resolve
    return self.opener(connection)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 249, in open
    connect=connect_method_factory.get_connect_method(),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 102, in get_connect_method
    "region": self.credentials.host.split(".")[2],
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 190, in exception_handler
    yield
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/base/impl.py", line 1043, in execute_macro
    result = macro_function(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 21, in macro
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 21, in macro
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 39, in macro
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 52, in macro
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/base/impl.py", line 289, in execute
    return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 259, in execute
    _, cursor = self.add_query(sql, auto_begin)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 284, in add_query
    connection, cursor = super().add_query(
                         ^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/sql/connections.py", line 58, in add_query
    self.begin()
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/sql/connections.py", line 170, in begin
    self.add_begin_query()
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/sql/connections.py", line 153, in add_begin_query
    return self.add_query("BEGIN", auto_begin=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 284, in add_query
    connection, cursor = super().add_query(
                         ^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/sql/connections.py", line 67, in add_query
    with self.exception_handler(sql):
  File "/Users/alexkoch/.asdf/installs/python/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 203, in exception_handler
    self.rollback_if_open()
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/base/connections.py", line 127, in rollback_if_open
    if conn is not None and conn.handle and conn.transaction_open:
                            ^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/contracts/connection.py", line 94, in handle
    self._handle.resolve(self)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/contracts/connection.py", line 118, in resolve
    return self.opener(connection)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 249, in open
    connect=connect_method_factory.get_connect_method(),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 102, in get_connect_method
    "region": self.credentials.host.split(".")[2],
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 86, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 71, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 142, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 215, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 250, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/cli/main.py", line 346, in compile
    results = task.run()
              ^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 441, in run
    result = self.execute_with_hooks(selected_uids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 403, in execute_with_hooks
    self.before_run(adapter, selected_uids)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 390, in before_run
    self.populate_adapter_cache(adapter)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 381, in populate_adapter_cache
    adapter.set_relations_cache(self.manifest)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/base/impl.py", line 461, in set_relations_cache
    self._relations_cache_for_schemas(manifest, required_schemas)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/impl.py", line 150, in _relations_cache_for_schemas
    super()._relations_cache_for_schemas(manifest, cache_schemas)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/base/impl.py", line 438, in _relations_cache_for_schemas
    for relation in future.result():
                    ^^^^^^^^^^^^^^^
  File "/Users/alexkoch/.asdf/installs/python/3.11.2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/.asdf/installs/python/3.11.2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/alexkoch/.asdf/installs/python/3.11.2/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/utils.py", line 464, in connected
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/sql/impl.py", line 180, in list_relations_without_caching
    results = self.execute_macro(LIST_RELATIONS_MACRO_NAME, kwargs=kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/base/impl.py", line 1042, in execute_macro
    with self.connections.exception_handler(f"macro {macro_name}"):
  File "/Users/alexkoch/.asdf/installs/python/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 203, in exception_handler
    self.rollback_if_open()
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/base/connections.py", line 127, in rollback_if_open
    if conn is not None and conn.handle and conn.transaction_open:
                            ^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/contracts/connection.py", line 94, in handle
    self._handle.resolve(self)
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/contracts/connection.py", line 118, in resolve
    return self.opener(connection)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 249, in open
    connect=connect_method_factory.get_connect_method(),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexkoch/Library/Caches/pypoetry/virtualenvs/espark-dbt-7mBGFPKz-py3.11/lib/python3.11/site-packages/dbt/adapters/redshift/connections.py", line 102, in get_connect_method
    "region": self.credentials.host.split(".")[2],
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Environment

- OS: ubuntu 20.04, OSX 13.3.1
- Python: 3.11.2
- dbt-core: 1.5.0
- dbt-redshift: 1.5.0

Additional Context

The adapter is hard-coded to parse the region from the host value.

dbt-redshift/dbt/adapters/redshift/connections.py

Line 102 in aa98c95

"region": self.credentials.host.split(".")[2],

The text was updated successfully, but these errors were encountered:

padbk · 2023-04-27T20:31:50Z

Yes, same issue. Using a CNAME record for the redshift host, therefore region is not present in the host.

There needs to be some other way of specifying region, probably in profiles.yml.

dbeatty10 · 2023-04-28T11:40:36Z

Thanks for reporting this @kochalex and @padbk !

It looks like the region was hard-coded to be parsed from the hostname when switching from psycopg2 to redshift_connector in #251.

Agreed with your recommended path forward of either adding a new region option within the dbt profiles.yml or remove the parsing region from the hostname (gracefully handling its non-existence).

Acceptance criteria

Either of the following is sufficient as exit criteria:

Either SSH tunneling or CNAME records work as-is (which is possible in dbt-redshift 1.4.x)
or
The Redshift profile allows a region configuration.
The docs are updated accordingly to reflect the new region parameter.

Implementation note

The redshift_connector docs have region listed as optional, so maybe this line can just be removed altogether?

dbt-redshift/dbt/adapters/redshift/connections.py

Line 102 in aa98c95

"region": self.credentials.host.split(".")[2],

…ft hostnames (#420) * Fix for region parsing in host name.WIP * Edge cases and cleanup * Minor nits * clean up aws regions invocation * Log message and doc nits * Changie entry * Skip validation if aws regions are not determined * move the regions var earlier to effectively cache

…ft hostnames (#420) * Fix for region parsing in host name.WIP * Edge cases and cleanup * Minor nits * clean up aws regions invocation * Log message and doc nits * Changie entry * Skip validation if aws regions are not determined * move the regions var earlier to effectively cache (cherry picked from commit b644963)

…ft hostnames (#420) (#421) * Fix for region parsing in host name.WIP * Edge cases and cleanup * Minor nits * clean up aws regions invocation * Log message and doc nits * Changie entry * Skip validation if aws regions are not determined * move the regions var earlier to effectively cache (cherry picked from commit b644963) Co-authored-by: Neelesh Salian <nssalian@users.noreply.github.com>

pdebelak · 2023-05-18T15:09:37Z

This should probably be documented as a breaking change in the 1.5 changelog as well.

pdebelak · 2023-06-06T15:05:53Z

Looking into this, it seems like region isn't a required parameter at all at least if you are connecting with a username and password. Can we investigate if we can either not require region at all or only require it in situations where it is actually required?

dbeatty10 · 2023-06-07T22:19:25Z

@pdebelak doing that investigation is a very reasonable suggestion. Namely, investigate if we make region optional and only pass it to redshift_connector in certain situations.

If it's possible to not require it, it would solve the concern raised in #483.

I created this spike to investigate:
#484

…d redshift hostnames (dbt-labs#420) * Fix for region parsing in host name.WIP * Edge cases and cleanup * Minor nits * clean up aws regions invocation * Log message and doc nits * Changie entry * Skip validation if aws regions are not determined * move the regions var earlier to effectively cache

kochalex added type:bug Something isn't working triage:product labels Apr 27, 2023

github-actions bot changed the title ~~[Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames~~ [ADAP-500] [Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames Apr 27, 2023

dbeatty10 self-assigned this Apr 28, 2023

dbeatty10 removed the triage:product label Apr 28, 2023

dbeatty10 assigned nssalian and unassigned dbeatty10 Apr 28, 2023

dbeatty10 added the type:regression label Apr 28, 2023

nssalian mentioned this issue Apr 28, 2023

Fix for #419: dbt-redshift 1.5 does not work with non-standard redshift hostnames #420

Merged

6 tasks

nssalian closed this as completed in #420 Apr 28, 2023

jeremyyeo mentioned this issue Apr 30, 2023

[CT-2505] [Regression] IndexError: list index out of range in dbt build dbt-labs/dbt-core#7483

Closed

2 tasks

nssalian mentioned this issue May 1, 2023

[Backport 1.5.latest] Fix for #419: dbt-redshift 1.5 does not work with non-standard redshift hostnames #421

Merged

github-actions bot added the triage:product label May 18, 2023

dataders added the redshift_connector label May 26, 2023

dbeatty10 removed the triage:product label Jun 1, 2023

dbeatty10 mentioned this issue Jun 7, 2023

[ADAP-610] [Spike] Investigate if region is absolutely required #484

Closed

alison985 mentioned this issue Jun 14, 2023

Pass region param on to redshift conn as is #485

Merged

6 tasks

colin-rogers-dbt mentioned this issue Jun 20, 2023

backport 474 to 1.5.latest #502

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ADAP-500] [Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames #419

[ADAP-500] [Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames #419

kochalex commented Apr 27, 2023

padbk commented Apr 27, 2023

dbeatty10 commented Apr 28, 2023

pdebelak commented May 18, 2023

pdebelak commented Jun 6, 2023

dbeatty10 commented Jun 7, 2023

[ADAP-500] [Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames #419

[ADAP-500] [Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames #419

Comments

kochalex commented Apr 27, 2023

Is this a new bug in dbt-redshift?

Current Behavior

Expected Behavior

Steps To Reproduce

Relevant log output

Environment

Additional Context

padbk commented Apr 27, 2023

dbeatty10 commented Apr 28, 2023

Acceptance criteria

Implementation note

pdebelak commented May 18, 2023

pdebelak commented Jun 6, 2023

dbeatty10 commented Jun 7, 2023