Feature: DynamoDB DQL query runner #934

alexanderlz · 2016-03-17T10:54:06Z

Adds the ability to query DynamoDB using SQL!
(with the help of the following library
https://github.com/mathcamp/dql
)

arikfr · 2016-03-17T10:56:33Z

redash/utils/__init__.py

@@ -66,6 +66,9 @@ def default(self, o):
        if isinstance(o, datetime.timedelta):
            return str(o)

+        if isinstance(o, unicode):


Why is this needed? 😮

arikfr · 2016-03-23T09:15:27Z

requirements.txt

@@ -19,7 +19,8 @@ python-dateutil==2.4.2
 pytz==2015.7
 redis==2.7.5
 requests==2.3.0
-six==1.5.2
+botocore==1.4.4


had to update the six lib (to 1.10.0), to be compatible with dql

arikfr · 2016-03-29T19:00:36Z

Thanks!

mvolpato · 2016-05-16T19:54:52Z

Can you please give an explanation of the fields used by DynamoDB connection? Some are clear to me (Access_key, Secret_key, Region), the others are not.

I get this error:

Error running query: InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. The Canonical String for this request should have been 'POST / content-type:application/x-amz-json-1.0 host:dynamodb.eu-west-1.amazonaws.com user-agent:Botocore/1.4.4 Python/2.7.11 Linux/3.13.0-85-generic x-amz-date:20160516T194838Z x-amz-target:DynamoDB_20120810.DescribeTable content-type;host;user-agent;x-amz-date;x-amz-target [hash here]' The String-to-Sign should have been 'AWS4-HMAC-SHA256 20160516T194838Z 20160516/eu-west-1/dynamodb/aws4_request [hash here]' Args: {'TableName': 'tablename'}

I used dynamodb.eu-west-1.amazonaws.com as Host, is_secore false, Port 80 (or 8000), Region eu-west-1.

alexanderlz · 2016-05-16T21:29:50Z

Hi @mvolpato,
Which boto3 version are you using?

mvolpato · 2016-05-16T22:28:02Z

Hi @alexanderlz
No idea. I simply used the container provided by Bitnami on AWS.

arikfr · 2016-05-17T11:39:17Z

@mvolpato see #1059. It's a bug. Once #1059 is merged, just create a new DynamoDB data source without specifying host.

mvolpato · 2016-05-17T15:28:08Z

Thank you @arikfr. Looking forward for the release.

Feature: DynamoDB DQL query runner

* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes getredash#927, getredash#928: Schema refresh improvements. * Closes getredash#934, getredash#935: Remove type from schema browser and don't show empty example column in schema drawer (getredash#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes getredash#939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Co-authored-by: Alison <github@bankofknowledge.net>

* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes getredash#927, getredash#928: Schema refresh improvements. * Closes getredash#934, getredash#935: Remove type from schema browser and don't show empty example column in schema drawer (getredash#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes getredash#939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Schema Improvements Part 2: Add data source config options. Adding BigQuery schema drawer with data types and samples. Add empty migration to replace the removed schedule_until migration Add merge migration. Co-authored-by: Alison <github@bankofknowledge.net>

alexanderlz added 4 commits March 17, 2016 03:13

dynamodb SQL connector

d88d1a1

add requirements

158fc62

the dynamoDB SQL connector

629c16a

Merge remote-tracking branch 'upstream/master'

c2f3c55

arikfr reviewed Mar 17, 2016
View reviewed changes

alexanderlz added 2 commits March 17, 2016 13:09

fixes

2e93af3

requirements rearrangement

ad95a5e

arikfr reviewed Mar 23, 2016
View reviewed changes

code style, name method

7bb80e3

arikfr changed the title ~~DynamoDB SQL query runner~~ Feature: DynamoDB SQL query runner Mar 29, 2016

arikfr changed the title ~~Feature: DynamoDB SQL query runner~~ Feature: DynamoDB DQL query runner Mar 29, 2016

arikfr merged commit d83dfc9 into getredash:master Mar 29, 2016

dairyo pushed a commit to KiiCorp/redash that referenced this pull request Mar 1, 2019

Merge pull request getredash#934 from alexanderlz/master

c3eaaf4

Feature: DynamoDB DQL query runner

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: DynamoDB DQL query runner #934

Feature: DynamoDB DQL query runner #934

alexanderlz commented Mar 17, 2016

arikfr Mar 17, 2016

arikfr Mar 23, 2016

alexanderlz Mar 23, 2016

arikfr commented Mar 29, 2016

mvolpato commented May 16, 2016

alexanderlz commented May 16, 2016

mvolpato commented May 16, 2016

arikfr commented May 17, 2016

mvolpato commented May 17, 2016

Feature: DynamoDB DQL query runner #934

Feature: DynamoDB DQL query runner #934

Conversation

alexanderlz commented Mar 17, 2016

arikfr Mar 17, 2016

Choose a reason for hiding this comment

arikfr Mar 23, 2016

Choose a reason for hiding this comment

alexanderlz Mar 23, 2016

Choose a reason for hiding this comment

arikfr commented Mar 29, 2016

mvolpato commented May 16, 2016

alexanderlz commented May 16, 2016

mvolpato commented May 16, 2016

arikfr commented May 17, 2016

mvolpato commented May 17, 2016