diff --git a/docs/datasources.rst b/docs/datasources.rst index 01cc9db748..73a0539e3d 100644 --- a/docs/datasources.rst +++ b/docs/datasources.rst @@ -1,162 +1,117 @@ Supported Data Sources ###################### -re:dash supports several types of data sources (see below the full list) -and their management is done with the CLI (``manage.py``): +re:dash supports several types of data sources, and if you set it up using the provided images, it should already have +the needed dependencies to use them all. Starting from version 0.7 and newer, you can manage data sources from the UI +by browsing to ``/data_sources`` on your instance. -Create new data source -====================== +If one of the listed data source types isn't available when trying to create a new data source, make sure that: -.. code:: bash +1. You installed required dependencies. +2. If you've set custom value for the ``REDASH_ENABLED_QUERY_RUNNERS`` setting, it's included in the list. - $ cd /opt/redash/current - $ sudo -u redash bin/run ./manage.py ds new -n {name} -t {type} -o {options} - -If you omit any of the options (-n, -t, -o) it will show a prompt asking -for it. Options is a JSON string with the connection parameters. Unless -you're doing some sort of automation, it's probably easier to leave it -empty and fill out the prompt. +PostgreSQL / Redshift +--------------------- -See below for the different supported data sources types and the -relevant options string format. +- **Options**: -Listing existing data sources -============================= + - Database name (mandatory) + - User + - Password + - Host + - Port -.. code:: bash +- **Additional requirements**: - $ sudo -u redash bin/run ./manage.py ds list + - None -Supported data sources -====================== -PostgreSQL / Redshift ---------------------- +MySQL +----- -- **Type**: pg - **Options**: - - User (user) - - Password (password) - - Host (host) - - Port (port) - - Database name (dbname) (mandatory) + - Database name (mandatory) + - User + - Password + - Host + - Port -- **Options string format (for v0.5 and older)**: "user= password= - host= port=5439 dbname=" +- **Additional requirements**: -MySQL ------ + - ``MySQL-python`` python package + + +Google BigQuery +--------------- -- **Type**: mysql - **Options**: - - User (user) - - Password (passwd) - - Host (host) - - Port (port) - - Database name (db) (mandatory) + - Project ID (mandatory) + - JSON key file, generated when creating a service account (see `instructions `__). -- **Options string format (for v0.5 and older)**: - "Server=localhost;User=;Pwd=;Database=" -Note that you need to install the MySQLDb package as it is not included -in the ``requirements.txt`` file. +- **Additional requirements**: + + - ``google-api-python-client``, ``oauth2client`` and ``pyopenssl`` python packages (on Ubuntu it might require installing ``libffi-dev`` and ``libssl-dev`` as well). + Graphite -------- -- **Type**: graphite - **Options**: - - Url (url) (mandatory) - - User (username) - - Password (password) - - Verify SSL ceritficate (verify) + - Url (mandatory) + - User + - Password + - Verify SSL certificate -- **Options string format**: '{"url": - "https://graphite.yourcompany.com", "auth": ["user", "password"], - "verify": true}' -Google BigQuery ---------------- +MongoDB +------- -- **Type**: bigquery - **Options**: - - Service Account (serviceAccount) (mandatory) - - Project ID (projectId) (mandatory) - - Private Key filename (privateKey) (mandatory) + - Connection String (mandatory) + - Database name + - Replica set name -- **Options string format (for v0.5 and older)**: {"serviceAccount" : - "43242343247-fjdfakljr3r2@developer.gserviceaccount.com", - "privateKey" : "/somewhere/23fjkfjdsfj21312-privatekey.p12", - "projectId" : "myproject-123" } +- **Additional requirements**: -Notes: + - ``pymongo`` python package. -1. To obtain BigQuery credentials follow the guidelines at: - https://developers.google.com/bigquery/authorization#service-accounts -2. You need to install the ``google-api-python-client``, - ``oauth2client`` and ``pyopenssl`` packages (PyOpenSSL requires - ``libffi-dev`` and ``libssl-dev`` packages), as they are not included - in the ``requirements.txt`` file. +For information on how to write MongoDB queries, see :doc:`documentation `. -Google Spreadsheets -------------------- - -(supported from v0.6.4) -- **Type**: google\_spreadsheets -- **Options**: - - - Credentials filename (credentialsFilePath) (mandatory) +ElasticSearch +------------- -Notes: +... -1. To obtain Google ServiceAccount credentials follow the guidelines at: - https://developers.google.com/console/help/new/#serviceaccounts (save - the JSON version of the credentials file) -2. To be able to load the spreadsheet in re:dash - share your it with - your ServiceAccount's email (it can be found in the credentials json - file, for example - 43242343247-fjdfakljr3r2@developer.gserviceaccount.com) Note: all the - service account details can be seen inside the json file you should - obtain following step #1 -3. The query format is "DOC\_UUID\|SHEET\_NUM" (for example - "kjsdfhkjh4rsEFSDFEWR232jkddsfh\|0") -4. You (might) need to install the ``gspread``, ``oauth2client`` and - ``dateutil`` packages as they are not included in the - ``requirements.txt`` file. +InfluxDB +-------- -MongoDB -------- +... -- **Type**: mongo -- **Options**: +Presto +------ - - Connection String (connectionString) (mandatory) - - Database name (dbName) - - Replica set name (replicaSetName) +... -- **Options string format (for v0.5 and older)**: { "connectionString" - : "mongodb://user:password@localhost:27017/mydb", "dbName" : "mydb" } +Hive +---- -For ReplicaSet databases use the following connection string: \* -**Options string format**: { "connectionString" : -"mongodb://user:pasword@server1:27017,server2:27017/mydb", "dbName" : -"mydb", "replicaSetName" : "myreplicaSet" } +... -Notes: +Impala +------ -1. You need to install ``pymongo``, as it is not included in the - ``requirements.txt`` file. +... URL --- -A URL based data source which requests URLs that conforms to the -supported :doc:`results JSON +A URL based data source which requests URLs that return the :doc:`results JSON format `. Very useful in situations where you want to expose the data without @@ -165,81 +120,47 @@ connecting directly to the database. The query itself inside re:dash will simply contain the URL to be executed (i.e. http://myserver/path/myquery) -- **Type**: url - **Options**: - - Url (url) + - Url - set this if you want to limit queries to certain base path. -- **Options string format (optional) (for v0.5 and older)**: - http://myserver/path/ -Notes: - -1. All URLs must return the supported :doc:`results JSON - format `. -2. If the Options string is set, only URLs that are part of the supplied - path can be executed using this data source. Not setting the options - path allows any URL to be executed as long as it returns the - supported :doc:`results JSON - format `. - -Script ------- - -Allows executing any executable script residing on the server as long as -its standard output conforms to the supported :doc:`results JSON -format `. - -This integration is useful in situations where you need more than just a -query and requires some processing to happen. - -Once the path to scripts is configured in the datasource the query needs -to contain the file name of the script as well as any command line -parameters the script requires (i.e. myscript.py param1 param2 ---param3=value) +Google Spreadsheets +------------------- -- **Type**: script - **Options**: - - Scripts Path (path) (mandatory) + - JSON key file, generated when creating a service account (see `instructions `__). + +- **Additional requirements**: -- **Options string format (for v0.5 and older)**: /path/to/scripts/ + - ``gspread`` and ``oauth2client`` python packages. Notes: -1. You MUST set a path to execute the scripts, otherwise the data source - will not work. -2. All scripts must be executable, otherwise results won't return -3. The script data source does not allow relative paths in the form of - "../". You may use a relative sub path such as "./mydir/myscript". -4. All scripts must output to the standard output the supported :doc:`results - JSON format ` and - only that, otherwise the data source will not be able to load the - data. +1. To be able to load the spreadsheet in re:dash - share your it with + your ServiceAccount's email (it can be found in the credentials json + file, for example + 43242343247-fjdfakljr3r2@developer.gserviceaccount.com). +2. The query format is "DOC\_UUID\|SHEET\_NUM" (for example + "kjsdfhkjh4rsEFSDFEWR232jkddsfh\|0") + Python ------ -Execute other queries, manipulate and compute with Python code -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Python data source allows running Python code in a secure and safe -environment. It won't allow writing files to disk, importing modules -that were not pre-approved in the configuration etc. - -One of the benefits of using the Python data source is its ability to -execute queries (or saved queries) which you can store in a variable and -then manipulate/transform/merge with other data and queries. +**Execute other queries, manipulate and compute with Python code** -You can import data analysis libraries such as Pandas, NumPy and SciPy. +This is a special query runner, that will execute provided Python code as the query. Useful for various scenarios such as +merging data from different data sources, doing data transformation/manipulation that isn't trivial with SQL, merging +with remote data or using data analysis libraries such as Pandas (see `example query `__). -This saved the trouble of having outside scripts do the synthesis of -data from multiple sources to create a single data set that can then be -used in dashboards. +While the Python query runner uses a sandbox (RestrictedPython), it's not 100% secure and the security depends on the +modules you allow to import. We recommend enabling the Python query runner only in a trusted environment (meaning: behind +VPN and with users you trust). -- **Type**: Python - **Options**: - Allowed Modules in a comma separated list (optional). **NOTE:** You MUST make sure these modules are installed on the machine - running the Celery workers + running the Celery workers. diff --git a/docs/setup.rst b/docs/setup.rst index 8bb119bdd1..c28714fe20 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -81,7 +81,7 @@ Setup Once you created the instance with either the image or the script, you should have a running re:dash instance with everything you need to get -started. You can even login to it with the user "admin" (password: +started. You can now login to it with the user "admin" (password: "admin"). But to make it useful, there are a few more steps that you need to manually do to complete the setup: @@ -133,18 +133,14 @@ file. Datasources ----------- -To make re:dash truly useful, you need to setup your data sources in it. -Currently all data sources management is done with the CLI. +To make re:dash truly useful, you need to setup your data sources in it. Browse to ``/data_sources`` on your instance, +to create new data source connection. See :doc:`documentation ` for the different options. Your instance comes ready with dependencies needed to setup supported sources. -Follow issue -`#193 `__ to know -when UI was implemented to manage data sources. - How to upgrade? --------------- diff --git a/redash/settings.py b/redash/settings.py index 6ab7230f9a..aa7efa7572 100644 --- a/redash/settings.py +++ b/redash/settings.py @@ -121,7 +121,6 @@ def all_settings(): 'redash.query_runner.mongodb', 'redash.query_runner.mysql', 'redash.query_runner.pg', - 'redash.query_runner.script', 'redash.query_runner.url', 'redash.query_runner.influx_db', 'redash.query_runner.elasticsearch',