Add ability to open query in Iodide

[jezdez]

Following the discussion around #424 this is an issue to track the possibility to integrate Redash with Iodide(.io).

The general gist is:

• allow opening a query and its results in iodide
• keep security implications in mind (mitigation of data leakage and credential leakage)
• decide whether the data in the Iodide notebook would be allowed to be updated once refreshed in Redash
• is it sensible to allow opening a query in many different notebooks or is there a 1:1 relationship?
• what's the form of transport for the query results (e.g. query result table contains only paginated content IIRC) to the Iodide server given the limitations of query parameters

@wlach do you remember any other things we talked about on IRC?

[wlach]

We also talked a bit about adding some kind of POST function on the side of an iodide server to create a notebook with a specific payload, which would allow (for example) prepopulating the data for a query inside an iodide cell.

Let's circle back to this once we have a bit more of a server-side implementation in place.

[jezdez]

Ah right, so a possible UX flow would be:

• user clicks button [open in iodide] in redash query
• that sends request to redash backend handler
• which triggers a redash backend task sending the query results and metadata to iodide API via POST
• iodide stores data and notebook and returns URL of notebook
• redash returns url to user frontend for opening in new tab

[openjck]

I spoke with William about this yesterday. He was saying that, for now, we may want to send only JSMD over POST. The JSMD will include code which loads data from Redash.

In other words, the data itself wouldn't be sent over POST, but the code for loading that data would be.

Let me know if that's a problem.

[openjck]

@jezdez Can you point me in the right direction regarding building my first Redash plugin? Is there any documentation I should be looking at?

[jezdez]

@openjck There is no documentation to speak of right now, since the extension API is still in flux but we've made some progress fleshing it out in redash-stmo.

My apologies, but I'm afraid you'll need to look at the existing extensions that both cover backend and simple frontend extensions, but I'll try to summarize the current design below.

What is redash-stmo?

redash-stmo is a Python package that contains several extensions, each using some and in some cases all abilities. Since Redash currently lacks the ability to install additional dependencies during deploy time, we install it in our fork. This means that

• when we work on our extensions in redash-stmo we'll need to do a release of redash-stmo to PyPI (this automatically happens when a Git tag is created).
• our Redash fork does not need to be updated since it'll always fetch the latest version of redash-stmo, but it will need a new STMO deploy
• there are still pieces coming together in the upstream Redash project to make the extension writing ability easier

What is a Redash extension anyway?

The current extension design is separated into three parts:

1. backend extensions in Python that are loaded when the Redash backend starts
2. additional non-Python files that are automatically copied during the Docker image build proccess into the client/app/extensions directory so webpack can pick it up when it builds the client application bundle
3. periodic Celery tasks that can be registered with Redash's Celery cluster

For example the extension to add a link to the Datasource documentation to the schema browser can be found here: redash_stmo/data_sources/link

Backend extensions

A backend extension is simply Python code that is (e.g. redash_stmo/data_sources/link/__init__.py) imported and called by Redash's extension loading mechanism.

It does that by scanning all installed Python packages for package metadata called "entry points" that are just dotted Python import paths pointing to a function, e.g. redash_stmo.data_sources.link:extension:

• redash_stmo.data_sources.link is the import path of the redash_stmo/data_sources/link/__init__.py file
• extension the name of the function inside it that is called by the extension loading mechanism when the Redash backend starts

Client extensions

Client extensions are React components that are wrapped in AngularJS modules (until Redash has been ported to React natively), located in bundle directories of Python packages. 👐

So a Python package with an extension bundle could look like this:

my_extensions/
├── __init__.py
└── wide_footer
    ├── __init__.py
    └── bundle
        ├── extension.js
        └── styles.css

Equally to the backend extensions, Redash will use the entry point system to find those bundle directories by deriving its file system path from the dotted Python import path of the Python module the bundle directory is contained in. This all happens when when it copies the files into the client/app/extensions directory that webpack knows how to compile during build Docker image build process.

Client extensions should define the entry point of the Python module with the bundle directory just like backend extensions in the pyroject.toml file.

Once the client extension is part of the client application bundle, it'll need to be actually used in HTML files. That's sadly an unfinished part of the extension system at the moment and requires us to add it to our fork.

E.g. the datasource link extension from above is included in frontend HTML file in our fork.

In the future we hope to have a different system for this final piece of integration, e.g. pre-existing React components ("portals", "placeholders" etc) that we can register our extension components with. Alas that hasn't been tackled upstream yet.

Periodic Celery tasks

This is an area of current discussion and we should probably not using this part of the extension API at the moment. Let me know if you have a strong use case for this.

[jezdez]

I should note that the redash-stmo repo is set up to allow development of extensions without knowing all of this, as long as the entry for a new extension is added to the pyproject.toml file.

[openjck]

Excellent. Thank you for this detailed information!

I forgot that there were the existing extensions. Those, along with this documentation, will be a huge help.

🤗

[jezdez]

This was done! 🥳 🎉