Manage tokens mode: UI for creating database-backed tokens with restricted permissions

[simonw]

I need this for Datasette Cloud. This would be a database-backed alternative to the new default signed token UI added in Datasette 1.0:

[simonw]

Here's the ambitious plan for this feature:

• Users can create tokens using a UI similar to the default one, but those tokens are stored in a database table
• Token permissions are stored in the database, probably as JSON to match the format shown here: https://docs.datasette.io/en/1.0a2/authentication.html#restricting-the-actions-that-a-token-can-perform
• This means tokens can be revoked and audited
• Auditing: tokens store a created date and a last-used date. The last-used date is only accurate to the nearest 60s (or 5m perhaps), to avoid having to write to the database every time a token is authenticated (we only write if it hasn't had a write in the last 60s). This means we can see which tokens are actively being used.
• Token permissions can be edited after the token has been created. This means users can issue broad-scoped tokens and then reduce permissions on them once they figure out what they need.

The most ambitious version:

• Token permission checks are also logged on a short-term basis (maybe just 10m or so) such that the UI can reflect which permissions a token actually needs to get its job done.

That last feature is my dream feature for API tokens, but I don't know how feasible it is to implement.

[simonw]

First attempt at schema design:

CREATE TABLE _datasette_auth_tokens (
   id INTEGER PRIMARY KEY,
   secret TEXT,
   permissions TEXT,
   actor_id TEXT,
   created_timestamp INTEGER,
   last_used_timestamp INTEGER
);

[simonw]

I'm going to turn this mode on with a plugin configuration option:

plugins:
  datasette-auth-tokens:
    manage_tokens: true

Where manage_tokens = "you should manage the tokens".

[simonw]

Updated schema:

CREATE TABLE _datasette_auth_tokens (
   id INTEGER PRIMARY KEY,
   secret TEXT,
   description TEXT,
   permissions TEXT,
   actor_id TEXT,
   created_timestamp INTEGER,
   last_used_timestamp INTEGER,
   expires_after_seconds INTEGER
);

Now has expires_after_seconds for storing e.g. 3600 to have the token expire after an hour.

And description for an optional note the user can specify describing what the token is being used for - useful for auditing.

[simonw]

Moving the work on this to a PR.

[simonw]

Schema update:

CREATE TABLE _datasette_auth_tokens (
    id INTEGER PRIMARY KEY,
    token_status TEXT DEFAULT 'L', -- [L]ive, [R]evoked, [E]xpired
    description TEXT,
    actor_id TEXT,
    permissions TEXT,
    created_timestamp INTEGER,
    last_used_timestamp INTEGER,
    expires_after_seconds INTEGER,
    secret_version INTEGER DEFAULT 0
);

The token_status column is new - it shows if a token is live, revoked or expired. This exists for two reasons:

• To enable faceting by those statuses on a regular table view
• To provide a column that a render_cell() hook can use to show a bit of custom UI, screenshot below

The secret column is gone. I decided that being able to publicly (at least to registered users) display the API keys table would save a bunch of time on implementing features for listing tokens, but preventing authorized users from select secret from _datasette_auth_tokens was too much hassle. So I'm switching to signed tokens instead, which means the tokens table doesn't need to store any secrets.

That's what secret_version is for - it allows for secret rotation without invalidating all existing API tokens. A value of 0 indicates that the DATASETTE_SECRET for the instance is being used. I'll add a mechanism later whereby you can instead configure 0 to be a fixed secret, then 1 to be another secret etc - so you can roll out new secrets without disabling existing API keys.

I moved actor_id to be next to description because it's a nicer order.

Table screenshot:

[simonw]

I started work on a view for this before deciding that it would be better to use an existing table implementation:

async def list_tokens(request, datasette):
    _check_permission(datasette, request)
    db = datasette.get_database()
    tokens = []
    for row in (
        await db.execute(
            "select id, description, permissions, created_timestamp, expires_after_seconds "
            "from _datasette_auth_tokens where actor_id = :actor_id order by id desc",
            {"actor_id": request.actor["id"]},
        )
    ).rows:
        tokens.append(
            {
                "id": row[0],
                "description": row[1],
                "permissions": json.loads(row[2]),
                "created_timestamp": row[3],
                "expires_after_seconds": row[4],
            }
        )
    context = await _shared(datasette, request)
    context.update({"tokens": tokens, "datasette": datasette})
    return Response.html(
        await datasette.render_template(
            "list_api_tokens.html", context, request=request
        )
    )

[simonw]

System should show a custom "token is revoked" or "token has expired" error.

[simonw]

I wanted to raise Forbidden("Token has expired") in actor_from_request() so that the user would get an error that their token had expired (or been revoked) - but it turns out that doesn't work cleanly in Datasette, exceptions raised from that hook turn into 500 errors at the moment.

So I return None instead and it looks like the token just fails silently.

[simonw]

Spotted a problem while working on this: if you grant a token access to view table for a specific table but don't also grant view database and view instance permissions, that token is useless.

This was a deliberate design decision in Datasette - it's documented on https://docs.datasette.io/en/1.0a2/authentication.html#access-permissions-in-metadata

If a user cannot access a specific database, they will not be able to access tables, views or queries within that database. If a user cannot access the instance they will not be able to access any of the databases, tables, views or queries.

I'm now second-guessing if this was a good decision.

[simonw]

I wanted to raise Forbidden("Token has expired") in actor_from_request() so that the user would get an error that their token had expired (or been revoked) - but it turns out that doesn't work cleanly in Datasette, exceptions raised from that hook turn into 500 errors at the moment.

Potential workaround: I could set the actor to {"token_error": "expired"} and then have a check permissions plugin hook that returns False if it sees that - though that might not solve the problem of wanting to display a custom error message. Maybe that method could raise Forbidden though.