Add Search Queries

[Tamriel]

When a block contains the query string {{[[query]]: {and: [[ex-A]] [[ex-B]]}}}, all blocks containing the strings ex-A and ex-B should be embedded below the query block.

The article Searching Roam With Queries describes this in detail.

[tangjeff0]

Agree something like this is needed, so we should keep this issue, but definitely don't like how Roam queries are implemented.

1. What is the underlying problem?
2. What is the solution space. Can the query interface actually map to a graph database?

[bmritz]

As a roam user, not having queries keeps me from switching to Athens. Roam has a lot of problems (as we all know), but the ability to template out a query to pull blocks from any part of your graph to a single spot is extremely powerful.

The underlying problem is: It is difficult to quickly collect all notes in your graph that meet the query criteria. Doing so without queries necessitates: 1) Go to a page 2: Go into the references section 3: Filter the mentions for what you want. That is a lot of clicking.

In Roam, I use queries for:

• For 1:1 catch ups. I collect all blocks that I've written over the week that were written to remind myself I need input from a specific person {{[[query]]: {and: #discussWith [[this person]] {not: [[DONE]]}}}}
• Follow ups from previous instances of the same recurring meetings. {{[[query]]: {and: [[meeting ref]] [[follow up]] {between: [[last month]] [[tomorrow]]} } }}

As a user who knows his way around code, I'd actually prefer to use datalog instead of a custom query language. but it is a MUST that the query results are nicely formatted for me. Roam pretty much nails this part. It could be better if it was more customizable (and the customizations were saved across instances of the graph being opened), but it is nice that query results in roam look nice without me having to do much.

[jtrakk]

Designing a query language is a very very big job and I don't know how much developer time will be dedicated to query-language design.

There are a lot of graph query languages out there, from very simple to complex. At the simple end is this blog post's proposed "TQL" which looks like (foo|bar) + baz. XPath or Cypher, for example, are much more complex.

It may be worth considering some meta-solutions:

1. picking an existing query language
2. designing a query-language plugin system, to allow users to import, develop, and experiment with alternative query languages within Athens

[agentydragon]

I'd propose just allowing people to issue Datascript queries against the database, and render the results nicely if they fit some format. For example, "if you issue a query that returns a single variable ?block, then we'll render the blocks; if it's ?page, we'll render page links".

Pros

• no need to add extra layer above DB
• allows full power of Datascript
  • (this is particularly an improvement over Roam's {{query}}, which has a very limited vocabulary)

Cons

• maybe security issues if running on server?
  • might need sanitization? (especially if executed outside client)
• hard to remove once added; if people can issue arbitrary queries, and the feature is taken away later, people's queries would break
  • so what I'd do would probably be initially add a big warning saying "this can break later", or even hide this behind a feature switch
• requires users to fiddle with underlying Datascript db instead of higher-level page/block structure
  • (but I expect power users would be okay with that)

[alexandergunnarson]

I'll add to the opinion that DataScript is a good way forward here. It makes it so we don't have to write a custom query language that everyone forgets anyway. So agreed about the pros.

Responses to cons:

• Security issues — wouldn't the query just be performed client-side on the local DataScript DB? If by "security" you mean authorization-related issues, we could just do a pre-filter on the DB for the datoms the user is allowed to see. Or something more efficient than that.
• Hard to remove once added — I'm not imagining a scenario where this feaure would be removed. Worst case it would be some feature flag that an admin would turn off for multiplayer DBs.
• Difficult to use — perhaps, but no more difficult than learning a different query language :)

[datokrat]

Some meta-considerations:

If we start with a very simple and restricted query language, there is no risk that the features will have to be taken away later (see @agentydragon's comment). If we then choose one day to modify the query language, we can provide a migration tool that parses all the blocks for query strings and replaces them with the new syntax.

The thing I like about this idea is that we can get our hands dirty quickly without really foreclosing any future possibilities, do we?

[alexandergunnarson]

I'd also say this would fit it in nicely to #2004. Basically you'd query just like you would with any other dynamic thing:

{{dynamic
;; Athens query API
(athens/query
  ;; DataScript query
  {:find ...
   :where ...})
}}

and a future /query command could just autofill the following template rather than having a special {{query ...}} syntax:

{{dynamic
(athens/query
  ...)
}}