[Zelda] Post/Tag/User auto-complete 'search' bar

[ErisDS]

In the new 'Zelda' interface, you may have noticed there's a search bar:

However, due to the technical challenges involved, Search is still a way off being implemented into the Ghost API.

Managing a large body of posts without any option to filter or search them is incredibly cumbersome, and as Ghost gets older and many of the blogs on it get bigger, this issue gets more problematic. Therefore, it seems adding a rudimentary search capability would be better than nothing at all. Originally we had an idea for a filtering menu but this never really took flight because it didn't feel very useful.

The idea in Zelda is to use the search box as an auto-complete style interface, much like we already have for tags:

This box should return results for posts with matching titles, tags with matching names, users with matching names names, and preferably posts which have any matching tag or user/author.

The display of the results should be split into groups for each of posts, tags, and users, much like you see in Slack. As different endpoints will be queried to find suggestions for each resource type, it is anticipated that each result set would load and update individually as you type. This is also what you see if you use slack's search box & it feels really nice to use.

There's no design for the Ghost version yet, as this is problem is better suited to being developed as best as it can be to start with, and designed afterwards.

The Tag suggestion box fetches all tags using the Tags API endpoint, and the filtering is done client side. The Posts & User API endpoints will also allow for fetching all content. One possible optimisation we could make would be to add a fields parameter to the endpoints, so that we can fetch just the object ids along with the attributes we want to filter on and maintain these versions of the models separately just for the suggestions box - but I'm not sure how well this would work with Ember Data?

I think it may make sense to use an existing solution like typeahead.js to implement this - it supports multiple datasets and there are even existing ember components although I haven't found one that's particularly up-to-date.

[acburdine]

There's also this library for autocomplete: http://leaverou.github.io/awesomplete/

I will look at both and see which one would work better with ember/what we need it to do.

[morficus]

I've done some work on the API side of these and have 2 implementations:

• Implantation 1: enhance existing /users, /tags and /posts API to support new query params to filter/search as well as limiting fields returned (commit: morficus@bc23f22)
• Implantation 2: create end end-points specifically for searching each entity (/users/search, /tags/search and /posts/search)(commit: morficus@919e090)

The first approach requires less code, but I'm not a big fan of it since the returned results will always be paginated (making it kind of messy for Ember to deal with) and it increases the complexity of the existing methods.
I personally prefer the 2nd approach, mainly because it keeps the search bits isolated from existing functionality and the results are not paginated. Keeping the search bits contained to their own methods is nice... because when we get full-blown real search, it would be a lot easier to identify what needs to be removed.

@ErisDS are you fine with this approach or did you have something else in mind?

[sebgie]

I'm also in favor of /.../search, 👍 to everything @morficus said :).

How would the actual query in this case look like? I would like to propose using /../search/?q=<your query>? Searches that are for certain fields only could be encoded in the query (shamelessly stolen from Goolgle):

• /../search/?q=test - search for test in all fields
• /../search/?q=author:test - search for test in author

[morficus]

@sebgie this is what I had in mind, but I'm always open to suggestions: https://gist.github.com/morficus/c1ea94d5aaa0478e6d74

[ErisDS]

I think this suggestion has strayed into true search territory? In my opinion, any search endpoints in the API should be offering full text search as per #5321/#306.

The aim here is to use a client-side search/autocomplete library, and load all the necessary data from the API using Ember Data. To optimise this, the idea was to potentially add a ?fields= parameter to the API so that we can fetch just the key fields (post title), rather than the entire body.

The fields parameter certainly needs a spec and some consideration on how it should work with relations/nested resources. I didn't do that here as there were too many unanswered questions about how the client side would work. For example, if I start typing 'photo' into the auto complete box, and that's one of my tags, do we expect that just the tag 'photo' appears in the result, or also posts with that tag?

I'd suggest the former purely as a first step and expanding after. However, I haven't got a clear enough idea of how typeahead might work with relations vs using Ember Data, and whether Ember Data can cope with having only a few fields populated and fetching more data when needed. So although building a super generalised API is definitely a goal, there's a certain amount of leading from the front(end) that needs to be done so that we build the most useful things first.

With regard to more advanced filtering on the API itself, I have been working on a discussion issue/spec for that (not sure where it's going to go just yet) and will incorporate the ideas in your gist.

[morficus]

The aim here is to use a client-side search/autocomplete library, and load all the necessary data from the API using Ember Data.

Hm. So the initial attempt is for it to be 100% client-side search/filtering?
I'm not sure how useful that is (specifically in the context of posts)... since Ember Data only has a limited sub-set of posts at any given moment (unless you have paginated to the end or max-out the "results per page" attribute when paginating). That would seem like a big hack. But maybe I'm missing something?

However, I haven't got a clear enough idea of how typeahead might work with relations vs using Ember Data, and whether Ember Data can cope with having only a few fields populated and fetching more data when needed.

@acburdine is looking in to that, so he should have some better insight on how (or even if) that works.

I have been working on a discussion issue/spec for that (not sure where it's going to go just yet) and will incorporate the ideas in your gist.

The gist was only intended as documentation for acburdine to have something to work off of :-)

[ErisDS]

Hm. So the initial attempt is for it to be 100% client-side search/filtering?

Yep - the spec is to do the same thing we're already doing for the tag suggestions.

Ember Data only has a limited sub-set of posts at any given moment (unless you have paginated to the end or max-out the "results per page" attribute when paginating).

Right - so there are going to be different ways to do this and optimise it, which is why I only mentioned limiting by field as a suggestion. It depends on what can be done with typeahead/awesomplete vs Ember Data here but the API already supports limit=all for all resources. With the tag suggestions, we simply load all tag data when the editor is loaded. That's an unnecessary pre-optimisation. Done correctly that loading can be delayed until a lookup is requested.

This way, only the first lookup will be slow, whereas if you're doing a server-side filter every time then it'll be slow every time. That tradeoff might be worthwhile if we had genuine search occurring on the server, but I don't see that there's much benefit to firing off an ajax request to the server every time a character is typed?

Perhaps it makes more sense to link directly to the multiple dataset typeahead demo:
http://twitter.github.io/typeahead.js/examples/#multiple-datasets

This is definitely a little hacky - but it's intended to be a lack-of-search workaround. Typeahead is neat because not only will it manage loading multiple datasets, but it has a nice set of events meaning we can hook into an item being selected and use it to load the post / tag / user in the Ghost admin, instead of just filling it out in the input field. I envisage using the same thing (more naturally) for both tag suggestions and for offering smart URL autocomplete on the navigation settings page (and perhaps even in the editor one day).

From reading around, it does seem that Ember Data can be convinced to handle partial loading of models and filling them out later. However, I think that's going to be an optimisation - typeahead can have it's own dataset of strings separate to Ember Data to start with.

If we're not using Ember Data with partial objects, then the API browse endpoints probably do also need options for filtering a Post/Tag/User by the the exact title/name so that the full requested object can be fetched, but as those fields aren't unique it's going to be interesting to figure out a smart way to handle multiple results.

[ErisDS]

Going to close this as complete. The remaining work - to add tags to the search, is tracked by #5845.