feat(core): introduce Reshape API #647

francoischalifour · 2021-08-06T12:56:26Z

This PR introduces the Reshape API to Autocomplete. (The reshape preset package will come later.)

Live playground

▶️ Try the Reshape API in the sandbox →

Summary

A web UI is not a one-to-one mapping with your database, and a search UI doesn't have to be a one-to-one mapping with your search engine. The Reshape API unlocks this new search representation in your UI.

The Reshape API allows to apply transformations to a group of sources:

Apply a limit of items (or rows) for a group of sources (limit operation)
- Example: display 9 items total, including recent searches, query suggestions, categories and FAQ
Remove duplicates in a group of sources (uniqBy operation)
- Example: remove query suggestions which are already in recent searches (we currently have a dedicated API in the Recent Searches plugin for this)
Group by attribute of a group of sources (groupBy operation)
- Example: group items by category or by brand, as seen on DocSearch and on the Algolia Documentation website search
Sort sources (sort operation)
- Example: sort sources based on static or dynamic values like the search query (currently, users need to turn their sources into plugins if they want control over the order).

This API is a transformation that happens after the sources are resolved, as a last step before rendering them. It is an enabler to apply operations like Lodash/Ramda for Autocomplete experiences.

Preview

Example of reshape with a groupBy operation on the category:

Usage

import { autocomplete } from '@algolia/autocomplete-js';
import { pipe } from 'ramda';
import { groupBy, limit, uniqBy } from './functions'; // Will come from a preset package in the future

// You can pipe reshape functions
const combineSuggestions = pipe(
  uniqBy(({ source, item }) =>
    source.sourceId === 'querySuggestionsPlugin' ? item.query : item.label
  ),
  limit(4)
);
const groupByCategory = groupBy((hit) => hit.categories[0], { /* ... */ });

autocomplete({
  container: '#autocomplete',
  plugins: [recentSearchesPlugin, querySuggestionsPlugin, productsPlugin],
  reshape({ sourcesBySourceId }) {
    const {
      recentSearchesPlugin,
      querySuggestionsPlugin,
      products,
      ...rest
    } = sourcesBySourceId;

    return [
      combineSuggestions(recentSearchesPlugin, querySuggestionsPlugin),
      groupByCategory(products),
      Object.values(rest),
    ];
  },
});

Detailed design

The Reshape is a post-process step that happens after sources have been resolved in getSources(). During the getSources step, not all sources are resolved yet because some can be static (like a static list of items), and some can be async (like a search to Algolia).

During the Reshape phase, sources are static because they've been resolved. This allows simple items' manipulation.

(Combine is the old name of Reshape.)

Rolling out

Release the Reshape API in the next minor. This PR enables post-transformations of sources with the new reshape option, which we can introduce in the next minor version.
Release Reshape functions later. This PR doesn't include the @algolia/autocomplete-preset-reshape package that we'll expose yet. I'd like to try them user-land in a wide varieties of app before exposing them as APIs.
Write documentation in the next days. I need to add the new reshape param to the documentation, as well as write a guide to create custom reshape functions.

Sandbox	Source
@algolia/autocomplete-example-github-repositories-custom-plugin	Configuration
@algolia/autocomplete-example-instantsearch	Configuration
@algolia/autocomplete-example-playground	Configuration
@algolia/autocomplete-example-react-renderer	Configuration
@algolia/autocomplete-example-starter-algolia	Configuration
@algolia/autocomplete-example-starter	Configuration
@algolia/autocomplete-example-reshape	Configuration
@algolia/autocomplete-example-vue	Configuration
@algolia/autocomplete-example-reshape	PR

Haroenv

consistent with the other ones, but why does it have a readme if it's empty?

overall, clean implementation!

Haroenv · 2021-08-06T13:12:42Z

examples/reshape/app.tsx

+import { createQuerySuggestionsPlugin } from '@algolia/autocomplete-plugin-query-suggestions';
+import { createLocalStorageRecentSearchesPlugin } from '@algolia/autocomplete-plugin-recent-searches';
+import { h, Fragment } from 'preact';
+import { pipe } from 'ramda';


I like that it's not needed to have pipe builtin

Haroenv · 2021-08-06T13:13:20Z

examples/reshape/app.tsx

+  debug: true,
+  openOnFocus: true,
+  plugins: [recentSearchesPlugin, querySuggestionsPlugin, productsPlugin],
+  reshape({ sourcesBySourceId }) {


reasonable name!

Haroenv · 2021-08-06T13:13:43Z

examples/reshape/app.tsx

+  placeholder: 'Search',
+  debug: true,
+  openOnFocus: true,
+  plugins: [recentSearchesPlugin, querySuggestionsPlugin, productsPlugin],


why is products a plugin and not a regular source? could indicate more "regular" usage?

this gives me the idea if longer term plugins and sources maybe could be merged into a single argument? plugins could have no source, and it would allow you to order before using reshape already

why is products a plugin and not a regular source? could indicate more "regular" usage?

I wanted to focus the example on the reshape API, and not pollute the file with a huge source.

this gives me the idea if longer term plugins and sources maybe could be merged into a single argument? plugins could have no source, and it would allow you to order before using reshape already

You mean something like this?

autocomplete({ // ... getSources() { return [ recentSearchesPlugin, querySuggestionsPlugin, { // custom source } ] } })

I was more thinking of plugins: [somePlugin, { getSources() { customStuff }}, maybe that way you don't really need the callback form (although that limits some use cases for conditional requests, in which case your suggestion fits better)

I was just wondering if we really need to distinguish between the two

examples/reshape/functions/AutocompleteReshapeFunction.ts

packages/autocomplete-core/src/reshape.ts

examples/reshape/functions/uniqBy.ts

packages/autocomplete-core/src/__tests__/reshape.test.ts

Haroenv

this looks amazing!

shortcuts

Love it! Small suggestions but nothing blocking

examples/reshape/app.tsx

examples/reshape/functions/groupBy.ts

sarahdayan

Great job!

Nothing blocking, only a few remarks.

sarahdayan · 2021-08-25T19:07:21Z

examples/reshape/functions/uniqBy.ts

+  item: TItem;
+}) => TItem;
+
+export const uniqBy: AutocompleteReshapeFunction<UniqByPredicate<any>> = <


Since we've pulled Ramda already in this example, I'd use it there instead of rolling our own uniqBy.

I'm not really a fan of using ramda in the examples, I don't think it's so common, and might cause people to include it just because our example does so

Besides, it will not make it possible for them to copy/paste.

@francoischalifour You recommended that I go with Ramda's groupBy in the Tags plugin example though :) Also, we do use Ramda in the example for pipe. Why do you think this is okay for this specific function?

Anyway it's no biggie and not blocking. We can discuss that later.

examples/reshape/functions/limit.ts

sarahdayan · 2021-08-25T19:16:27Z

examples/reshape/functions/groupBy.ts

+  getSource(params: { name: string; items: TItem[] }): Partial<TSource>;
+};
+
+export const groupBy: AutocompleteReshapeFunction = <


Since we've pulled Ramda already in this example, I'd use it there instead of rolling our own groupBy.

See #647 (comment).

examples/reshape/app.tsx

examples/reshape/functions/normalizeReshapeSources.ts

packages/autocomplete-core/src/__tests__/reshape.test.ts

francoischalifour added 2 commits August 6, 2021 14:48

feat(core): introduce Reshape API

a8505cd

chore(docs): deploy reshape example to CodeSandbox

8e4ef0f

francoischalifour requested review from sarahdayan, Haroenv and shortcuts August 6, 2021 12:56

Haroenv reviewed Aug 6, 2021

View reviewed changes

francoischalifour added 2 commits August 16, 2021 10:31

test(core): update fixtures

12ae2db

docs(examples): use sets for uniqBy

c36da9c

francoischalifour requested a review from Haroenv August 16, 2021 08:38

Haroenv approved these changes Aug 16, 2021

View reviewed changes

shortcuts approved these changes Aug 16, 2021

View reviewed changes

examples/reshape/app.tsx Outdated Show resolved Hide resolved

examples/reshape/functions/groupBy.ts Outdated Show resolved Hide resolved

sarahdayan approved these changes Aug 25, 2021

View reviewed changes

francoischalifour added 6 commits August 26, 2021 10:49

refactor: use Object.entries

d197428

refactor: rename dedupeAndLimitSuggestions

13a1608

refactor: create sourceLimit variable

f61f432

test: update test names

74d5a4e

docs: add link to guide

785936e

docs: add Reshape params TSDoc

b6db429

francoischalifour merged commit d6180d2 into next Aug 26, 2021

francoischalifour deleted the feat/reshape-api branch August 26, 2021 14:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): introduce Reshape API #647

feat(core): introduce Reshape API #647

francoischalifour commented Aug 6, 2021 •

edited

Loading

codesandbox-ci bot commented Aug 6, 2021 •

edited

Loading

Haroenv left a comment

Haroenv Aug 6, 2021

Haroenv Aug 6, 2021

Haroenv Aug 6, 2021

Haroenv Aug 6, 2021

francoischalifour Aug 16, 2021

francoischalifour Aug 16, 2021

Haroenv Aug 16, 2021

Haroenv left a comment

shortcuts left a comment

sarahdayan left a comment

sarahdayan Aug 25, 2021

Haroenv Aug 26, 2021

francoischalifour Aug 26, 2021

sarahdayan Aug 26, 2021 •

edited

Loading

sarahdayan Aug 25, 2021

francoischalifour Aug 26, 2021

feat(core): introduce Reshape API #647

feat(core): introduce Reshape API #647

Conversation

francoischalifour commented Aug 6, 2021 • edited Loading

Live playground

Summary

Preview

Usage

Detailed design

Rolling out

Related

codesandbox-ci bot commented Aug 6, 2021 • edited Loading

Haroenv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Haroenv left a comment

Choose a reason for hiding this comment

shortcuts left a comment

Choose a reason for hiding this comment

sarahdayan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarahdayan Aug 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

francoischalifour commented Aug 6, 2021 •

edited

Loading

codesandbox-ci bot commented Aug 6, 2021 •

edited

Loading

sarahdayan Aug 26, 2021 •

edited

Loading