Torus

Torus is a plug-and-play Elixir library that seamlessly integrates PostgreSQL's search into Ecto, streamlining the construction of advanced search queries.

Usage

The package can be installed by adding torus to your list of dependencies in mix.exs:

def deps do
  [
    {:torus, "~> 0.2"}
  ]
end

Then, in any query, you can (for example) add a full-text search:

Post
|> Torus.full_text_dynamic([p], [p.title, p.body], "uncovered hogwarts")
|> Repo.all()

See full_text_dynamic/5 for more details.

6 types of search:

1. Pattern matching: Searches for a specific pattern in a string.

   iex> insert_posts!(["Wand", "Magic wand", "Owl"])
   ...> Post
   ...> |> Torus.ilike([p], [p.title], "wan%")
   ...> |> select([p], p.title)
   ...> |> Repo.all()
   ["Wand"]

   See like/5, ilike/5, and similar_to/5 for more details.

2. Similarity: Searches for items that are closely alike based on attributes, often using measures like cosine similarity or Euclidean distance. Is great for fuzzy searching and ignoring typos in short texts.

   iex> insert_posts!(["Hogwarts Secrets", "Quidditch Fever", "Hogwart’s Secret"])
   ...> Post
   ...> |> Torus.similarity([p], [p.title], "hoggwarrds", limit: 2)
   ...> |> select([p], p.title)
   ...> |> Repo.all()
   ["Hogwarts Secrets", "Hogwart’s Secret"]

   See similarity/5 for more details.

3. Text Search Vectors: Uses term-document matrix vectors for full-text search, enabling efficient querying and ranking based on term frequency. - PostgreSQL: Documentation: 17: Chapter 12. Full Text Search. Is great for large datasets to quickly return relevant results.

      iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
      ...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
      ...> insert_post!(title: "Completely unrelated", body: "No magic here!")
      ...>  Post
      ...> |> Torus.full_text_dynamic([p], [p.title, p.body], "uncov hogwar")
      ...> |> select([p], p.title)
      ...> |> Repo.all()
      ["Diagon Bombshell"]

   See full_text_dynamic/5 for more details.

4. Semantic Search: Understands the contextual meaning of queries to match and retrieve related content, often utilizing natural language processing. Semantic Search with PostgreSQL and OpenAI Embeddings

   Will be added soon.

5. Hybrid Search: Combines multiple search techniques (e.g., keyword and semantic) to leverage their strengths for more accurate results.

   Will be added soon.

6. 3rd Party Engines/Providers: Utilizes external services or software specifically designed for optimized and scalable search capabilities, such as Elasticsearch or Algolia.

Torus support

For now, Torus supports pattern match, similarity, and full-text search, with plans to expand support further. These docs will be updated with more examples on which search type to choose and how to make them more performant (by adding indexes or using specific functions).

Debugging your queries

Torus offers a few helpers to debug, explain, and analyze your queries before using them on production. See Torus.QueryInspector for more details.

Future plans

• Implement more search types and functions from PostgreSQL docs, provide examples and docs for them
• Make full_text_dynamic/5 more extensible by splitting it to building blocks and defining more arguments. Leave the default (without args) version fit for most cases.
• Add full_text_stored/5 for full-text search on stored vector columns
• Add support for highlighting search results. (Base off of a ts_headline function)
• Create a clean API for semantic search, make it easy to abstract embedding creation and storage

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/torus.