Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Branch] - major new UI redesign #2396

Merged
merged 164 commits into from
Jun 24, 2024
Merged

Conversation

manyoso
Copy link
Collaborator

@manyoso manyoso commented Jun 1, 2024

This PR is to track the development of the feature branch for major new UI redesign. This branch will included much more than a UI redesign though including extensive updates and changes to localdocs for example.

The current goal is for this branch is to be merged into main by June 24th. Commits and PR+commits will be merged into this feature branch and this PR will be used to track that progress.


🚀 This description was created by Ellipsis for commit 81d0c91

Summary:

Major UI redesign with new QML views and extensive updates to localdocs functionality and backend support.

Key points:

  • Major UI redesign with new QML views: HomeView.qml, AddCollectionView.qml, AddModelView.qml, ModelsView.qml, LocalDocsView.qml, SettingsView.qml
  • Updated existing QML views: ChatView.qml, ChatDrawer.qml, Theme.qml, ApplicationSettings.qml, ModelSettings.qml
  • Extensive updates to localdocs functionality in database.cpp, database.h, localdocs.cpp, localdocs.h
  • New theme properties and colors in Theme.qml
  • Backend updates to support new UI and functionalities in server.cpp, mysettings.cpp, mysettings.h

Generated with ❤️ by ellipsis.dev

manyoso added 10 commits May 30, 2024 15:59
…sending

mixpanel events. These changes added complexity to the Database code which
we're looking to simplify not increase complexity and all info that we care
about from mixpanel should be available from the GUI thread.

NOTE: This change will not compile by itself. It must be included with subsequent
changes made in PR #2302 but is here broken out into a standalone commit
to aid in review.

Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
…ut of date.

Signed-off-by: Adam Treat <treat.adam@gmail.com>
…eaking

change that requires re-indexing of all collections.

- Adds a general mechanism for handling breaking changes to localdocs database
  upgrades

- Upgrades the database format to version 2 and includes numerous changes to
  the underlying tables including:
    a) adds word and token count fields for chunk table
    b) adds a boolean field indicating whether the chunk has a generated embedding
    c) adds a last updated time field for collection table which can be displayed in GUI
    d) adds an embedding model field for collection table
    e) adds a boolean field for collection table to allow forced re-indexing

- Adds several statistics about the collection which can be displayed in GUI
    a) total documents
    b) total words
    c) total tokens
    d) last updated time
    e) which document is currently being indexed
    f) which document is currently being embedded
    g) which model is being used to generate embeddings for the collection

- Simplifies communication between the database thread and the GUI thread
  removing a lot of extra code and makes clear what methods/signals in database
  class are soley used for communicating state to the GUI

- Provides a pool for sending embedding requests to the embedding thread to
  increase speed of embedding when the chunks per document are small

- Provides a mechanism for resuming indexing/embedding when the app shutsdown
  or crashes in the middle of indexing/embedding

NOTE: This change is part of a larger set of commits, but can be built and
compiles up to this point

Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
follows the new FIGMA design made by Vincent. It also contains a lot of
changes to the way localdocs works.

This is now rebased on top of main and conflicts resolved. This is still
a monolithic commit that needs to be broken up into smaller pieces in order
to aid proper review. I'll be working on that now and force pushing as I
do so.

Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
@manyoso manyoso marked this pull request as draft June 3, 2024 14:25
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre

This comment was marked as outdated.

cebtenzzre and others added 12 commits June 5, 2024 17:46
Operations outside of scanQueue that remove embeddings are now more
atomic.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
This makes indexing over 10x faster by syncing the database to disk less
often.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
I didn't really understand the code that was here, so I replaced it with
something I know works. Key differences:
- Always open the latest database that has been *written*, not the
  latest one that exists on disk.
- Never remove databases. We don't have to do it to make this code work,
  and it's not easy to guarantee that the new db will be available next
  time before we remove the old one. Plus, we don't want to create a
  confusing situation if the user downgrades GPT4All (which we do
  support).

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
The `Q_ASSERT(dir.exists());` assertion in scanDocuments can fail if we
`rm -rf` a localdocs collection. Since a folder can be removed at any
time, asserting that it exists is fundamentally flawed.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Calling addFolder for an existing folder works, but it is confusing at
best. All we actually intend to do is add it to the watch and scan it,
so just do that explicitly.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
We were not actually watching subdirectories, because the QDirIterator
was missing QDir::Dirs.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
manyoso and others added 17 commits June 24, 2024 18:38
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Now we show the release notes link instead of using word 'changelog' and
the release notes are also shown in a dialog when you update to a new version.

Signed-off-by: Adam Treat <treat.adam@gmail.com>
Pinning this label to the top is confusing, because you may be scrolled
into another category, such as "Advanced."

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
This newly added function was returning duplicates and missing a call to
`q.next()`, so it was actually returning chunk IDs.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
This button allows the user to rebuild a collection from scratch without
having to remove and re-add the folder, in case that is ever necessary.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
This fixes progress display never reaching 100% when deleting a document
from a collection while it is being embedded.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Otherwise, empty folders show "INSTALLING" forever.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
There is already a global setting for n_threads. Use it for embeddings
models too.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre marked this pull request as ready for review June 24, 2024 22:48
Copy link

ellipsis-dev bot commented Jun 24, 2024

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev


Generated with ❤️ by ellipsis.dev

@cebtenzzre cebtenzzre merged commit 9273b49 into main Jun 24, 2024
7 of 22 checks passed
cebtenzzre added a commit that referenced this pull request Jun 25, 2024
This fixes a regression in PR #2396.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
manyoso pushed a commit that referenced this pull request Jun 25, 2024
This fixes a regression in PR #2396.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants