Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/create correspondents #41

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+435 −119
Open

Feat/create correspondents #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5dadbcb
feat: improve auto throughput & logging
JonasHess Oct 31, 2024
16281be
feat: auto create missing correspondents
JonasHess Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 15 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ services:
OLLAMA_HOST: 'http://host.docker.internal:11434' # If using Ollama
VISION_LLM_PROVIDER: 'ollama' # Optional, for OCR
VISION_LLM_MODEL: 'minicpm-v' # Optional, for OCR
LOG_LEVEL: 'info' # Optional or 'debug', 'warn', 'error'
volumes:
- ./prompts:/app/prompts # Mount the prompts directory
ports:
Expand Down Expand Up @@ -122,6 +123,7 @@ If you prefer to run the application manually:
-e LLM_LANGUAGE='English' \
-e VISION_LLM_PROVIDER='ollama' \
-e VISION_LLM_MODEL='minicpm-v' \
-e LOG_LEVEL='info' \
-v $(pwd)/prompts:/app/prompts \ # Mount the prompts directory
-p 8080:8080 \
paperless-gpt
Expand All @@ -131,17 +133,19 @@ If you prefer to run the application manually:

### Environment Variables

| Variable | Description | Required |
|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
| `PAPERLESS_BASE_URL` | The base URL of your paperless-ngx instance (e.g., `http://paperless-ngx:8000`). | Yes |
| `PAPERLESS_API_TOKEN` | API token for accessing paperless-ngx. You can generate one in the paperless-ngx admin interface. | Yes |
| `LLM_PROVIDER` | The LLM provider to use (`openai` or `ollama`). | Yes |
| `LLM_MODEL` | The model name to use (e.g., `gpt-4o`, `gpt-3.5-turbo`, `llama2`). | Yes |
| `OPENAI_API_KEY` | Your OpenAI API key. Required if using OpenAI as the LLM provider. | Cond. |
| `LLM_LANGUAGE` | The likely language of your documents (e.g., `English`, `German`). Default is `English`. | No |
| `OLLAMA_HOST` | The URL of the Ollama server (e.g., `http://host.docker.internal:11434`). Useful if using Ollama. Default is `http://127.0.0.1:11434`. | No |
| `VISION_LLM_PROVIDER` | The vision LLM provider to use for OCR (`openai` or `ollama`). | No |
| `VISION_LLM_MODEL` | The model name to use for OCR (e.g., `minicpm-v`). | No |
| Variable | Description | Required |
|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------|----------|
| `PAPERLESS_BASE_URL` | The base URL of your paperless-ngx instance (e.g., `http://paperless-ngx:8000`). | Yes |
| `PAPERLESS_API_TOKEN` | API token for accessing paperless-ngx. You can generate one in the paperless-ngx admin interface. | Yes |
| `LLM_PROVIDER` | The LLM provider to use (`openai` or `ollama`). | Yes |
| `LLM_MODEL` | The model name to use (e.g., `gpt-4o`, `gpt-3.5-turbo`, `llama2`). | Yes |
| `OPENAI_API_KEY` | Your OpenAI API key. Required if using OpenAI as the LLM provider. | Cond. |
| `LLM_LANGUAGE` | The likely language of your documents (e.g., `English`, `German`). Default is `English`. | No |
| `OLLAMA_HOST` | The URL of the Ollama server (e.g., `http://host.docker.internal:11434`). Useful if using Ollama. Default is `http://127.0.0.1:11434`. | No |
| `VISION_LLM_PROVIDER` | The vision LLM provider to use for OCR (`openai` or `ollama`). | No |
| `VISION_LLM_MODEL` | The model name to use for OCR (e.g., `minicpm-v`). | No |
| `LOG_LEVEL` | The log level for the application (`info`, `debug`, `warn`, `error`). Default is `info`. | No |
| `CORRESPONDENT_BLACK_LIST` | A comma-separated list of names to exclude from the correspondents suggestions. Example: `John Doe, Jane Smith`. | No |

**Note:** When using Ollama, ensure that the Ollama server is running and accessible from the paperless-gpt container.

Expand Down
19 changes: 9 additions & 10 deletions app_http_handlers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package main

import (
"fmt"
"log"
"net/http"
"os"
"strconv"
Expand Down Expand Up @@ -59,7 +58,7 @@ func updatePromptsHandler(c *gin.Context) {
titleTemplate = t
err = os.WriteFile("prompts/title_prompt.tmpl", []byte(req.TitleTemplate), 0644)
if err != nil {
log.Printf("Failed to write title_prompt.tmpl: %v", err)
log.Errorf("Failed to write title_prompt.tmpl: %v", err)
}
}

Expand All @@ -73,7 +72,7 @@ func updatePromptsHandler(c *gin.Context) {
tagTemplate = t
err = os.WriteFile("prompts/tag_prompt.tmpl", []byte(req.TagTemplate), 0644)
if err != nil {
log.Printf("Failed to write tag_prompt.tmpl: %v", err)
log.Errorf("Failed to write tag_prompt.tmpl: %v", err)
}
}

Expand All @@ -87,7 +86,7 @@ func (app *App) getAllTagsHandler(c *gin.Context) {
tags, err := app.Client.GetAllTags(ctx)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error fetching tags: %v", err)})
log.Printf("Error fetching tags: %v", err)
log.Errorf("Error fetching tags: %v", err)
return
}

Expand All @@ -98,10 +97,10 @@ func (app *App) getAllTagsHandler(c *gin.Context) {
func (app *App) documentsHandler(c *gin.Context) {
ctx := c.Request.Context()

documents, err := app.Client.GetDocumentsByTags(ctx, []string{manualTag})
documents, err := app.Client.GetDocumentsByTags(ctx, []string{manualTag}, 25)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error fetching documents: %v", err)})
log.Printf("Error fetching documents: %v", err)
log.Errorf("Error fetching documents: %v", err)
return
}

Expand All @@ -115,14 +114,14 @@ func (app *App) generateSuggestionsHandler(c *gin.Context) {
var suggestionRequest GenerateSuggestionsRequest
if err := c.ShouldBindJSON(&suggestionRequest); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("Invalid request payload: %v", err)})
log.Printf("Invalid request payload: %v", err)
log.Errorf("Invalid request payload: %v", err)
return
}

results, err := app.generateDocumentSuggestions(ctx, suggestionRequest)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error processing documents: %v", err)})
log.Printf("Error processing documents: %v", err)
log.Errorf("Error processing documents: %v", err)
return
}

Expand All @@ -135,14 +134,14 @@ func (app *App) updateDocumentsHandler(c *gin.Context) {
var documents []DocumentSuggestion
if err := c.ShouldBindJSON(&documents); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("Invalid request payload: %v", err)})
log.Printf("Invalid request payload: %v", err)
log.Errorf("Invalid request payload: %v", err)
return
}

err := app.Client.UpdateDocuments(ctx, documents)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error updating documents: %v", err)})
log.Printf("Error updating documents: %v", err)
log.Errorf("Error updating documents: %v", err)
return
}

Expand Down
85 changes: 80 additions & 5 deletions app_llm.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,52 @@ import (
"bytes"
"context"
"fmt"
"log"
"strings"
"sync"

"github.com/tmc/langchaingo/llms"
)

// getSuggestedCorrespondent generates a suggested correspondent for a document using the LLM
func (app *App) getSuggestedCorrespondent(ctx context.Context, content string, suggestedTitle string, availableCorrespondents []string, correspondentBlackList []string) (string, error) {
likelyLanguage := getLikelyLanguage()

templateMutex.RLock()
defer templateMutex.RUnlock()

var promptBuffer bytes.Buffer
err := correspondentTemplate.Execute(&promptBuffer, map[string]interface{}{
"Language": likelyLanguage,
"AvailableCorrespondents": availableCorrespondents,
"BlackList": correspondentBlackList,
"Title": suggestedTitle,
"Content": content,
})
if err != nil {
return "", fmt.Errorf("error executing correspondent template: %v", err)
}

prompt := promptBuffer.String()
log.Debugf("Correspondent suggestion prompt: %s", prompt)

completion, err := app.LLM.GenerateContent(ctx, []llms.MessageContent{
{
Parts: []llms.ContentPart{
llms.TextContent{
Text: prompt,
},
},
Role: llms.ChatMessageTypeHuman,
},
})
if err != nil {
return "", fmt.Errorf("error getting response from LLM: %v", err)
}

response := strings.TrimSpace(completion.Choices[0].Content)
return response, nil
}

// getSuggestedTags generates suggested tags for a document using the LLM
func (app *App) getSuggestedTags(ctx context.Context, content string, suggestedTitle string, availableTags []string) ([]string, error) {
likelyLanguage := getLikelyLanguage()
Expand All @@ -30,7 +69,7 @@ func (app *App) getSuggestedTags(ctx context.Context, content string, suggestedT
}

prompt := promptBuffer.String()
log.Printf("Tag suggestion prompt: %s", prompt)
log.Debugf("Tag suggestion prompt: %s", prompt)

completion, err := app.LLM.GenerateContent(ctx, []llms.MessageContent{
{
Expand Down Expand Up @@ -119,7 +158,7 @@ func (app *App) getSuggestedTitle(ctx context.Context, content string) (string,

prompt := promptBuffer.String()

log.Printf("Title suggestion prompt: %s", prompt)
log.Debugf("Title suggestion prompt: %s", prompt)

completion, err := app.LLM.GenerateContent(ctx, []llms.MessageContent{
{
Expand Down Expand Up @@ -155,6 +194,18 @@ func (app *App) generateDocumentSuggestions(ctx context.Context, suggestionReque
availableTagNames = append(availableTagNames, tagName)
}

// Prepare a list of document correspodents
availableCorrespondentsMap, err := app.Client.GetAllCorrespondents(ctx)
if err != nil {
return nil, fmt.Errorf("failed to fetch available correspondents: %v", err)
}

// Prepare a list of correspondent names
availableCorrespondentNames := make([]string, 0, len(availableCorrespondentsMap))
for correspondentName := range availableCorrespondentsMap {
availableCorrespondentNames = append(availableCorrespondentNames, correspondentName)
}

documents := suggestionRequest.Documents
documentSuggestions := []DocumentSuggestion{}

Expand All @@ -176,14 +227,15 @@ func (app *App) generateDocumentSuggestions(ctx context.Context, suggestionReque

var suggestedTitle string
var suggestedTags []string
var suggestedCorrespondent string

if suggestionRequest.GenerateTitles {
suggestedTitle, err = app.getSuggestedTitle(ctx, content)
if err != nil {
mu.Lock()
errorsList = append(errorsList, fmt.Errorf("Document %d: %v", documentID, err))
mu.Unlock()
log.Printf("Error processing document %d: %v", documentID, err)
log.Errorf("Error processing document %d: %v", documentID, err)
return
}
}
Expand All @@ -194,29 +246,52 @@ func (app *App) generateDocumentSuggestions(ctx context.Context, suggestionReque
mu.Lock()
errorsList = append(errorsList, fmt.Errorf("Document %d: %v", documentID, err))
mu.Unlock()
log.Printf("Error generating tags for document %d: %v", documentID, err)
log.Errorf("Error generating tags for document %d: %v", documentID, err)
return
}
}

if suggestionRequest.GenerateCorrespondents {
suggestedCorrespondent, err = app.getSuggestedCorrespondent(ctx, content, suggestedTitle, availableCorrespondentNames, correspondentBlackList)
if err != nil {
mu.Lock()
errorsList = append(errorsList, fmt.Errorf("Document %d: %v", documentID, err))
mu.Unlock()
log.Errorf("Error generating correspondents for document %d: %v", documentID, err)
return
}

}

mu.Lock()
suggestion := DocumentSuggestion{
ID: documentID,
OriginalDocument: doc,
}
// Titles
if suggestionRequest.GenerateTitles {
log.Printf("Suggested title for document %d: %s", documentID, suggestedTitle)
suggestion.SuggestedTitle = suggestedTitle
} else {
suggestion.SuggestedTitle = doc.Title
}

// Tags
if suggestionRequest.GenerateTags {
log.Printf("Suggested tags for document %d: %v", documentID, suggestedTags)
suggestion.SuggestedTags = suggestedTags
} else {
suggestion.SuggestedTags = removeTagFromList(doc.Tags, manualTag)
}

// Correspondents
if suggestionRequest.GenerateCorrespondents {
log.Printf("Suggested correspondent for document %d: %s", documentID, suggestedCorrespondent)
suggestion.SuggestedCorrespondent = suggestedCorrespondent
} else {
suggestion.SuggestedCorrespondent = ""
}

documentSuggestions = append(documentSuggestions, suggestion)
mu.Unlock()
log.Printf("Document %d processed successfully.", documentID)
Expand Down
3 changes: 2 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ require (
github.com/Masterminds/sprig/v3 v3.2.3
github.com/gen2brain/go-fitz v1.24.14
github.com/gin-gonic/gin v1.10.0
github.com/google/uuid v1.6.0
github.com/sirupsen/logrus v1.9.3
github.com/stretchr/testify v1.9.0
github.com/tmc/langchaingo v0.1.12
golang.org/x/sync v0.7.0
Expand All @@ -29,7 +31,6 @@ require (
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.20.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/huandu/xstrings v1.3.3 // indirect
github.com/imdario/mergo v0.3.13 // indirect
github.com/json-iterator/go v1.1.12 // indirect
Expand Down
3 changes: 3 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/spf13/cast v1.3.1 h1:nFm6S0SMdyzrzcmThSipiEubIDy8WEXKNZ0UOgiRpng=
github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
Expand Down Expand Up @@ -123,6 +125,7 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
Expand Down
Loading
Loading