Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful Queues: Issue Indexing and Tasks #9363

Merged
merged 38 commits into from
Jan 7, 2020
Merged
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
8504263
Queue: Add generic graceful queues with settings
zeripath Nov 13, 2019
9fb0516
Queue & Setting: Add worker pool implementation
zeripath Nov 20, 2019
0edb70a
Queue: Add worker settings
zeripath Dec 7, 2019
e6ebb47
Queue: Make resizing worker pools
zeripath Dec 7, 2019
85d1a7f
Queue: Add name variable to queues
zeripath Dec 7, 2019
2927bc6
Queue: Add monitoring
zeripath Dec 7, 2019
9ad9070
Queue: Improve logging
zeripath Dec 11, 2019
d6b5404
Issues: Gracefulise the issues indexer
zeripath Nov 15, 2019
4658b2f
Task: Move to generic queue and gracefulise
zeripath Nov 16, 2019
cc123c3
Issues: Standardise the issues indexer queue settings
zeripath Dec 16, 2019
1013ced
Fix test
zeripath Dec 16, 2019
1fb9104
Queue: Allow Redis to connect to unix
zeripath Dec 17, 2019
a492b30
Prevent deadlock during early shutdown of issue indexer
zeripath Dec 23, 2019
b1c9fa7
Add MaxWorker settings to queues
zeripath Dec 30, 2019
4d8b8ed
Merge branch 'master' into graceful-queues
zeripath Dec 30, 2019
a763cca
Update modules/indexer/issues/indexer.go
zeripath Dec 31, 2019
632757b
Update modules/indexer/issues/indexer.go
zeripath Dec 31, 2019
6c6d3ea
Update modules/queue/queue_channel.go
zeripath Dec 31, 2019
402e4df
Update modules/queue/queue_disk.go
zeripath Dec 31, 2019
a10129f
Update modules/queue/queue_disk_channel.go
zeripath Dec 31, 2019
6306cd4
Rename queue.Description to queue.ManagedQueue as per @guillep2k
zeripath Dec 31, 2019
8798a61
Cancel pool workers when removed
zeripath Dec 31, 2019
030b6d9
Remove dependency on queue from setting
zeripath Dec 31, 2019
9941ae2
Update modules/queue/queue_redis.go
zeripath Dec 31, 2019
e852cb6
As per @guillep2k add mutex locks on shutdown/terminate
zeripath Jan 4, 2020
21b9778
move unlocking out of setInternal
zeripath Jan 4, 2020
1cb7a86
Add warning if number of workers < 0
zeripath Jan 4, 2020
e4ddaab
Small changes as per @guillep2k
zeripath Jan 4, 2020
c7550a4
No redis host specified not found
zeripath Jan 4, 2020
90eeb14
Clean up documentation for queues
zeripath Jan 4, 2020
8649dfa
Update docs/content/doc/advanced/config-cheat-sheet.en-us.md
zeripath Jan 4, 2020
ec83b83
Merge branch 'master' into graceful-queues
lafriks Jan 4, 2020
bde0cf2
Merge branch 'master' into graceful-queues
techknowlogick Jan 5, 2020
99a6f48
Update modules/indexer/issues/indexer_test.go
zeripath Jan 6, 2020
52ee212
Ensure that persistable channel queue is added to manager
zeripath Jan 6, 2020
a74b473
Merge branch 'master' into graceful-queues
lunny Jan 7, 2020
1f83b4f
Rename QUEUE_NAME REDIS_QUEUE_NAME
zeripath Jan 7, 2020
0c345db
Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME"
zeripath Jan 7, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions custom/conf/app.ini.sample
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,39 @@ REPO_INDEXER_INCLUDE =
; A comma separated list of glob patterns to exclude from the index; ; default is empty
REPO_INDEXER_EXCLUDE =

[queue]
; Specific queues can be individually configured with [queue.name]. [queue] provides defaults
;
; General queue queue type, currently support: persistable-channel, channel, level, redis, dummy
; default to persistable-channel
TYPE = persistable-channel
; data-dir for storing persistable queues and level queues, individual queues will be named by their type
DATADIR = queues/
; Default queue length before a channel queue will block
LENGTH = 20
; Batch size to send for batched queues
BATCH_LENGTH = 20
; Connection string for redis queues this will store the redis connection string.
CONN_STR = "addrs=127.0.0.1:6379 db=0"
; Provide the suffix of the default redis queue name - specific queues can be overriden within in their [queue.name] sections.
QUEUE_NAME = "_queue"
; If the queue cannot be created at startup - level queues may need a timeout at startup - wrap the queue:
WRAP_IF_NECESSARY = true
; Attempt to create the wrapped queue at max
MAX_ATTEMPTS = 10
; Timeout queue creation
TIMEOUT = 15m30s
; Create a pool with this many workers
WORKERS = 1
; Dynamically scale the worker pool to at this many workers
MAX_WORKERS = 10
; Add boost workers when the queue blocks for BLOCK_TIMEOUT
BLOCK_TIMEOUT = 1s
; Remove the boost workers after BOOST_TIMEOUT
BOOST_TIMEOUT = 5m
; During a boost add BOOST_WORKERS
BOOST_WORKERS = 5

[admin]
; Disallow regular (non-admin) users from creating organizations.
DISABLE_REGULAR_ORG_CREATION = false
Expand Down
20 changes: 20 additions & 0 deletions docs/content/doc/advanced/config-cheat-sheet.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,7 @@ relation to port exhaustion.

- `ISSUE_INDEXER_TYPE`: **bleve**: Issue indexer type, currently support: bleve or db, if it's db, below issue indexer item will be invalid.
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: Index file used for issue search.
- The next 4 configuration values are deprecated and should be set in `queue.issue_indexer` however are kept for backwards compatibility:
- `ISSUE_INDEXER_QUEUE_TYPE`: **levelqueue**: Issue indexer queue, currently supports:`channel`, `levelqueue`, `redis`.
- `ISSUE_INDEXER_QUEUE_DIR`: **indexers/issues.queue**: When `ISSUE_INDEXER_QUEUE_TYPE` is `levelqueue`, this will be the queue will be saved path.
- `ISSUE_INDEXER_QUEUE_CONN_STR`: **addrs=127.0.0.1:6379 db=0**: When `ISSUE_INDEXER_QUEUE_TYPE` is `redis`, this will store the redis connection string.
Expand All @@ -239,6 +240,24 @@ relation to port exhaustion.
- `MAX_FILE_SIZE`: **1048576**: Maximum size in bytes of files to be indexed.
- `STARTUP_TIMEOUT`: **30s**: If the indexer takes longer than this timeout to start - fail. (This timeout will be added to the hammer time above for child processes - as bleve will not start until the previous parent is shutdown.) Set to zero to never timeout.

## Queue (`queue` and `queue.*`)

- `TYPE`: **persistable-channel**: General queue type, currently support: `persistable-channel`, `channel`, `level`, `redis`, `dummy`
- `DATADIR`: **queues/**: Base DataDir for storing persistent and level queues. `DATADIR` for inidividual queues can be set in `queue.name` sections but will default to `DATADIR/`**`name`**.
- `LENGTH`: **20**: Maximal queue size before channel queues block
- `BATCH_LENGTH`: **20**: Batch data before passing to the handler
- `CONN_STR`: **addrs=127.0.0.1:6379 db=0**: Connection string for the redis queue type.
- `QUEUE_NAME`: **_queue**: The suffix for default redis queue name. Individual queues will default to **`name`**`QUEUE_NAME` but can be overriden in the specific `queue.name` section.
- `WRAP_IF_NECESSARY`: **true**: Will wrap queues with a timeoutable queue if the selected queue is not ready to be created - (Only relevant for the level queue.)
- `MAX_ATTEMPTS`: **10**: Maximum number of attempts to create the wrapped queue
- `TIMEOUT`: **GRACEFUL_HAMMER_TIME + 30s**: Timeout the creation of the wrapped queue if it takes longer than this to create.
- Queues by default come with a dynamically scaling worker pool. The following settings configure this:
- `WORKERS`: **1**: Number of initial workers for the queue.
- `MAX_WORKERS`: **10**: Maximum number of worker go-routines for the queue.
- `BLOCK_TIMEOUT`: **1s**: If the queue blocks for this time, boost the number of workers - the `BLOCK_TIMEOUT` will then be doubled before boosting again whilst the boost is ongoing.
- `BOOST_TIMEOUT`: **5m**: Boost workers will timeout after this long.
- `BOOST_WORKERS`: **5**: This many workers will be added to the worker pool if there is a boost.

## Admin (`admin`)
- `DEFAULT_EMAIL_NOTIFICATIONS`: **enabled**: Default configuration for email notifications for users (user configurable). Options: enabled, onmention, disabled

Expand Down Expand Up @@ -614,6 +633,7 @@ You may redefine `ELEMENT`, `ALLOW_ATTR`, and `REGEXP` multiple times; each time

## Task (`task`)

- Task queue configuration has been moved to `queue.task` however, the below configuration values are kept for backwards compatibilityx:
- `QUEUE_TYPE`: **channel**: Task queue type, could be `channel` or `redis`.
- `QUEUE_LENGTH`: **1000**: Task queue length, available only when `QUEUE_TYPE` is `channel`.
- `QUEUE_CONN_STR`: **addrs=127.0.0.1:6379 db=0**: Task queue connection string, available only when `QUEUE_TYPE` is `redis`. If there redis needs a password, use `addrs=127.0.0.1:6379 password=123 db=0`.
Expand Down
9 changes: 8 additions & 1 deletion integrations/issue_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ import (
"strconv"
"strings"
"testing"
"time"

"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/indexer/issues"
"code.gitea.io/gitea/modules/references"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/test"
Expand Down Expand Up @@ -87,7 +89,12 @@ func TestViewIssuesKeyword(t *testing.T) {
defer prepareTestEnv(t)()

repo := models.AssertExistsAndLoadBean(t, &models.Repository{ID: 1}).(*models.Repository)

issue := models.AssertExistsAndLoadBean(t, &models.Issue{
RepoID: repo.ID,
Index: 1,
}).(*models.Issue)
issues.UpdateIssueIndexer(issue)
time.Sleep(time.Second * 1)
lafriks marked this conversation as resolved.
Show resolved Hide resolved
const keyword = "first"
req := NewRequestf(t, "GET", "%s/issues?q=%s", repo.RelLink(), keyword)
resp := MakeRequest(t, req, http.StatusOK)
Expand Down
4 changes: 4 additions & 0 deletions modules/indexer/issues/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ func (db *DBIndexer) Delete(ids ...int64) error {
return nil
}

// Close dummy function
func (db *DBIndexer) Close() {
}

// Search dummy function
func (db *DBIndexer) Search(kw string, repoIDs []int64, limit, start int) (*SearchResult, error) {
total, ids, err := models.SearchIssueIDsByKeyword(kw, repoIDs, limit, start)
Expand Down
Loading