Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve sync performance for pull-mirrors #19125

Merged
merged 25 commits into from
Mar 31, 2022
Merged

Improve sync performance for pull-mirrors #19125

merged 25 commits into from
Mar 31, 2022

Commits on Mar 31, 2022

  1. optimize tag-release sync procedure for pull-mirrors 

    For large repositories with many tags, SyncReleasesWithTags can be a costly
operation (taking several minutes to complete). The reason is two-fold

1. on sync, every upstream repo tag is compared (for changes) against existing
local entries in the release table to ensure that they are up-to-date.

2. the procedure for getting each tag involves several git operations

     git show-ref --tags -- v8.2.4477
     git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f

  of which the 'git rev-list --count' can be particularly heavy.

This commit optimizes performance for pull-mirrors. We utilize the fact that a
pull-mirror is always identical to its upstream and rebuild the entire release
table on every sync and use a batch 'git for-each-ref .. refs/tags' call to
retrieve all tags in one go.

For large mirror repos, with hundreds of annotated tags, this brings down the
duration of the sync operation from several minutes to a few seconds.

Signed-off-by: Peter Gardfjäll <peter.gardfjall.work@gmail.com>
    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    aa3a762 View commit details
    Browse the repository at this point in the history

  2. optimize tag-release sync procedure for pull-mirrors 

    For large repositories with many tags, SyncReleasesWithTags can be a costly
operation (taking several minutes to complete). The reason is two-fold

1. on sync, every upstream repo tag is compared (for changes) against existing
local entries in the release table to ensure that they are up-to-date.

2. the procedure for getting each tag involves several git operations

     git show-ref --tags -- v8.2.4477
     git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f

  of which the 'git rev-list --count' can be particularly heavy.

This commit optimizes performance for pull-mirrors. We utilize the fact that a
pull-mirror is always identical to its upstream and rebuild the entire release
table on every sync and use a batch 'git for-each-ref .. refs/tags' call to
retrieve all tags in one go.

For large mirror repos, with hundreds of annotated tags, this brings down the
duration of the sync operation from several minutes to a few seconds.

Signed-off-by: Peter Gardfjäll <peter.gardfjall.work@gmail.com>
    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    33f5a5e View commit details
    Browse the repository at this point in the history

  3. godoc

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    5cd3114 View commit details
    Browse the repository at this point in the history

  4. please linter

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    4ce50be View commit details
    Browse the repository at this point in the history

  5. please linter: copyright notices

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    d87be92 View commit details
    Browse the repository at this point in the history

  6. test foreachref.Format

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    0e67646 View commit details
    Browse the repository at this point in the history

  7. test foreachref.Parser

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    1fc0bb4 View commit details
    Browse the repository at this point in the history

  8. stream git for-each-ref output to Parser

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    6d30da1 View commit details
    Browse the repository at this point in the history

  9. copyright header for test files

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    5c91e3d View commit details
    Browse the repository at this point in the history

  10. gofumpt

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    2847d13 View commit details
    Browse the repository at this point in the history

  11. explicitly ignore return value

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    5d6c962 View commit details
    Browse the repository at this point in the history

  12. explicitly ignore return value (again)

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    73bb43e View commit details
    Browse the repository at this point in the history

  13. please linter: no encoding/json

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    4274804 View commit details
    Browse the repository at this point in the history

  14. rename foreachref.Parser test function

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    638d690 View commit details
    Browse the repository at this point in the history

  15. distinguish external imports

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    38a1b66 View commit details
    Browse the repository at this point in the history

  16. distinguish external imports part 2

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    20f6dc5 View commit details
    Browse the repository at this point in the history

  17. sort prior to pagination

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    a5220cf View commit details
    Browse the repository at this point in the history

  18. include payload for annotated tags with signature

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    fcd1de6 View commit details
    Browse the repository at this point in the history

  19. less verbose signature payload construction

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    ef6352c View commit details
    Browse the repository at this point in the history

  20. code cleanup: remove dead code

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    2f3908c View commit details
    Browse the repository at this point in the history

  21. delimiters can be byte slices

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    aa267d2 View commit details
    Browse the repository at this point in the history

  22. avoid closing writer twice

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    61eae9e View commit details
    Browse the repository at this point in the history

  23. avoid repetitive memory allocations

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    fd7f58c View commit details
    Browse the repository at this point in the history

  24. make sure tests add newline to every reference in simulated output

    @petergardfjall
    petergardfjall committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    c1da4e0 View commit details
    Browse the repository at this point in the history

  25. Merge branch 'main' into pull-mirror-tag-sync-optimization

    @wxiaoguang
    wxiaoguang authored Mar 31, 2022
    Configuration menu
    Copy the full SHA
    b5ef68f View commit details
    Browse the repository at this point in the history