Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a storage layer for attachments #11387

Merged
merged 59 commits into from
Aug 18, 2020
Merged

Add a storage layer for attachments #11387

merged 59 commits into from
Aug 18, 2020

Conversation

lunny
Copy link
Member

@lunny lunny commented May 12, 2020

  • This PR introduced a storage abstract layer to store attachments in local disk or minio(via s3 protocol). It's inspired by NOT MERGED: Add bucket storage support #9567 but with a smaller interface to handle the object storages. According minio's document, it should support to access any Amazon S3 compatible object storage.(https://docs.min.io/docs/golang-client-quickstart-guide.html)
  • Changed mysql intergration tests to store attachments to a minio docker.
  • It also introduced a sub command to migrate attachments to a new sotrage placement.

Below are the steps to migrate your attachments to a minio service.

  • First, migrate local attachments to a minio server
./gitea migrate-storage --type=attachments --store=minio --minio-endpoint=localhost:9000 --minio-access-key-id=123456 --minio-secret-access-key=987654321 --minio-bucket=gitea --minio-location=us-east-1 --minio-base-path=attachments/ --minio-use-ssl=false
  • Second, change your app.ini section attachment to change STORE_TYPE as minio and other information like above.

  • Last, restart gitea. It's done.

Next steps (On other PRs):

@lunny lunny added type/feature Completely new functionality. Can only be merged if feature freeze is not active. pr/wip This PR is not ready for review labels May 12, 2020
@lunny lunny added this to the 1.13.0 milestone May 12, 2020
Copy link
Member

@techknowlogick techknowlogick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few minor nits

cmd/migrate_storage.go Outdated Show resolved Hide resolved
models/admin.go Outdated Show resolved Hide resolved
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label May 12, 2020
@lunny lunny force-pushed the lunny/bucket branch 3 times, most recently from cb5249f to 2281f93 Compare May 17, 2020 17:08
@lunny lunny changed the title WIP: Add a storage layer for attachments Add a storage layer for attachments May 17, 2020
@lunny lunny removed the pr/wip This PR is not ready for review label May 17, 2020
@lunny
Copy link
Member Author

lunny commented May 17, 2020

It's ready to review now.

cmd/migrate_storage.go Outdated Show resolved Hide resolved
cmd/migrate_storage.go Outdated Show resolved Hide resolved
cmd/migrate_storage.go Outdated Show resolved Hide resolved
cmd/migrate_storage.go Outdated Show resolved Hide resolved
cmd/migrate_storage.go Outdated Show resolved Hide resolved
models/attachment.go Outdated Show resolved Hide resolved
models/migrations/v96.go Outdated Show resolved Hide resolved
models/repo.go Show resolved Hide resolved
modules/storage/local.go Outdated Show resolved Hide resolved
modules/storage/local.go Outdated Show resolved Hide resolved
@lunny lunny force-pushed the lunny/bucket branch 2 times, most recently from e7ce453 to e4456c1 Compare June 2, 2020 16:14
@lunny lunny force-pushed the lunny/bucket branch 3 times, most recently from 9124d4e to 5ac9912 Compare June 23, 2020 12:16
@lunny lunny force-pushed the lunny/bucket branch 2 times, most recently from 169c5e0 to 461e546 Compare June 26, 2020 08:59
@lunny
Copy link
Member Author

lunny commented Jun 26, 2020

It's ready to review again now.

go.mod Outdated Show resolved Hide resolved
@techknowlogick
Copy link
Member

Writing out my thoughts in case others thinking the same as me. I was thinking about suggesting renaming minio flags, and storage provider to be s3, as s3 is mostly generic, however having it specifically be minio is better because not all s3 providers act the same. (ex. DO Spaces only supports a legacy way of listing objects in a bucket). So by having it be minio means we can later add providers as needed in case there are any edge cases they have.

@lunny
Copy link
Member Author

lunny commented Jul 7, 2020

@techknowlogick Yeah. For a self-hosted private deployment, a self-hosted minio is a better chosen. For those public gitea instances, they may want s3 compitable services.

I kept a simple interface for ObjectStorage and not use an exist package so that future implementations for s3, DO or other public services are easy enough.

Copy link
Member

@mrsdizzie mrsdizzie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to fix check

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Aug 18, 2020
@lunny lunny merged commit 62e6c9b into go-gitea:master Aug 18, 2020
@lunny lunny deleted the lunny/bucket branch August 18, 2020 04:23
@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
@delvh delvh removed the type/changelog Adds the changelog for a new Gitea version label Oct 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants