Incorporate install.sh into the docker image

[jakdevmail]

Problem Statement

As it stands right now, the self-hosted setup requires the install.sh script to be run.
I have only tried sentry out for a few hours and have only glimpsed at some code snippets so far. The following info may therefore be inaccurate or just plain wrong. If you notice inaccuracies, please tell me about it! I see this more as an open discussion which could improve sentry :)

The install.sh script does basically these things:

• Pull the necessary container images
• Installs a package in the host OS (package name on Debian: "jq")
• Ensures docker & docker-compose are installed on the host
• Asks if reporting analytical data is fine (or not)
• Sets up the docker volumes
• Run migrations of different databases such as the Django migrations or for ClickHouse
• Validates the configuration
• Creates the topic for Kafka
• Makes sure a user (Owner) is present, and asks if one should be created if not
• Runs tasks required for updating the service(s) -- I'm not sure if that's just a fancy docker-compose pull and run migration step
• Setup some initial data and initializes configuration files

Thanks to @aldy505 over on discord for sharing some insight on what the script is supposed to do.

I run a fair share of containerized services in my environment, over the years, through a bit of work and/or changing some docker images i've been able to get all my services to a point where they are just a docker-compose.yml file away from "just working".
Sentry is the exception to that rule. As the install.sh seems to do quite a bit, this could potentially require a considerable amount of work to get to that point.

Solution Brainstorm

Now that its clear what the script does, and what pain point i have, what is the point of all of this?
Why isn't the install.sh script simply a part of the containers themselves?

Things like docker volume creation and initial configuration can be easily achieved by simply providing an example docker-compose.yml and documenting the configuration options.

This change would mean that updating sentry is just a matter of pulling the new images and recreating the containers (basically a docker-compose pull && docker-compose up -d --force-recreate).

I'd be interested in helping with the sentry containers as i'm quite familiar with django and its migration system, but i'm not sure what i could contribute to the other container images.

Is this something that was already in the making possibly? Or is this maybe not wanted/possible due to something i've overlooked?

Looking forward to your responses :)

[androidacy-user]

Problem Statement

As it stands right now, the self-hosted setup requires the install.sh script to be run. I have only tried sentry out for a few hours and have only glimpsed at some code snippets so far. The following info may therefore be inaccurate or just plain wrong. If you notice inaccuracies, please tell me about it! I see this more as an open discussion which could improve sentry :)

The install.sh script does basically these things:

• Pull the necessary container images
• Installs a package in the host OS (package name on Debian: "jq")
• Ensures docker & docker-compose are installed on the host
• Asks if reporting analytical data is fine (or not)
• Sets up the docker volumes
• Run migrations of different databases such as the Django migrations or for ClickHouse
• Validates the configuration
• Creates the topic for Kafka
• Makes sure a user (Owner) is present, and asks if one should be created if not
• Runs tasks required for updating the service(s) -- I'm not sure if that's just a fancy docker-compose pull and run migration step
• Setup some initial data and initializes configuration files

Thanks to @aldy505 over on discord for sharing some insight on what the script is supposed to do.

I run a fair share of containerized services in my environment, over the years, through a bit of work and/or changing some docker images i've been able to get all my services to a point where they are just a docker-compose.yml file away from "just working". Sentry is the exception to that rule. As the install.sh seems to do quite a bit, this could potentially require a considerable amount of work to get to that point.

Solution Brainstorm

Now that its clear what the script does, and what pain point i have, what is the point of all of this? Why isn't the install.sh script simply a part of the containers themselves?

Things like docker volume creation and initial configuration can be easily achieved by simply providing an example docker-compose.yml and documenting the configuration options.

This change would mean that updating sentry is just a matter of pulling the new images and recreating the containers (basically a docker-compose pull && docker-compose up -d --force-recreate).

I'd be interested in helping with the sentry containers as i'm quite familiar with django and its migration system, but i'm not sure what i could contribute to the other container images.

Is this something that was already in the making possibly? Or is this maybe not wanted/possible due to something i've overlooked?

Looking forward to your responses :)

The install script does not install anything on the host. jq is installed in a container. Everything is ran in the appropriate containers, including the necessary migrations.

[jakdevmail]

Problem Statement

As it stands right now, the self-hosted setup requires the install.sh script to be run. I have only tried sentry out for a few hours and have only glimpsed at some code snippets so far. The following info may therefore be inaccurate or just plain wrong. If you notice inaccuracies, please tell me about it! I see this more as an open discussion which could improve sentry :)
The install.sh script does basically these things:

• Pull the necessary container images
• Installs a package in the host OS (package name on Debian: "jq")
• Ensures docker & docker-compose are installed on the host
• Asks if reporting analytical data is fine (or not)
• Sets up the docker volumes
• Run migrations of different databases such as the Django migrations or for ClickHouse
• Validates the configuration
• Creates the topic for Kafka
• Makes sure a user (Owner) is present, and asks if one should be created if not
• Runs tasks required for updating the service(s) -- I'm not sure if that's just a fancy docker-compose pull and run migration step
• Setup some initial data and initializes configuration files

Thanks to @aldy505 over on discord for sharing some insight on what the script is supposed to do.
I run a fair share of containerized services in my environment, over the years, through a bit of work and/or changing some docker images i've been able to get all my services to a point where they are just a docker-compose.yml file away from "just working". Sentry is the exception to that rule. As the install.sh seems to do quite a bit, this could potentially require a considerable amount of work to get to that point.

Solution Brainstorm

Now that its clear what the script does, and what pain point i have, what is the point of all of this? Why isn't the install.sh script simply a part of the containers themselves?
Things like docker volume creation and initial configuration can be easily achieved by simply providing an example docker-compose.yml and documenting the configuration options.
This change would mean that updating sentry is just a matter of pulling the new images and recreating the containers (basically a docker-compose pull && docker-compose up -d --force-recreate).
I'd be interested in helping with the sentry containers as i'm quite familiar with django and its migration system, but i'm not sure what i could contribute to the other container images.
Is this something that was already in the making possibly? Or is this maybe not wanted/possible due to something i've overlooked?
Looking forward to your responses :)

The install script does not install anything on the host. jq is installed in a container. Everything is ran in the appropriate containers, including the necessary migrations.

Alright then, so no installs on the host (that one sounded weird to me anyways).
There are still some things which aren't run in the containers such as the environment checks (Like what platform you're running on, for pulling the right images and so on).

The migrations have to run in the containers, anything else wouldn't make sense (or even work). But the migrations aren't actually started on startup.

Is there a reason for not doing that?

I cannot imagine a single scenario in which this makes sense. Conceptually speaking: If you are running an updated version of some software, it HAS to run the database migration as well, otherwise the database may very well be in a state the application doesn't expect and cannot handle.

Docker gives us a way of specifying what version of an image you want, why have some random script sit in the middle?

[hubertdeng123]

We would love to simplify the installation process for self-hosted by reducing the number of containers in use, it's actually on our radar but we haven't quite gotten to it. To answer some of your questions

Now that its clear what the script does, and what pain point i have, what is the point of all of this?
Why isn't the install.sh script simply a part of the containers themselves?

It is expected the ./install.sh script will be run on platforms that are different from each other. There will be different requirements based on the platform being used to run self-hosted. Additionally, many of the commands in the install script are not required to be ran at runtime.

The migrations have to run in the containers, anything else wouldn't make sense (or even work). But the migrations aren't actually started on startup.

The migrations are run during the ./install.sh script. So, the script will have to be rerun on every upgrade.

[jakdevmail]

We would love to simplify the installation process for self-hosted by reducing the number of containers in use, it's actually on our radar but we haven't quite gotten to it. To answer some of your questions

Now that its clear what the script does, and what pain point i have, what is the point of all of this?
Why isn't the install.sh script simply a part of the containers themselves?

It is expected the ./install.sh script will be run on platforms that are different from each other. There will be different requirements based on the platform being used to run self-hosted. Additionally, many of the commands in the install script are not required to be ran at runtime.

The migrations have to run in the containers, anything else wouldn't make sense (or even work). But the migrations aren't actually started on startup.

The migrations are run during the ./install.sh script. So, the script will have to be rerun on every upgrade.

Interesting! Simplifying the underlying container structure would surely be nice, i don't want to hinder the enthusiasm, but this feature-request was more about removing the install.sh script from a user standpoint.
I can't find an issue in the repo for that, is there one you could refer me to? I just want to keep engaged in the conversation.

To get back to one of your points:

It is expected the ./install.sh script will be run on platforms that are different from each other. There will be different requirements based on the platform being used to run self-hosted. Additionally, many of the commands in the install script are not required to be ran at runtime.

Are you referring to the install script picking out the platform like ARM or x86?
If so, i'd go with the somewhat standard route - Educate the user on what tags are available (what platforms) through documentation, and let the user pick the appropriate one.
Thats the way most containerized systems are presented as far as i know.

If user-friendliness is a concern for some install configuration which isn't yet in the post-install webpage, or which simply can't be handled beforehand, then i can also recommend another approach to getting a valid docker-compose with all the proper environmental variables set:

Installation wizard

A publicly hosted configuration wizard (No magic here, sorry). That idea is stolen shamelessly from the mailu project, if you're interested to what i mean, check it out here:

https://setup.mailu.io/2.0/

It just asks the user a sequence of questions in an easily understandable way and then generates the appropriate docker-compose.yml and .env files according to your specifications.

That would mean, that a beginner can quickly get up and running, and someone who is more experienced can deal with the configuration from a familiar docker-compose and env-file environment.

I personally think that this encompasses the best of both worlds and eliminates the need of some script to be run on the self-hoster's machine.

Open-sourcing the public instance's code would also be a practical requirement, as people are (and should be) very hesitant to enter environment specific information into a black box. I personally think the effort for creating, and running such a service is negligible in comparison to the potential uses it brings to the table.

Running migrations on container start

The migrations are run during the ./install.sh script. So, the script will have to be rerun on every upgrade.

On that note, is there anything which would prevent those migrations from being run on container startup?
Is there maybe too much overhead in the current migration-checking implementation which would prevent this due to slowing down startup-time too much? Anything on that front?

Implementing migrations on startup would basically eliminate the need for the install.sh when upgrading to newer releases, right?

[hubertdeng123]

Interesting! Simplifying the underlying container structure would surely be nice, i don't want to hinder the enthusiasm, but this feature-request was more about removing the install.sh script from a user standpoint.
I can't find an issue in the repo for that, is there one you could refer me to? I just want to keep engaged in the conversation.

#1763

Are you referring to the install script picking out the platform like ARM or x86?

Yes, as our goal was to have an easy installation as possible for our users.

On that note, is there anything which would prevent those migrations from being run on container startup?
Is there maybe too much overhead in the current migration-checking implementation which would prevent this due to slowing down startup-time too much? Anything on that front?

Any sort of db migration will incur some sort of risk, and it's possible data will be lost in that process and the installation could result in corrupted data. Additionally, with large amounts of data this migration could take a long time which could slow the startup-time in the scenario that the server crashes. I don't think that's a super ideal way to go about this.

We can take this feedback and discuss it to see if any improvements will be made. Thanks for writing in :)

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[jakdevmail]

I'm just itching to go over those points :)

For anyone reading this, and not knowing what migrations are, or why they matter, here is a quick rundown (if you know what they are, just skip this section)

What are migrations?

Database schemas may change in the lifetime of software. New tables will be added, relationships between them may change, the types or names of columns may be changed (deleted or created).
To facilitate those changes migrations are used. They describe the new state of the database and are applied through some process (which depends on the language, database and so on).

Deeper dive

Now, lets get to some of your points.

Any sort of db migration will incur some sort of risk, [...]

Yes, totally!
But with that logic, anything run against a database incurs risk.
Migrations are usually run in transactions and dropping tables or columns is pretty much the only really destructive part. Of course, migrations tend to do things which don't occur in normal operation. Tables and columns normally aren't created or changed during the runtime of software. Only their data will be (normally - there may be exceptions).

[...] and it's possible data will be lost in that process and the installation could result in corrupted data

Still on board with your statement. However, there is one big issue with it:
You have to run those migrations anyways. At some point migrations will be applied (and they should!), if you like it or not.

Why should the migrations be run? The manual way is fine, isn't it?

If we go back to "What are migrations", we'll notice a single purpose:
Get the database to a state the application expects. There is not much more to it.
It's simply a mechanism to ensure the application gets a database which is in a manageable state.

If a migration fails, the application should be unable to start. Because if it does start you get in the territory of undefined behavior. All bets are off at that point.

Taking the django migrations as an example:
Django maintains a table in the database which holds the information about what migrations have been applied already. Every time you run migrations django will automagically (still no magic here, i'm still sorry) check if those migrations have to be applied, or not.
This means that nothing is done, if nothing needs to be done.

Meaning, the only time a migration is run, is during setup and after upgrading to a newer release.
We can skip the setup part, as i think nobody has a problem with running them there (apart from some confusion about why the service isn't up yet, documentation, logging and educating the user can practically solve that issue).

When you migrate any software in production, you better make a backup of that system anyways.
None of the above problems are actually an issue if the user/customer is doing as they should be doing anyways (and if they fail to make a backup, db transactions may still save them - this is a foreshadowing for later).

Startup time

Additionally, with large amounts of data this migration could take a long time

Startup-time is certainly a concern. That's what i was worried about in my earlier posts as well. This one i feel like has to be tested and could break the camel's back. Though, again, migrations are only really applied on new releases (other systems may not be as forgiving as django, so this has to be carefully considered), a bit of educating the users can go a long way.

which could slow the startup-time in the scenario that the server crashes.

I'm kind of lost here, you're saying
Server crashes ---- (result in)> slow startups

I'd argue the application doesn't start up at all, but maybe i got lost in the words there :)

Ideas

I think i've made my thought process at least somewhat clear, yet i can practically hear some of the issues you didn't mention but are probably thinking about.

Lets get to a few:

Will this result in problems for users (or more concretely your customers)

Lets answer this with the most annoying answer: another question.

Is the current system not resulting in issues?
If someone doesn't run the migrations, we're in trouble. If they do, and they don't work as expected and break something, we're still in the same boat as running them automatically.
But i'm curious to any input on this front in any case.

What if someone gets confused as to why the service isn't up and running?

Startup may take longer due to running migrations, as discussed earlier. Impatient users (or those with a lot of data which has to be changed in the migration) may get confused/worried about the startup time. They might even choose to stop the service because they think its misbehaving or something similar.

This is not really a technical problem, but more of an educational one.
We have to make sure the user knows that migrations are being run, and everything is fine (as long as everything really is fine). The user shouldn't have to proactively check for running migrations, its something the user is told. Making the website just show a message about running migrations, the logs and docs telling a clear story can alleviate this problem to some degree.

What is the impact on bigger systems (namely those which have a lot of historical data)

I'll throw in a bold statement:
The impact will only be of positive nature.

How do i reach that conclusion? A few reasons (in no particular order):

1. Backups don't just exist, but many system administrations have learned that they make sense.
   This is of course incredibly naive. They can be forgotten, not possible due to the current state of things, or any of the thousand reasons. But its the last safeguard against any problem we might have so i'll leave it up here on this list :)
2. Simplified upgrade and installation
   This is a no brainer. You can tell anyone running sentry using docker that they can just pull the new images, recreate the containers, and grab a cup of coffee. Done. No jank script to run, not much docs to read, just one/two command(s) from a CLI they know and (probably) love. (One might go even further and call this free marketing material)
3. Database transactions are a thing
   Any failing migration can result in a roll-back of all applied migrations. I know that transactions aren't free (or even cheap sometimes), but running migrations atomically certainly makes sense here i think.
4. Potentially even less downtime
   Sounds weird in combination with the "Startup time" section, but hear me out.
   The application will start as soon as the migrations finish running. There is no "wait for the user to actually start up the service after running the migrations". They start the service, it does its thing and immediately starts up. You have to wait for the migrations in any case, why not have the service be up and running as soon as possible?

Looking forward to your response. Sorry for the wall of text, i couldn't condense it down further without losing important points in my opinion.

[hubertdeng123]

Thanks for the writeup @jakdevmail. We really appreciate the time you took here.

You have to run those migrations anyways. At some point migrations will be applied (and they should!), if you like it or not.

Yeah, you're right here, as the migrations must be run at some point regardless if upgrading to a new version. It technically shouldn't matter the timing when the migrations are run, and it is a one time process.

You've made a lot of good points here regarding impact. I'd like to keep this issue open for discussion. Unfortunately, we don't have bandwidth right now to tackle this problem but you are welcome to submit PR's to help us out.

[binaryfire]

I've just started experimenting with self-hosting Sentry and would love a solution with fewer containers and which has the install logic integrated into the entrypoint script.

You may want to check out GitLab and Mattermost for inspiration. They both apply all upgrades, including migrations, automatically at container startup. GitLab is particularly slick in this regard. It's a single omnibus container which contains all their services (and there are a lot of them). Upgrading is as simple as pulling the latest image and running it.

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀