Data persistency on shared hosting

[johnd0e]

Consider the following use case: the only aspect under user control is the Dockerfile.
To modify an option or update the tg-spam application, a new (edited) Dockerfile must be redeployed, but this process does not preserve any existing data.

Here are a couple of ideas that come to mind:

1. Synchronization Job: We could implement a synchronization job, for example, using rclone. However, we would need a mechanism to run it periodically. Perhaps tg-spam could handle this task.

2. Cloud Database: Another option would be to use a cloud database instead of relying on text files. There are some free options available, such as Cloudflare R2, which is S3-compatible.

Do you have any suggestions for this scenario?

[johnd0e]

Synchronization Job

I found some suggestions on Stack Overflow, and inotifywait seems to be the most promising option.

Cloud Database

Alternatively, we could use SQLite with cloud replication through Litestream.
I discovered this suggestion here.

[umputun]

Switching all the persistency to the database layer will address the issue you are describing and, in addition, will simplify the code. However, it will also make direct editing of sample and blacklist files impossible unless we introduce some automatic conversion from files to database records, which will add some complexity and a need to sync two sources inside the service.

I'm not sure yet if this trade-off is worth it.

[umputun]

And, because in the hosting environment you are describing, it has no access to a shared volume, the initial preparation and loading of lists is another problem. This will probably need new API endpoints (i.e., POST /spamfile or something like this) that get the whole file in the request body and the corresponding change in the UI.

[johnd0e]

unless we introduce some automatic conversion from files to database records, which will add some complexity

Given that the format of our text files is quite simple, transforming text lines into SQL statements (and vice versa) should be relatively straightforward.
Alternatively, we can also consider using the sqlean-fileio extension.

I believe there should be a single source of truth: the database. The text files could then serve solely for data interchange.

I'm not sure yet if this trade-off is worth it.

Well, here is only an idea, you decide.

the initial preparation and loading of lists is another problem.

In the scenario with SQLite and Litestream, it can be resolved simply by adding a command to the Dockerfile.
But I agree that the new API endpoint can be useful for other use cases.

[umputun]

I believe there should be a single source of truth: the database. The text files could then serve solely for data interchange.

I agree, but we still have to support an automatic conversion of all the existing lists. In addition (this one is rather nice to have), we need the ability to upload new data in the same text format if the user wishes to update lists in batch mode with some strategy for confilcts resolution . We probably need to do it the opposite way as well, which is another part that needs some development.

[johnd0e]

an automatic conversion of all the existing lists.

This process could be permanent: it would monitor for new files, import them, and delete the processed ones.

if the user wishes to update lists in batch mode with some strategy for confilcts resolution

I propose the following simple strategy:

• The new file replaces the entire database table.
• If we decide to support some form of updating, we could use files with the suffix _update.
• Since we don't have IDs in the text, the only viable conflict resolution method would be to ensure no duplicates.

We probably need to do it the opposite way as well

Dump db tables and return them as files on web-request? Or something different?

...which is another part that needs some development.

Indeed, it appears to be a substantial amount of work.