Langfuse v3 container fails to start due to Clickhouse permission issue

[guusfrenken-wk]

We are currently attempting to upgrade from Langfuse 2.93.7 to 3.5.1 (self-hosted) on Azure, using Terraform to manage the resources.
The Lanfgfuse web and worker apps are Linux Web Apps and Minio, Redis and Clickhouse live in an Azure Container Instances resource. The 2 Clickhouse volmes are mounted to File Shared in an Azure Storage account.

The issue we are running into is that the Langfuse Web container fails during startup because the database migrations fail due to an apparent permission issue. This is the log from the Langfuse Web container:

2025-01-06T09:22:33.975768221Z Datasource "db": PostgreSQL database "<pg dbname>", schema "public" at "<host>:5432"
2025-01-06T09:22:34.228976336Z
2025-01-06T09:22:34.229041837Z 273 migrations found in prisma/migrations
2025-01-06T09:22:34.229047337Z
2025-01-06T09:22:34.342101374Z
2025-01-06T09:22:34.346808251Z No pending migrations to apply.
2025-01-06T09:22:35.438437288Z error: code: 1001, message: std::__1::__fs::filesystem::filesystem_error: filesystem error: in rename: Permission denied ["/var/lib/clickhouse/store/e4c/e4c2ff7c-6ace-4596-b8cf-f8e06ff51e82/tmp_insert_all_3_3_0/"] ["/var/lib/clickhouse/store/e4c/e4c2ff7c-6ace-4596-b8cf-f8e06ff51e82/all_3_3_0/"]
2025-01-06T09:22:35.441087731Z Applying clickhouse migrations failed. This is mostly caused by the database being unavailable.
2025-01-06T09:22:35.441115431Z Exiting...

From the logs, it seems obvious that Clickhouse doesn't have write permissions in the filesystem, but the thing that confuses me is that the file share is full of Clickhouse-related files and directories so Clickhouse clearly has write access. Am I missing something, do any other resources need access to the Clickhouse data? Is there a working setup for Langfuse v3 in Azure?

This is the Terraform config for the Clickhouse container:

resource "azurerm_container_group" "langfuse_container_group" {
  name                = "<redacted>"
  location            = azurerm_service_plan.service_plan.location
  resource_group_name = azurerm_resource_group.resource_group.name
  ip_address_type     = "Public"
  os_type             = "Linux"
  dns_name_label      = "<redacted>"

  # redis and minio containers...

  container {
    name  = "clickhouse"
    image = "clickhouse/clickhouse-server:24.12.1"
    cpu   = "1"
    memory = "1"

    environment_variables = {
      CLICKHOUSE_DB       = "default"
      CLICKHOUSE_USER     = "clickhouse"
      CLICKHOUSE_PASSWORD = data.azurerm_key_vault_secret.langfuse_clickhouse_password.value
    }

    ports {
      port     = 8123
      protocol = "TCP"
    }

    ports {
      port     = 9000
      protocol = "TCP"
    }

    volume {
      name       = "langfuse-clickhouse-data"
      mount_path = "/var/lib/clickhouse"
      read_only  = false
      share_name = azurerm_storage_share.langfuse_clickhouse_data.name

      storage_account_name = azurerm_storage_account.storage_account.name
      storage_account_key  = azurerm_storage_account.storage_account.primary_access_key
    }

    volume {
      name       = "langfuse-clickhouse-logs"
      mount_path = "/var/log/clickhouse-server"
      read_only  = false
      share_name = azurerm_storage_share.langfuse_clickhouse_logs.name

      storage_account_name = azurerm_storage_account.storage_account.name
      storage_account_key  = azurerm_storage_account.storage_account.primary_access_key
    }
  }
}

[dosubot[bot]]

The error you're encountering suggests that Clickhouse is unable to rename files due to permission issues, even though it appears to have write access. Here are a few things you can check or try to resolve this issue:

1. File Share Permissions: Ensure that the Azure File Share permissions are correctly set to allow the necessary read/write operations. Double-check the permissions on the Azure Storage account and ensure that the Clickhouse container has the necessary access rights.

2. Container User Permissions: Verify that the user running the Clickhouse process inside the container has the necessary permissions to perform file operations on the mounted volumes. You might need to adjust the user or group permissions within the container.

3. Volume Mount Options: Check if there are any specific mount options required for Azure File Shares that might affect file operations. Ensure that the volumes are mounted with the correct options to allow renaming and other file operations.

4. Azure Storage Account Configuration: Ensure that the storage account is configured to support the necessary operations. This includes checking the performance tier and any potential throttling or access restrictions.

5. Clickhouse Configuration: Review the Clickhouse configuration files to ensure there are no settings that might restrict file operations. This includes checking any custom configurations that might have been applied.

6. Logs and Diagnostics: Enable detailed logging and diagnostics for both the Azure Storage account and the Clickhouse container to gather more information about the permission errors.

If these steps do not resolve the issue, consider testing the setup with a different storage solution or environment to isolate the problem further.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[marcklingen]

let's move this to an issue as this might be a bug

#4893