Redis configuration

[rtrompier]

How Shlink is set-up

• Shlink Version: 2.10.1
• PHP Version: N.C
• How do you serve Shlink: Docker image
• Database engine used: PostgreSQL

Summary

Hello,
We use Shlink in a cluster mode with 2 instances, and one redis server.
Sometimes, we can see somes OOM for our Redis instance.
Wich sizing and TTL configuration do you suggest to use for redis server ?

Current Redis resources limitations :

resources:
        limits:
          cpus: "1"
          memory: 512M

Current Redis TTL configuration :

save 900 1
save 300 10
save 60 10000

[acelaya]

At the moment Shlink is barely using Redis for anything. I plan to change this to apply performance improvements when possible, but I find very unlikely that those memory issues are caused by Shlink.

It could be using 1% of those 512Mb, tops.

Are you maybe using that Redis instance for something else?

[acelaya]

Maybe this helps deciding how much memory to allocate https://docs.redis.com/latest/rs/administering/designing-production/hardware-requirements/, but according to that, Redis recommends way more than 512Mb.

[rtrompier]

Hi, and many thanks for the quick reply
We have increase the RAM to 4GB and it should be better.

In the Redis logs, we can see it persist more than "1% of the resource". Do you have the same things on your side ?

In the same time, we have started a stress test on the Shlink cluster with 10000 queries, and we can see the memory usage is increasing with never flush it. Can you reproduce the same issue ?

[acelaya]

Hmm, that's unexpected. Shlink is not actively using that much.

Shlink is currently using some abstractions to interact with redis, and I know some other Shlink dependencies rely on the same abstractions.

I'm wondering if there are some extra unintended usages. I'll investigate.

[acelaya]

I can confirm the database abstraction layer is also using redis when configured, and it creates cache entries that never expire, but I suspect they are not always relevant, because it grows and grows as long as you create new URLs.

It is possible to set a default lifetime, though, so I'm going to do it.

[acelaya]

I have marked this discussion as resolved. You can follow-up the resolution of the bug here #1308

[acelaya]

I have just released v2.10.2 (the docker image will be available once this build finishes).

This release adds a 24h lifetime for cache entries. That should mitigate the issue for a regular Shlink usage.

[acelaya]

Hey @rtrompier @chirogocki did you have the chance to try with Shlink 2.10.2? Does it clear the cache every 24h as expected? Is it still caching too much during those 24h?

[rtrompier]

Hi @acelaya

Many thanks for the new version.
The stress test is in progress with the latest version ;)

I come back to you when I have a feedback.

[acelaya]

The workaround won't have an impact on a stress test context, as the cache entries expire after 24h. If you create 10k URLs in one go, it will still consume more memory in redis.

It would be good to know if you no longer see redis running out of memory for a real use-case.

However, if you expect that kind of load on a regular bases, I may need to give it another thought, probably by completely disabling this caching logic from the database abstraction layer, or at least, by allowing it to be disabled. It would need some testing though, to make sure it does not have a negative performance impact.

[rtrompier]

Since we have increase the RAM of our Redis instance to 4Gi we don't see anymore OOM.

Understood for the stress test. I just wait couple of hours to be sur the new version is stable before to run in production.

I work for the Geneva Hospitals, and we use Shlink to generate shorturl to send QR Code to our patients / visitor, so I can't send the new version without some tests ;)

I come back to you in few hours with the result of my memory monitoring.

[acelaya]

It makes complete sense. Thanks for checking 🙂

[rtrompier]

Hi @acelaya,
Here is a little feedback about the memory usage of our 2 Shlink containers in our Staging env (light usage).

As you can see, the memory usage increase.
I'm gonna wait a few hours to see how the container is working when reach 100%.

[rtrompier]

Just to be sure, I'm not talking anymore about the Redis container, but Shlink containers ;)

[rtrompier]

Little update from yesterday night :

At 18:20 we have seen an OOM followed by a restart of the Shlink container due to Memory fully used

[acelaya]

Oh! I thought we were talking about redis memory consumption all the time.

Then a couple of questions pop-up:

• Is the issue with redis solved with v2.10.2?
• Did you ever experience this memory leak with previous Shlink versions, or did it start to happen with v2.10.2?

[rtrompier]

No problem, my bad.

The issue with Redis seem not be really solved. Until this morning I have no more OOM. but I have redeployed it this morning to add ports forwarding to check the datas present inside the container, and now the container is not starting (OOM due to lot of datas).

Concerning Shlink, i'm not sur, but I think its due to migration to cluster (2 instances + redis).
I'm gonna revert to only 1 instance and check the memory usage ;)

[acelaya]

Thanks, that's very helpful.

I'll create two separate tickets to investigate this a bit further.

[semangard]

Hello, my colleague @rtrompier still have the issue with only 1 instance ...
@acelaya : could you please create tickets ? (do not hesitate to ask questions to @rtrompier if you need more details)

[acelaya]

Could you, by any chance, share the env vars you are using (no need for the values)? Just to have an idea of what features are you using.

Also, when you say you experience this with only one instance, do you mean one Shlink instance using redis, or one Shlink instance without redis.

[rtrompier]

Hi @acelaya,
Here is a part of our docker-compose.
As you can see, we have only one instance of shlink server, without Redis (env var is commented).

shlink:
    image: shlinkio/shlink:2.10.2
    environment:
      DB_DRIVER: postgres
      DB_NAME: shlink
      DB_USER: shlink
      DB_PASSWORD: xxxxxxxx
      DB_HOST: shlink-postgres
      DB_PORT: 5432
      SHORT_DOMAIN_HOST: mydomain.ch
      SHORT_DOMAIN_SCHEMA: https
      DEFAULT_SHORT_CODES_LENGTH: 15
      GEOLITE_LICENSE_KEY: xxxxxxxxx
      # REDIS_SERVERS: tcp://shlink-redis:6379
    networks:
      - network
      - traefik_network
    depends_on: 
      - shlink-postgres
    healthcheck:
      test: ["CMD","curl","-s","--connect-timeout", "2","--max-time","5","--output","/dev/null","http://localhost:8080/rest/health"]
      interval: 10s
      timeout: 20s
      retries: 4
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.shlink-unsecure.rule=Host(`mydomain.ch`)"
        # - "traefik.http.routers.shlink-unsecure.middlewares=redirect@file"
        - "traefik.http.routers.shlink-unsecure.entrypoints=http"
        - "traefik.http.routers.shlink.rule=Host(`mydomain.ch`)"
        - "traefik.http.routers.shlink.tls=true"
        - "traefik.http.routers.shlink.entrypoints=https"
        - "traefik.http.services.shlink.loadbalancer.server.port=8080"
        # https://doc.traefik.io/traefik/routing/services/#health-check
        - "traefik.http.services.shlink.loadbalancer.healthCheck.path=/rest/health"
        - "traefik.http.services.shlink.loadbalancer.healthCheck.interval=30s"
        - "traefik.http.services.shlink.loadbalancer.healthCheck.timeout=20s"
        - "traefik.http.services.shlink.loadbalancer.healthCheck.scheme=http"
        - "traefik.http.services.shlink.loadbalancer.healthCheck.port=8080"
      replicas: 1
      placement:
        constraints:
          - node.role == worker
        preferences:
          - spread: node.labels.datacenter
      resources:
        limits:
          cpus: "2"
          memory: 1G
        reservations:
          cpus: "2"
          memory: 1G

[acelaya]

Thanks!

This will help me get to the bottom of the issue.

[acelaya]

I think I have found what's causing the memory leak. The library I use for database persistence uses some cache adapter internally, from a library they used to maintain and is currently deprecated.

That library has a bug which is generating new cache entries non-stop. However, since it's there just for BC, they are not fixing it and the recommendation is to migrate away from it (which totally makes sense).

Shlink was still using it because there were still some "traces" of it in some other of their libraries, but it's relatively easy to bypass that and avoid using it at all.

After that, I can see there no longer thousands of entries created for every request to Shlink, and the amount of memory used by Shlink and redis is reasonable.

Also, the memory leak was there even without redis, because when Shlink does not detect redis, still uses the local memory for caching, using the same abstraction that was affected by the bug.

I will publish v2.10.3 later today, including the fix for the memory leak. It would be amazing if you can test it afterwards.

[rtrompier]

Hi @acelaya
Of course I will test it ;)
I come back to you as soon as I can provide you a feedback.
Many thank for this fix !

[acelaya]

Released v2.10.3, which fixes the memory leak.

The docker image will be available as soon as https://github.com/shlinkio/shlink/runs/4913844311?check_suite_focus=true has finished.

[rtrompier]

Hi @acelaya
Everything seem fixed with this version ;)
Thank you very much for your help !

I have another question : Is this fix is embed in the v3.x ?

[acelaya]

Amazing. Glad to hear that :)

Yes, v3.0.0 includes the fix.