Longterm caching with Polars

[insanpraja]

We were trying to do this with Dozer, but we can't find ways to tell Dozer to read Polars when the query is, let's say, not exceeding 30 days/1 month, and read from DB (PgSQL/MySQL) if it's older than Polars cache.

Any suggestions?

[chubei]

Hi insanpraja, can you explain why you need Dozer to only read data within a month? I can't get it from the diagram you posted.

[insanpraja]

Hi @chubei, our intention is to cache 30 days of data in polars, and the rest of the data stay in MySQL database.

Thanks.

[chubei]

In this case, Dozer can directly connect to MySQL. It's not necessary to go though the polars cache.

Dozer does one snapshot of your database when it's first started, then it listens to CDC events to keep data up to date.

[insanpraja]

Thanks for insight!