Eagerload at the controller or job layer

[mamhoff]

What is this pull request for?

(Extracted from #2309)

This moves preloading of stuff to the Job layer rather than do it at the controller layer. Two advantages: We don't preload anything we don't need, and the element repository of the page version can be used in other contexts as well (I had a case where I wanted to load multiple pages and preload their ingredients, and using the previous element repository would break that).

Notable changes (remove if none)

We now also preload ingredients. We do that explicitly at the controller level, which means we can use the PageVersion#element_repository in contexts like the following:

@page.children.where(page_layout: "blog_post).includes(public_version: :elements).each { |p| p.public_version.elements } 

Checklist

• I have followed Pull Request guidelines
• I have added a detailed description into each commit message

[tvdeyen]

This is really great. I have a question about the publish job

[wuservices]

@mamhoff We just upgraded to 6.0.12 from 4.0 and noticed a significant performance regression and it appears to be due to this change. We verified by temporarily rolling back to 6.0.5 and the previous response times returned.

After we upgraded, we immediately noticed a lot more time spent in pg/postgres and that transactions were taking 2.5x as long overall at p50 in Datadog. Most of the requests we observe are just a small Alchemy page that's used as a health check. Therefore, the partials for this page are typically cached.

It seems like even if we don't need to get any data, it's eagerly being loaded. There were originally some eager loading optimizations added in Alchemy 4.4 (#1674), but @tvdeyen mentioned there that eager loading was not added non-API HTML requests to avoid unnecessary work for cached pages.

It appears that the performance slowdown I observed is related to that consideration where we now perform extra queries even if they wouldn't have been needed. However, on a cold cache, it does seem to improve performance. Maybe even adding nested elements could further reduce queries, but that's a separate question.

Is it intended that we'll now have slower performance on cached page fragment hits? Is there any way to eager load, but only if the page isn't cached?

[tvdeyen]

@wuservices hey, thanks for bringing this to our attention. Would you be so kind and create a fresh Alchemy app where we can reproduce the problem? That would really help us to fix this, because - no - it is not intended to have slower performance 😏

cc @mamhoff

[tvdeyen]

@wuservices @mamhoff I created an issue to further discuss the problem