Can WeasyPrint produce a large and heavy documents?

[survtur]

I want to use WeasyPrint to produce a document with a lot of pages with rasterized images, all different, with high resolution. This kind of documents weight 2Gb or even more. Will WeasyPrint produce it incrementally? Or it will prepare it consuming a lot of memory and then save to disk?

I looked at the code. Seems like there is no generators that would prepare document incrementally. Am I right that whole document should fit into memory?

[liZe]

Hi!

I looked at the code. Seems like there is no generators that would prepare document incrementally. Am I right that whole document should fit into memory?

Unfortunately, you are, and there are many reasons for that. The last steps of the PDF generation (including internal links) require the whole document to be available in memory. We also have many cases where the last page have to be rendered once before re-rendering the first one, for example when the document includes the total number of pages on each page.

It’s not impossible to find workarounds, but that would probably require a lot of complex code.

The "real" fix would be to use less memory. WeasyPrint shouldn’t require the very large amount of memory it currently uses. Hunting for useless memory uses takes a lot of time, can be very rewarding when we find a way to save a lot of memory, but actually leads to nothing interesting most of the time :/. To be honest, the major memory use improvements done by WeasyPrint in the last years are mainly caused by the amazing improvements of the Python interpreter.

If most of your memory is eaten by images, it could be possible to find a way to not keep these images in memory, but use temporary files instead. I’ve never deeply checked what’s done with images, it could be useful to look carefully at images.py. If you know a little bit of Python, you can even try to find a dirty workaround to see if it helps.

Also note that, depending on your document, it could also be possible to split the HTML file to generate different PDF files, and then concatenate them using another PDF edition library.

I want to use WeasyPrint to produce a document with a lot of pages with rasterized images, all different, with high resolution. This kind of documents weight 2Gb or even more.

Then you’ll probably have to either buy a lot of RAM, split your HTML files, or find a way to make WeasyPrint use dramatically less RAM 😄.

[survtur]

I agree that whole document should be formed in order to get counters etc. But also it does not have to keep in memory whole images. It need just dimensions.

I could free some memory by stop keeping pillow_images in memory. I'll just keep its filename so I could read its content in a proper moment. But for HTML.render() function only dimensions are required.

I do some tests on document with 13700 images totalling 13,7GB

Original WeasyPrint ate about 10.7G or my RAM and died with "9: SIGKILL". It happend somewhere at 1378 page.
After stop keeping them in memory it took 1.1G of RAM for whole rendered document. It was placed 1828 pages.
Just renredered, not written out.

Now I see that creating PDF by pydyf module eats a lot of memory. I see if i can postpone keeping image in memory till that final writing-to-file moment...

[liZe]

Thanks a lot for your comment.

I could free some memory by stop keeping pillow_images in memory. I'll just keep its filename so I could read its content in a proper moment. But for HTML.render() function only dimensions are required.

👍

Original WeasyPrint ate about 10.7G or my RAM and died with "9: SIGKILL". It happend somewhere at 1378 page.
After stop keeping them in memory it took 1.1G of RAM for whole rendered document. It was placed 1828 pages.
Just renredered, not written out.

That’s really interesting. If we find a way to avoid the problem of the PDF generation (and it’s probably possible, even if we have to change pydyf’s API), that would be a great way to save memory.

Now I see that creating PDF by pydyf module eats a lot of memory. I see if i can postpone keeping image in memory till that final writing-to-file moment...

Good luck, don’t hesitate to ping us if you are stuck!

[survtur]

I've solved my problem. My whole process consumes max 1.3G of RAM. It produces 3GB .pdf on output. Original images were 13GB but JPEG recompression makes them smaller (I think I'll make a switch to disable that compression).

What have been done:

1. weasyprint.images.RasterImage does not contain full pillow_image. Now it contains smaller object that can load image from disk on demand.
2. pydyf._to_bytes() asks for bytes from that object when it is time to write data to final pdf.
3. weasyprint.document.Document.write_pdf() now writes to disk file, instead of io.BinaryIO (memory)

I made very tiny changes to pydyf and some changes to WeasyPrint. What so you think I should do now? Should I publish it somewhere?

[survtur]

I made forks of WeasyPrint and pydyf, and I made commits with my changes.

I would be happy if some of my code will be useful and arrive at WeasyPrint code.

[liZe]

I made forks of WeasyPrint and pydyf, and I made commits with my changes.

That’s great, thanks a lot! 💜

I think that we may be able to provide this feature with a clean API, and even try to do so without changing pydyf. Are we allowed to include some of your code into WeasyPrint?

[survtur]

Are we allowed to include some of your code into WeasyPrint?

Yes of course. Having my code in so popular project is a reason to be proud.

[survtur]

try to do so without changing pydyf.

I see that my DelayedPillowImage can be based on pydyf.Object class with data property. This datashould work the same way as my first get_bytes attempt. So pydyf changes not required.

I've pushed new commit to my weasyprint fork.

[liZe]

Thanks a lot!

I’ve pushed a new save-memory branch that you can try. The code is not tested nor documented yet, but may already work well for you.

Without changing anything in the way you use WeasyPrint, it should be able to save a lot of memory: Pillow images are now created, transformed into ready-to-store images and then forgotten, so that we avoid keeping them too long. That’s by far the main improvement according to my tests.

There’s also a new --cache-folder <folder> option that keeps the images on disk instead of memory. It saves a few more megabytes for my use cases, but may save much more for your documents.

Don’t hesitate to report some feedback! I’m really interested in knowing if you hit a speed penalty (I don’t, and I’m a bit surprised about that), and if the --cache-folder option is worth it for you (or if the first improvement is already enough).

[survtur]

Good. Now it can produce documents with infinite size.
Just note that draw_first_line() calls for RasterImage constructor. It requires cache parameter. There should be something. It seems that it is emoji-related thing.

P.S.: Unfortunately I still can't use It without my own upgrades. I think I'll explain it in separate issue.

[liZe]

Just note that draw_first_line() calls for RasterImage constructor. It requires cache parameter. There should be something. It seems that it is emoji-related thing.

Thanks.

P.S.: Unfortunately I still can't use It without my own upgrades. I think I'll explain it in separate issue.

👍🏽