Multiprocessing

[ArtyomMaiorov]

Hi! Is multiprocessing available for reading tables in pdf?

[foarsitter]

Can you elaborate a little more why you want to use multiprocessing? Currently it isn't supported.

[ArtyomMaiorov]

I have a program which scrapes relatively small tables from thousands of pdfs and processes them. I use camelot.read_pdf() to extract them. Currently, the read_pdf() is the main slowing point. I wonder if there is a way to speed this process up.

[bosd]

FYI, There was a related WIP in camelot-dev#237

[foarsitter]

multiprocessing can be done in many ways and I'm not convinced that we need to support it out of the box.

You have a producer (your scraper) and a consumer (camelot), that is a common paradigm. I don't know if you store your pdfs in between but if you do you can start multiple processes consuming the pdf's. Doesn't seems something we need to support out of the box.

[ArtyomMaiorov]

I will look into it, thanks!

[foarsitter]

This looks promising: https://gist.github.com/ownport/63167dbb162f998964f309a5046bef58

[bosd]

This is implemented in #17