implement multiprocessing

[aogier]

ciao, this PR implements #172 by using multiprocessing.Processes. The choice is up to user via a Text parameter, and pool size is ATM locked to core numbers, but I'd want it to be configurable as well. Tests have been extended to basically run twice on either mp options, but I'll review that adding some cross mp/non-mp check.

Buut this is definitely ready for a first review so what do you think? Thank you, regards

[aogier]

@jsvine ping because this is still a draft :)

[aogier]

not completely sure if I correctly understand the lint errors, could it depends on a newer, stricter flake8 config file parsing?

[jsvine]

No worries, I'll investigate what's happening 👍

[jsvine]

@aogier Required a few tooling changes, but now linters are working and tests seem to be flagging a meaningful issue (as opposed just to infrastructure problems). Want to look into that?

I also had to put the call to os.sched_getaffinity in a try block because Python's scheduler interface is not supported on all platforms. Is there another way you'd prefer to set the default pool size?

[jsvine]

@aogier, whoops, I can fix the error being raised by python -m coverage html in the latest run.

[aogier]

ok @jsvine thx, let me know whatever should you need

[jsvine]

@aogier Could you tell me a bit more about your thinking here (and related uses below), esp. re. how it relates to multiprocessing?

[jsvine]

Ah, thank you for the explanation. markovify currently uses these markers:

markovify/markovify/chain.py

Lines 7 to 8 in 2d1fb78

	BEGIN = "___BEGIN__"
	END = "___END__"

Is that the sort of thing you're aiming to do here, or is it slightly different?

[codecov]

Codecov Report

Merging #173 (6033293) into master (031d236) will decrease coverage by 2.52%.
The diff coverage is 82.35%.

❗ Current head 6033293 differs from pull request most recent head 48a35fa. Consider uploading reports for the commit 48a35fa to get more accurate results

@@             Coverage Diff             @@
##            master     #173      +/-   ##
===========================================
- Coverage   100.00%   97.47%   -2.53%     
===========================================
  Files            6        5       -1     
  Lines          313      357      +44     
===========================================
+ Hits           313      348      +35     
- Misses           0        9       +9     

Impacted Files	Coverage Δ
markovify/text.py	95.13% <82.35%> (-4.87%)	⬇️
markovify/__init__.py

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[aogier]

I will check on coverage stuff asap, now ATM I'm more interested in a beer or two but I'll take a look later or tomorrow :P - I'm also curious on what windows is doing, we will see at timeout I guess, ciao!

[jsvine]

Thanks for the ping, @aogier. Review request received :) I haven't had the time to review closely, but it's in my personal queue. In the meantime, if anyone in the community has feedback, that would also be welcome.

[aogier]

Thanks for the ping, @aogier. Review request received :) I haven't had the time to review closely, but it's in my personal queue. In the meantime, if anyone in the community has feedback, that would also be welcome.

yeah kid, it looks ur personal queue is non-existent, which is ok, just a bit rude not to forewarn this :) I'll fork asap, thx for your work

[jsvine]

Hi @aogier, although I'm not a fan of that message's tone, I recognize that it has taken me longer than I hoped to prioritize this PR. As is true for many open-source projects, markovify is an entirely volunteer/unpaid endeavor, work has been busy the past few months, and the package has been fairly stable for many years. That said, let's try to find a time to discuss your PR synchronously, which will help me to review it more quickly. You can find my email address in my profile.

[aogier]

no, I am not a fan of such a behaviour when I ask if I can do some work if I can get at least some feedback and yes go w/ it, but then it's a no. My time is free too, my effort is volunteering too, so my friend at least try not to bullshit me by telling me you have it in the todo when that is simply not true (which could be fine, although a little rude given I've spent some time because you told me you could then put some).
So stop kidding ourselves, if you're not comfortable with people contributing things that you didn't do that's perfectly fine, it means I'll propose the change, which is of value to me, in a fork to which I'll give the attention I like best. Greetings