Add retries functionality

[MatthewFlamm]

NWS servers are flaky. This PR adds in ability to use tenacity to do retries. This can be done by downstream applications already, but some downstream applications, such as homeassistant prefer to have this logic done inside the upstream libraries.

This implements a simple retry strategy to try every X unit of time until Y unit of time elapses.

[lymanepp]

Hi @MatthewFlamm, you may want to consider only retrying for server errors (5xx).

[MatthewFlamm]

Thanks I should have made this draft as there are a few things I need to change. I didn't yet think of your point, which is a good one.

I'm also unsure why the tests fail on python 3.8 and 3.9 right now.

[lymanepp]

I'm also unsure why the tests fail on python 3.8 and 3.9 right now.

This might be a good time to drop support for those old versions.

[MatthewFlamm]

Hi @MatthewFlamm, you may want to consider only retrying for server errors (5xx).

I need to look into how to best detect catch 500 errors on the client-side in tenacity. The server side errors are nicely grouped in aiohttp.web, but these aren't appropriate for the client-side. Might need to provide a custom retry predicate for tenacity.retry.

At least all the python versions are failing now that I have the right test. I expect the 3.8 and 3.9 tests to continue to fail when this is fixed. I have an outside hope that we can figure out this issue, as it could be a testing problem or a testing environment problem. Just removing support is also an option as these are very old versions of python and this package is widely used for one downstream app that is always on the bleeding edge.

[MatthewFlamm]

This is ready for full review. It still fails in python 3.8 and python 3.9. I will leave for now in case we can figure that out. Otherwise we can drop support for those versions. The retry behavior should be generally useful given NWS flakiness.

[MatthewFlamm]

I was able to get a MRE for this and submitted an issue jd/tenacity#456

[MatthewFlamm]

I suspect that there is some inspection logic in tenacity that prevents detecting AsyncMock as async only in previous Python versions. The solution is to wrap the mock object in a regular async function, but have the mock object defined outside the scope of the function. It looks ugly, but it works. If you have time to review @lymanepp, I would appreciate it

[lymanepp]

I question adding the retry logic to the existing classes as they're not used internally. Can they be moved to a separate class (or function)?

[lymanepp]

It looks ugly, but it works. If you have time to review @lymanepp, I would appreciate it

It looks fine to me--I added some suggestions inline.

[MatthewFlamm]

Your comment makes sense to me, nws.call_with_retry(NWS.update_observation) is awkward. I made it a function. Now it is call_with_retry(NWS.update_observation).

[lymanepp]

I forgot to remove .pylintrc, can you include that in this PR?

[lymanepp]

LGTM