Add caching for mimedata comparison #282
Conversation
It seems that with the hierarchical algorithm we end up comparing the same things several times. Due to their low level in the hierarchy, and comparatively large size, MIME data comparisons are especially affected by this. For this reason, we simply add a LRU cache on the comparison.
We can only cache string type mime data comparisons (hashable). Also added a cache on compare_text_approximate, as profiling showed that a hash there gives a reasonable amount of hits.
|
Are there any higher-level comparisons that should be similarly memoized? |
|
I think ideally, you would want the memoization on output comparison, but currently the outputs include lists, which are unhashable. We could ensure that all such lists are tuples before we start processing, but I'm not sure how much other code that would affect. Another point of improvement could be to make the memoization commutable on the two comparison operands, assuming that the overhead for such a feature is small (e.g simply sort the two hashes before combining), but I'm not sure the gains are worth the implementation and maintenance effort. |
|
Fair enough. Merging this now, but we can explore further memoization later. Since we shouldn't be modifying notebooks during diff, we could do a one-time transform to immutable / hashable containers (list -> tuple, dict would need a custom hashable subclass, I think). |
It seems that with the hierarchical algorithm we end up comparing the same things several times. Due to their low level in the hierarchy, and comparatively large size, MIME data comparisons are especially affected by this. This PR addresses this by simply adding an LRU cache on the MIME data comparison.
Related to #237.