Support extreme off-plot data points in scatter lines #2060

alexcjohnson · 2017-10-04T04:13:08Z

Fixes #403 and #1660 (which is a duplicate as expected) - when scatter line points go extremely far off the plot, the SVG would break. Turns out both Chrome and FF have limits at +/- 2^29 px (about 500 million), after which the path (and sometimes other paths too) disappears. I didn't check other browsers (except nw, I'll make a note in the code about that).

Basically a brute-force solution: once you get 20 screen widths away (so, max of ~100k px for a very large screen), clip the path to where it intersects that rectangle, then follow it around on that rectangle as it moves outside the rectangle (so that fills will be handled correctly) then revert to a regular path when it passes back inside that rectangle. That way paths inside the +/- 20-screen rectangle are precisely correct, preserving fix #353 of bug #217. One can ponder the effects of this on other line shapes - they all have edge cases in which this alters them, but to my mind it's not worth the extra complexity of dealing with that.

For in-bounds points the overhead is pretty minimal - 6 or 8 extra simple logic operations, quite a bit less than we pick up from clustering. For out-of-bounds points (which hopefully are rare enough that we don't often need to be too worried about their impact) there are a few line intersections to calculate, quite a bunch of annoying logic, and perhaps more importantly one out-of-bounds point can turn into 2 or 3 clipped points in the path, though on the flip side, many out-of-bounds points together generally collapse into just a few perimeter points.

cc @etpinard

alexcjohnson · 2017-10-04T04:36:14Z

test/image/mocks/axes_range_manual.json

        4,
        5,
        6,
        7,
        8
      ],
-      "type": "scatter"
+      "type": "scatter",
+      "mode": "lines"


nw has problems with displaying the markers for this trace if you include them: you get the axes and gridlines but no traces drawn at all. If you drop the exponent 100 to 12 or less, it works again. Clipping markers should be easier than lines, but I opted to leave that for another time. If it's only nw that cares, perhaps image-exporter will moot the issue - but it bears a little more investigation -> #2061

BTW I modified this mock because, while ultra_zoom does test the functionality, I wanted to cover a few more cases with the image tests. But scatter_test covers many many more cases.

using the new image exporter and 🔪 that "mode": "lines", line, we get:

alexcjohnson · 2017-10-04T04:37:14Z

test/jasmine/assets/custom_matchers.js

@@ -127,7 +127,7 @@ var matchers = {
                    'to be close to',
                    arrayToStr(expected.map(arrayToStr)),
                    msgExtra
-                ].join(' ');
+                ].join('\n');


This way the two arrays always start at the same column, so it's easy to match points and see what failed.

etpinard · 2017-10-04T20:58:27Z

src/traces/scatter/line_points.js

+        lastXEdge = lastYEdge = 0;
+    }
+
+    function addPt(pt) {


So, if I understand correctly, this here gets calls for all line.shape values?

If so, it might be nice to add one or two non-'linear' line.shape traces in the ultra_zoom mock.

That's correct. And good call on adding some test cases for other line.shape values. I actually might want to think about this a little - spline is probably going to have to stay untouched, but [hv]+ may have relatively simple ways to make them behave perfectly correctly, which they don't now because we linearly interpolate between the on-screen and extreme points.

Spline actually works remarkably well using the linear algorithm - because Catmull-Rom smoothing reaches a steady limit to the curve shape on-screen as the next point moves farther and farther off-screen.

37b5cdc adds special handling for the right-angle line shapes - not too bad, certainly less involved than linear! Test-wise, in the interest of time I only altered ultra-zoom, did not add extra jasmine tests for these, I hope that's OK.

I hope that's OK.

Yeah, image test support should suffice here. 👍

spline is left working like linear, which works surprisingly well!

alexcjohnson · 2017-10-05T19:41:41Z

test/jasmine/tests/plots_test.js

+            // hacky: simulate getting stuck with these flags due to an error
+            // see #2055 and commit 6a44a9a - before fixing that error, we would
+            // end up in an inconsistent state that prevented future Plotly.newPlot
+            // because _dragging and _dragged were not cleared by purge.


That's the first time I'd seen an error that blocked all future Plotly.newPlot calls! We really need to encapsulate all these gd.*** vars better...

Yep, very nice. Thanks!

etpinard · 2017-10-05T19:47:29Z

src/plots/plots.js

-    delete gd.hmlumcount;
-    delete gd.hmpixcount;
+    delete gd._hmlumcount;
+    delete gd._hmpixcount;


good catch!

etpinard · 2017-10-05T19:54:28Z

src/traces/scatter/line_points.js

+    // hvh and vhv we sometimes have to move one of the intersection points
+    // out BEYOND the clipping rect, by a maximum of a factor of 2, so that
+    // the midpoint line is drawn in the right place
+    function getABAEdgeIntersections(dim, limit0, limit1) {


Nice job making this file very readable, even though the algo is quite heavy 💪

I'm thinking we might want to release this module here as a separate package at some point.

etpinard · 2017-10-05T19:54:48Z

test/image/mocks/ultra_zoom.json

+        {"x": [10000, 0.0001], "y": [0.0001, 10000], "fill": "tozeroy", "line": {"shape": "vhv"}},
+        {"x": [0, 5001], "y": [5002, 0], "line": {"shape": "hv"}},
+        {"x": [5002, 0], "y": [0, 5003], "line": {"shape": "vh"}},
+        {"x": [0, 5000, 5001, 5002, 5003], "y": [-5000, 5000, 5000, 5000, 15000], "line": {"shape": "spline"}}


Solid mock 🔒

etpinard · 2017-10-05T20:00:10Z

A very impressive PR. line_points.js is definitively in the running for plotly's most elegant algorithm 🏆

💃 💃 💃

Don't forget to close #1660 too (I don't think Github is smart enough to auto-close multiple issues)

alexcjohnson added 3 commits October 3, 2017 22:41

Algorithm to clip huge line points to a merely large box

3130d1c

lint and factor out constants

231883f

updated baselines for extreme-point fix

4b0b5d3

alexcjohnson mentioned this pull request Oct 4, 2017

Investigate extreme data in all trace types #2061

Closed

7 tasks

alexcjohnson commented Oct 4, 2017

View reviewed changes

alexcjohnson added status: reviewable bug something broken labels Oct 4, 2017

etpinard reviewed Oct 4, 2017

View reviewed changes

alexcjohnson added 3 commits October 5, 2017 10:27

🔪 old comment

5ec5dbf

tweak purge related to @etpinard's extra fix in #2055

8d3ef29

handle hv/vh/hvh/vhv line shapes in extreme point clipping

37b5cdc

spline is left working like linear, which works surprisingly well!

alexcjohnson commented Oct 5, 2017

View reviewed changes

etpinard reviewed Oct 5, 2017

View reviewed changes

src/plots/plots.js

delete gd.hmlumcount;

delete gd.hmpixcount;

delete gd._hmlumcount;

delete gd._hmpixcount;

Copy link

Contributor

etpinard Oct 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch!

etpinard reviewed Oct 5, 2017

View reviewed changes

alexcjohnson merged commit 10f99a2 into master Oct 5, 2017

alexcjohnson deleted the huge-points branch October 5, 2017 20:03

alexcjohnson mentioned this pull request Oct 5, 2017

y-range and extreme outliers: traces vanish progressively #1660

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support extreme off-plot data points in scatter lines #2060

Support extreme off-plot data points in scatter lines #2060

alexcjohnson commented Oct 4, 2017

alexcjohnson Oct 4, 2017

alexcjohnson Oct 4, 2017

etpinard Oct 4, 2017

alexcjohnson Oct 4, 2017

etpinard Oct 4, 2017

alexcjohnson Oct 4, 2017

alexcjohnson Oct 5, 2017

etpinard Oct 5, 2017

alexcjohnson Oct 5, 2017

etpinard Oct 5, 2017

etpinard Oct 5, 2017

etpinard Oct 5, 2017

etpinard Oct 5, 2017

etpinard commented Oct 5, 2017

Support extreme off-plot data points in scatter lines #2060

Support extreme off-plot data points in scatter lines #2060

Conversation

alexcjohnson commented Oct 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etpinard commented Oct 5, 2017