[0.14] Time sorting seems broken

[cheribral]

Bug report

System info: [Include InfluxDB version, operating system name, and other relevant details]
Influxdb 0.14.0~n201605260800

Steps to reproduce:

1. Query data

Expected behavior:
Points come back in order

Actual behavior:
2016-05-26T22:40:00Z 1038
2016-05-26T22:50:00Z 962
2016-05-26T23:00:00Z 3483
2016-05-26T22:50:00Z 1896
2016-05-26T23:00:00Z 1754
2016-05-26T22:40:00Z 113
2016-05-26T22:50:00Z 801
2016-05-26T23:00:00Z 12076
2016-05-26T22:50:00Z 837
2016-05-26T23:00:00Z 40338
2016-05-26T23:10:00Z 4938
2016-05-26T22:40:00Z 64
2016-05-26T22:50:00Z 826
2016-05-26T23:00:00Z 110
2016-05-26T23:10:00Z 2250

Additional info:

We noticed this on one of the servers running the nightly build. While it provided quite a bit of amusement for people passing around grafana graphs that looked like scribbles, I figured I would report it as a bug.

[e-dard]

@cheribral thanks for the report. We're going to need more details. Do you have the raw data available? What was the query you executed? Can you reproduce this with an empty database and some sample data?

[cheribral]

@e-dard Apologies for the delay.

There are 13376 series in that particular measurement, structured like:

batch_trip_writer.time_to_write_trips,customer=xxx,host=xxx,is_completed=True,is_reprocess=False,location=xxx,service=xxx,thread=4,tuple=xxx

show field keys from "batch_trip_writer.time_to_write_trips"
name: batch_trip_writer.time_to_write_trips
fieldKey    fieldType
duration    float
total_points    float

The query is:

SELECT count("duration") FROM "batch_trip_writer.time_to_write_trips" WHERE time > now() - 24h GROUP BY time(10m), "location" fill(null)

If it helps, a git bisect gives me this as the breaking commit:

0b481ff6271a404d440684d3c68ab02515674715 is the first bad commit
commit 0b481ff6271a404d440684d3c68ab02515674715
Author: Jason Wilder <mail@jasonwilder.com>
Date:   Wed May 25 08:55:46 2016 -0600

    Fix pathalogical TSM query case

    This fixes a pathalogical query condition cause by and problematic
    structuring of TSM files based on how points were written.  The
    condition can occur when there are multiple TSM files and a large
    number of points are written into the past.  The earlier existing
    TSM files must also have points in the past and close to the present
    causing their time range to eclipse the later files.

    When this condition occurs, some queries can spend an excessive amount
    of time merge all the overlapping blocks.

    The fix was to constrain the window of overlapping blocks based on
    the first one we ran into.  There was also a simple case in the Merge
    where we could skip the binary search path and just append the two
    inputs.

[jwilder]

@cheribral Would you be able to test #6897 to see if it resolves this?

[cheribral]

Just to confirm, the issue is fixed.

[cheribral]

@jwilder Sorry, I was just tipped off by someone that they are still seeing this problem. I've tested with the latest nightly, and it is indeed still a problem.

[cheribral]

@e-dard Would you like a new issue for this, or should this one be opened again? The data is coming back jumbled, and the problem still bisects to that same commit.

Also, would having some backups files to test with be helpful?

[jwilder]

@cheribral Yes, if you could share your shard data for this that would be very helpful.

[cheribral]

@jwilder Is there an influxdata email address I can use to send the access information?

[e-dard]

@cheribral our email addresses are our first names @influxdb.com, e.g., edd and jason.