Panic error with inserting values with commas

[michal800106]

I receive panic error when I'm trying to write message with commas in fields value.

# curl -G 'http://localhost:8086/query' --data-urlencode "q=CREATE DATABASE mmm"
{"results":[{}]}

# curl -XPOST 'http://localhost:8086/write?db=mmm' --data-binary '@t.g'
curl: (52) Empty reply from server

# cat t.g
{
  "database" : "mmm",
  "retentionPolicy" : "",
  "points" :  [ {
    "measurement" : "http_status",
    "tags" : {
      "service" : "identity-service",
      "hostname" : "host1",
      "url" : "http://localhost:35774/"
    },
    "timestamp" : "2015-06-16T09:28:26.000Z",
    "fields" : {
      "value_meta" : "{\"error\":\"Pattern match failed! \\\".*v3.0.*\\\" not in \\\"{\\\"version\\\": {\\\"status\\\": \\\"stable\\\", \\\"updated\\\": \\\"2015-03-30T00:00:00Z\\\", \\\"media-types\\\": [{\\\"base\\\": \\\"application/json\\\", \\\"type\\\": \\\"application/json\\\"}], \\\"id\\\": \\\"v3.4\\\", \\\"links\\\": [{\\\"href\\\": \\\"http://localhost:35774/\\\", \\\"rel\\\": \\\"self\\\"}]}}\\\"\"}",
      "value" : 1.0
    }
  } ],
  "tags" : { }
}

I see stacktrace in log (I added some printf to investigate what happens with these values):

field |value|
value |1|
field |value_meta|
value |{"error":"Pattern match failed! ".*v3.0.*" not in "{"version": {"status": "stable\|
field | \"updated\": \"2015-03-30T00:00:00Z\", \"media-types\": [{\"base\": \"application/json\", \"type\": \"application/json\"}], \"id\": \"v3.4\", \"links\": [{\"href\": \"http://localhost:35774/\", \"rel\": \"self\"}]}}\""}"|
panic: unsupported value type during encode fields: <nil>

goroutine 46 [running]:
github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc208104180, 0xc20813a0c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/zielonkam/gocodez/src/github.com/influxdb/influxdb/tsdb/shard.go:582 +0xf3b
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc208110240, 0xc2081461b0, 0x1, 0x1, 0x0, 0x0)
    /home/zielonkam/gocodez/src/github.com/influxdb/influxdb/tsdb/shard.go:145 +0x427
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080376c0, 0x1, 0xc2081461b0, 0x1, 0x1, 0x0, 0x0)
    /home/zielonkam/gocodez/src/github.com/influxdb/influxdb/tsdb/store.go:260 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x1, 0x1, 0xc2081461b0, 0x1, 0x1)
    /home/zielonkam/gocodez/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /home/zielonkam/gocodez/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b





panic: unsupported value type during encode fields: <nil>

goroutine 43 [running]:
github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2080ed5f0, 0xc2081193b0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc2080fe240, 0xc2080ed220, 0x1, 0x1, 0x0, 0x0)
    /root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tsdb/shard.go:142 +0x427
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc208065800, 0x2, 0xc2080ed220, 0x1, 0x1, 0x0, 0x0)
    /root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tsdb/store.go:256 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x2, 0x1, 0xc2080ed220, 0x1, 0x1)
    /root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cluster/points_writer.go:213 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cluster/points_writer.go:243 +0x2f

I checked that if I escape commas in field as below then influxdb is working properly:

{
  "database" : "mmm",
  "retentionPolicy" : "",
  "points" :  [ {
    "measurement" : "http_status",
    "tags" : {
      "service" : "identity-service",
      "hostname" : "host1",
      "url" : "http://localhost:35774/"
    },
    "timestamp" : "2015-06-16T09:28:26.000Z",
    "fields" : {
      "value_meta" : "{\"error\":\"Pattern match failed! \\\".*v3.0.*\\\" not in \\\"{\\\"version\\\": {\\\"status\\\": \\\"stable\\\"\\, \\\"updated\\\": \\\"2015-03-30T00:00:00Z\\\"\\, \\\"media-types\\\": [{\\\"base\\\": \\\"application/json\\\"\\, \\\"type\\\": \\\"application/json\\\"}]\\, \\\"id\\\": \\\"v3.4\\\"\\, \\\"links\\\": [{\\\"href\\\": \\\"http://localhost:35774/\\\"\\, \\\"rel\\\": \\\"self\\\"}]}}\\\"\"}",
      "value" : 1.0
    }
  } ],
  "tags" : { }
}

[beckettsean]

@michal800106 I'm not sure I understand the problem. Commas have special syntactical meaning in JSON so it makes sense to me that they would need to be escaped, especially in a nested JSON array like you provide.

In any event the JSON protocol is deprecated in favor of the line protocol.

[michal800106]

@beckettsean
Ok, But there is no any warning. Influxdb stops with panic error. It's not good shutdown db when somebody tries insert wrong value into. And additionally in rc28 it was working without escaping commas. It has been changed.

[beckettsean]

@michal800106 it's a good point that it shouldn't panic. I'll re-open this to make sure that while the JSON protocol code is still around it doesn't panic in this instance. The regression is unfortunate, looks like there was a gap in our test suite related to nested commas in the JSON block.

[michal800106]

I prepared some fix for this error maybe it's not the best but is working. Could you take a look at it?
fix_escape_field_value

[michal800106]

@beckettsean Have you looked at my proposition of fix #3043?

[michal800106]

@beckettsean I debug deeper and I made a new commit after rebasing to your master branch.
The result is in a new pull request #3073