out of memory when using HTTP API

[moof2k]

Just following the getting started documentation (https://docs.influxdata.com/influxdb/v1.2/guides/writing_data/) hit an OOM error.

System configuration

• ubuntu 16.04
• influxdb 0.10.0+dfsg1-1

Dockerfile:

FROM ubuntu:16.04

RUN apt-get update

RUN apt-get install -y \
    apt-utils \
    curl \
    perl \
    python \
    less \
    telnet \
    wget

# influxdb
RUN curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
RUN echo "deb https://repos.influxdata.com/ubuntu xenial stable" | tee /etc/apt/sources.list.d/influxdb.list
RUN apt-get install -y influxdb
RUN apt-get install -y influxdb-client

Steps to reproduce:

• Create an empty database named 'test'
• Generate some test data

import math
import numpy as np

t0 = 1491004800 * 1000 * 1000 * 1000
t1 = 1491048000 * 1000 * 1000 * 1000
ts = 1 * 1000 * 1000

for t in np.arange(t0, t1, ts):
    print "u0,a0=%f %d " % (math.sin(float(t)/1000/1000/10), t)

• Upload data to influxdb over HTTP API

$ python gen_data.py > data.txt
$ wc data.txt
  43200000   86400000 1576800000 data.txt
$ curl -i -XPOST 'http://localhost:8086/write?db=test' --data-binary @data.txt

Expected behavior:

• Data is uploaded to influxdb

Actual behavior:

• influxdb crashed with OOM error

[http] 2017/04/16 05:04:21 ::1 - - [16/Apr/2017:05:04:21 +0000] POST /write?db=test HTTP/1.1 204 0 - curl/7.47.0 27455
7c4-2262-11e7-801b-000000000000 130.563<C2><B5>s
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xd89a20, 0x16)
        /usr/lib/go/src/runtime/panic.go:530 +0x90
runtime.sysMap(0xca32d90000, 0xc4fc0000, 0x437300, 0x11f5018)
        /usr/lib/go/src/runtime/mem_linux.go:206 +0x9b
runtime.(*mheap).sysAlloc(0x11da440, 0xc4fc0000, 0xc82019ed80)
        /usr/lib/go/src/runtime/malloc.go:429 +0x191
runtime.(*mheap).grow(0x11da440, 0x627e0, 0x0)
        /usr/lib/go/src/runtime/mheap.go:651 +0x63
runtime.(*mheap).allocSpanLocked(0x11da440, 0x627da, 0xc814133e00)
        /usr/lib/go/src/runtime/mheap.go:553 +0x4f6
runtime.(*mheap).alloc_m(0x11da440, 0x627da, 0x100000000, 0x11db8d0)
        /usr/lib/go/src/runtime/mheap.go:437 +0x119
runtime.(*mheap).alloc.func1()
        /usr/lib/go/src/runtime/mheap.go:502 +0x41
runtime.systemstack(0x7fce109f6e28)
        /usr/lib/go/src/runtime/asm_amd64.s:307 +0xab
runtime.(*mheap).alloc(0x11da440, 0x627da, 0x10100000000, 0xc820012000)
        /usr/lib/go/src/runtime/mheap.go:503 +0x63
runtime.largeAlloc(0xc4fb3eff, 0x1, 0x0)
        /usr/lib/go/src/runtime/malloc.go:766 +0xb3

[moof2k]

Updated to the latest influxdb by modifying my Dockerfile, problem still exists in latest.

FROM ubuntu:16.04

RUN apt-get update

RUN apt-get install -y \
    apt-utils \
    curl \
    perl \
    python \
    less \
    telnet \
    wget

# influxdb
RUN wget https://dl.influxdata.com/influxdb/releases/influxdb_1.2.2_amd64.deb
RUN dpkg -i influxdb_1.2.2_amd64.deb
RUN chmod 755 /usr/lib/influxdb/scripts/init.sh
RUN ln -s /usr/lib/influxdb/scripts/init.sh /etc/init.d/influxdb

It seems odd to me that 43200000 datapoints (~330MB worth of doubles) would require the system to allocate >8GB RAM.

[hpbieker]

You are writing the value to a tag a0, try writing to a field instead. Just replace the comma between u0 and a0 with a space.

[moof2k]

You are writing the value to a tag a0, try writing to a field instead. Just replace the comma between u0 and a0 with a space.

Ooops! Good catch. However it still crashes on an out of memory.

Here's the new script:

import math
import numpy as np

t0 = 1491004800 * 1000 * 1000 * 1000
t1 = 1491048000 * 1000 * 1000 * 1000
ts = 1 * 1000 * 1000

for t in np.arange(t0, t1, ts):
    print "u0 a0=%f %d " % (math.sin(float(t)/1000/1000/10), t)

[jwilder]

You're curl statement is also posting all 43M points in a single batch. You'll need to write those in much smaller batches (5-10k usually).

[hpbieker]

I guess Influx should prevent situations like this were it cannot handle the request by by failing the write instead of crashing the server.

[stuartcarnie]

@jwilder we have middleware to catch panics and log the error, so the server does not crash completely. Currently, the recovery middleware returns a 200 OK with no body, however it should return a 500 Internal Server Error along with any error information.

To prevent abuse of the HTTP API, we can check the Content-Length header and add a limit reader to prevent excessive memory usage for write requests. We would also add a new configuration option to allow users to adjust the max length, if necessary.