Delete by query doesn't work in persistent http connection

[bobrik]

This is probably connected with #2665.

Here's test-case in php:

<?php

$conn = curl_init();
$host = 'http://127.0.0.1:9200';

function send($conn, $host, $method, $path, $query = array(), $body = null) {
    // uncomment the next line to make new connection for every request
    // $conn = curl_init();

    $url = $host . $path . '?' . http_build_query($query);

    curl_setopt($conn, CURLOPT_URL, $url);

    if ($body) {
        curl_setopt($conn, CURLOPT_POSTFIELDS, $body);
    }

    curl_setopt($conn, CURLOPT_CUSTOMREQUEST, $method);

    ob_start();
    curl_exec($conn);
    $result = ob_get_clean();

    echo $method . ' ' . $url . ' ' . $body."\n";
    echo '> ' . $result . "\n";

    return $result;
}

send($conn, $host, "DELETE", '/elastica_test/');
send($conn, $host, "PUT", '/elastica_test/', array(), '{"index":{"number_of_shards":1,"number_of_replicas":0}}');
send($conn, $host, "PUT", '/elastica_test/test/1', array(), '{"name":"ruflin nicolas"}');
send($conn, $host, "PUT", '/elastica_test/test/2', array(), '{"name":"ruflin"}');
send($conn, $host, "PUT", '/elastica_test/test/3', array(), '{"name":"ian"}');
send($conn, $host, "POST", '/elastica_test/_refresh');
send($conn, $host, "POST", '/elastica_test/_search', array(), '{"query":{"query_string":{"query":"ruflin*"}}}');
send($conn, $host, "POST", '/elastica_test/_search', array(), '{"query":{"query_string":{"query":"nicolas"}}}');
// comment the next request to fix everything
send($conn, $host, "POST", '/elastica_test/_search', array(), '{"query":{"query_string":{"query":"ian"}}}');
send($conn, $host, "DELETE", '/elastica_test/_query', array('q' => 'nicolas'));
send($conn, $host, "POST", '/elastica_test/_refresh');
send($conn, $host, "POST", '/elastica_test/_search', array(), '{"query":{"query_string":{"query":"ruflin*"}}}');

The last query should respond with just one hit, but it returns document that was removed. This works as it should if new connection is made for every request (see function send), it also works if I comment search (sic!) request before delete-by-query.

I'm sorry for php, I couldn't make it work with curl or nc.

[bobrik]

Any news about this issue?

Using new connection for every single request doesn't look like a good idea.

[bobrik]

On 0.9.11 you could spot in logs:

[2014-03-11 21:14:54,830][DEBUG][action.deletebyquery     ] [Nick Fury] [elastica_test][0], node[v4P07UGDRMeJm8z-q3fWlg], [P], s[STARTED]: Failed to execute [delete_by_query {[elastica_test][], query [{"query":{"query_string":{"query":"ian"}}}]}]
org.elasticsearch.index.query.QueryParsingException: [elastica_test] No query registered for [query]
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:211)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:284)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:225)
        at org.elasticsearch.index.shard.service.InternalIndexShard.prepareDeleteByQuery(InternalIndexShard.java:426)
        at org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnPrimary(TransportShardDeleteByQueryAction.java:113)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

And this is after:

send($conn, $host, "DELETE", '/elastica_test/_query', array('q' => 'nicolas'));

where query is different. Clearly requests are messed up.

[kimchy]

it doesn't seem like the delete by query relates to persistent http connections from the client? Maybe the client resets the connections on errors and it simply tries to reconnect and fail all the time?

Note that the format of the body of delete by query changed between 0.90. and 1.0, maybe the failure is because of that.

[clintongormley]

send($conn, $host, "DELETE", '/elastica_test/_query', array('q' => 'nicolas'));

Judging by your other request, that q is being sent in the body, not in the query string as it should be.

Alternatively, you could do:

DELETE /elastica_test/_query
{ 
    "query_string":  {
        "query": "nicholas"
    }
}

And from 1.0 onwards:

DELETE /elastica_test/_query
{ 
    "query: {
        "query_string":  {
            "query": "nicholas"
        }
    }
}

[bobrik]

@clintongormley it's query string, body is the 6th parameter.

Here is output for 1.0.0:

DELETE http://127.0.0.1:9200/elastica_test/?
> {"acknowledged":true}
PUT http://127.0.0.1:9200/elastica_test/? {"index":{"number_of_shards":1,"number_of_replicas":0}}
> {"acknowledged":true}
PUT http://127.0.0.1:9200/elastica_test/test/1? {"name":"ruflin nicolas"}
> {"_index":"elastica_test","_type":"test","_id":"1","_version":1,"created":true}
PUT http://127.0.0.1:9200/elastica_test/test/2? {"name":"ruflin"}
> {"_index":"elastica_test","_type":"test","_id":"2","_version":1,"created":true}
PUT http://127.0.0.1:9200/elastica_test/test/3? {"name":"ian"}
> {"_index":"elastica_test","_type":"test","_id":"3","_version":1,"created":true}
POST http://127.0.0.1:9200/elastica_test/_refresh?
> {"_shards":{"total":1,"successful":1,"failed":0}}
POST http://127.0.0.1:9200/elastica_test/_search? {"query":{"query_string":{"query":"ruflin*"}}}
> {"took":31,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":2,"max_score":1.0,"hits":[{"_index":"elastica_test","_type":"test","_id":"1","_score":1.0, "_source" : {"name":"ruflin nicolas"}},{"_index":"elastica_test","_type":"test","_id":"2","_score":1.0, "_source" : {"name":"ruflin"}}]}}
POST http://127.0.0.1:9200/elastica_test/_search? {"query":{"query_string":{"query":"nicolas"}}}
> {"took":3,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max_score":0.8784157,"hits":[{"_index":"elastica_test","_type":"test","_id":"1","_score":0.8784157, "_source" : {"name":"ruflin nicolas"}}]}}
POST http://127.0.0.1:9200/elastica_test/_search? {"query":{"query_string":{"query":"ian"}}}
> {"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max_score":1.4054651,"hits":[{"_index":"elastica_test","_type":"test","_id":"3","_score":1.4054651, "_source" : {"name":"ian"}}]}}
DELETE http://127.0.0.1:9200/elastica_test/_query?q=nicolas
> {"_indices":{"elastica_test":{"_shards":{"total":1,"successful":1,"failed":0}}}}
POST http://127.0.0.1:9200/elastica_test/_refresh?
> {"_shards":{"total":1,"successful":1,"failed":0}}
POST http://127.0.0.1:9200/elastica_test/_search? {"query":{"query_string":{"query":"ruflin*"}}}
> {"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":2,"max_score":1.0,"hits":[{"_index":"elastica_test","_type":"test","_id":"1","_score":1.0, "_source" : {"name":"ruflin nicolas"}},{"_index":"elastica_test","_type":"test","_id":"2","_score":1.0, "_source" : {"name":"ruflin"}}]}}

@kimchy there are no errors as you can see.

[clintongormley]

@bobrik Sorry yes, I misread.

The curl docs say that if you plan on reusing the same connection for a GET after a POST, then you should reset it with CURLOPT_HTTPGET: http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTHTTPGET

I wonder if the same thing applies to a DELETE?

Perhaps try it?

[bobrik]

I've put tcp proxy in the middle and found out that curl reused body from previous requests if they were not cleared explicitly. Gonna fix that it ruflin/Elastica then.

[ruflin]

@clintongormley Thanks for the input. It's good to learn something new every day ;-)