Elasticsearch behind haproxy

[simonmorley]

We have a problem that seems to relate to #566 and #189. I cannot access via haproxy with multiple servers.

Initially we exposed a single elastic search server the world a reverse nginx proxy with basic auth. That and the latest kibana worked nicely together, once.

Until... having grown quickly, we've had to load balance the elastic servers with haproxy and now have four on 8443 (443 public).

We've load balanced kibana (which runs on the es servers).

No matter what I try, kibana cannot access the es servers through the proxy with basic auth in place. Standard errors:

2013/10/29 01:09:15 [error] 12331#0: *17 no user/password was provided for basic authentication, client: 172.0.100.14, server: es.xx.com, request: "OPTIONS /_aliases HTTP/1.1", host: "elastic.polkaspots.com", referrer: "https://logs.xx.com/"

I've tried with the same url in config.js as the browser but that's not working, doesn't throw an error because it's loading itself.

The only way I have managed it was to remove auth from elastic search completely. However, I can't leave es open to the world!

I can't hardcode the username / password either as everyone's got a different account. Plus, it's silly to do so.

Does anyone have some recommendations about this?

[bobrik]

Do you have elasticsearch and kibana on different ports?

[rashidkpc]

You're probably better off using a non-data elasticsearch node as the proxy than HAProxy. It will be more efficient as it will understand the cluster state and location of shards.

[simonmorley]

@bobrik Outside network they're both on 443. Inside they're on 8443.

@rashidkpc Fair point. We were just trying to use our existing infrastructure to prevent over complication. ES requires a pile of memory, does a non-data one require similar amounts? We don't really want to throw more beefy servers at it.

[bobrik]

@simonmorley regular data node can handle the load. Indices are sharded and queries are distributed across nodes after all. We have 2 nodes and one exposed via nginx to access from kibana.

We also have another cluster (not for analytics) with 5 nodes and if I only query 4 of them, cpu load stays uniform across nodes.

Do you have es and kibana on single hosnamte too? I think that browsers use http auth if only hostname+port is the same.

[simonmorley]

@bobrik Hey. I'm going to fire up a small, non-data instance for a few days running nginx reverse proxy and see what happens.

We tried running kibana on another cluster of servers but moved it back on to the data nodes to simplify the testing. We have tried with both same and different hosts name with limited success. I think that was more cors settings in nginx causing problems than kibana.

In an ideal world, I'd quite like kibana to have so server side options again. I feel a little nervous having anything exposed to the outside world.

[ruckc]

I agree with needing a server side option. Maybe a pseudo-reverse proxy
sinatra style wrapper that mimics the ES behavior, but allows injecting
acls/permissions on the ES REST api?

Curtis Ruck

On Tue, Oct 29, 2013 at 1:31 PM, Simon (Big Chief) Morley <
notifications@github.com> wrote:

@bobrik https://github.com/bobrik Hey. I'm going to fire up a small,
non-data instance for a few days running nginx reverse proxy and see what
happens.

We tried running kibana on another cluster of servers but moved it back on
to the data nodes to simplify the testing. We have tried with both same and
different hosts name with limited success. I think that was more cors
settings in nginx causing problems than kibana.

In an ideal world, I'd quite like kibana to have so server side options
again. I feel a little nervous having anything exposed to the outside world.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/624#issuecomment-27324782
.

[simonmorley]

Back to the point, this just isn't working the way it should for us. Maybe I've missed a major point. We can't run kibana and elasticsearch with the same hostname, using ssl on the same host.

I fired up a non-data node to load balance over haproxy. That works. Then I shoved kibana on it for good measure.

We've protected our elastic search servers with basic auth and a reverse proxy on es.xxx.com:443 as required - kibana needs access to es from the browser right? And we need to access es from outside too.

Then kibana doesn't need to use basic auth? ES is protected. That whole section in the nginx.conf is irrelevant. So I took that out and left the root path. That doesn't work.

If I disable basic auth for the es server, kibana is fine no matter what hostname I'm using. We can't do this obvs. So we put a username:password combination in the config.js which is a bad idea and actually doesn't work.

I don't see how you can have a protected es and kibana server running on the same host. Can someone explain whether I'm being plain dumb or if there's something fundamental I'm missing.

[bobrik]

You're missing that you should run kibana and es on the same hostname:port.

In our configuration we protect the whole server and proxy kibana on /es like this:

    location /es/ {
        rewrite ^/es/_aliases$ /_aliases break;
        rewrite ^/es/_nodes$ /_nodes break;
        rewrite ^/es/(.*/_search)$ /$1 break;
        rewrite ^/es/(.*/_mapping)$ /$1 break;
        rewrite ^/es/(kibana-int/.*)$ /$1 break;
        return 403;

        proxy_pass http://whatever.local:9200;
    }

Kibana runs on http://example.com and es is exposed via http://example.com/es (config.js). Browser asks password for kibana and uses it for es too.

[simonmorley]

Thanks, I'll try that and I get the point about same host port. It just seems a bit restrictive as per point earlier about some server side stuff.

In your conf js, you have no username / password then either?

[ruckc]

Ian, your example config doesn't restrict http verbs against ES (i.e.
DELETE) and could expose the ability for external modifications to the
indexes.

Curtis Ruck

On Tue, Oct 29, 2013 at 4:37 PM, Ian Babrou notifications@github.comwrote:

You're missing that you should run kibana and es on the same hostname:port.

In our configuration we protect the whole server and proxy kibana on /eslike this:

location /es/ {
    rewrite ^/es/_aliases$ /_aliases break;
    rewrite ^/es/_nodes$ /_nodes break;
    rewrite ^/es/(.*/_search)$ /$1 break;
    rewrite ^/es/(.*/_mapping)$ /$1 break;
    rewrite ^/es/(kibana-int/.*)$ /$1 break;
    return 403;

    proxy_pass http://whatever.local:9200;
}

Kibana runs on http://example.com and es is exposed via
http://example.com/es (config.js). Browser asks password for kibana and
uses it for es too.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/624#issuecomment-27341019
.

[simonmorley]

Does anyone have a secure nginx conf that they can share?

[bobrik]

@ruckc sorry, old config :)

    location /es/ {
        rewrite ^/es/_aliases$ /_aliases break;
        rewrite ^/es/_nodes$ /_nodes break;
        rewrite ^/es/(.*/_mapping)$ /$1 break;
        return 403;

        limit_except GET{
            deny all;
        }

        proxy_pass http://web245:9200;
    }

    location ~ ^/es/(.*/_search)$ {
        rewrite ^/es/(.*/_search)$ /$1 break;

        limit_except POST {
            deny all;
        }

        proxy_pass http://web245:9200;
    }

    location /es/kibana-int/ {
        rewrite ^/es/(kibana-int/.*)$ /$1 break;
        proxy_pass http://web245:9200;
    }

If you care about security, you should disable scripting too.

[simonmorley]

Ah, great. That looks so much better. Was half way through merging karmi's recipe for es::proxy with the previous one. I think yours is superior.

I guess there needs to be better documentation about actually how this integrates securely with an existing [closed] implementation, what can be removed etc.

When I get a moment I'll fork the kibana cookbook and merge this stuff.

[simonmorley]

Hello again. I've created an nginx configuration that uses most of the comments here including a few bits and pieces from the es cookbook.

Have basically lumped it all in for testing and there's a pile of repetition that needs refactoring. But it works. The cors sections are parts I really need to understand more. I'd appreciate any comments etc.

https://gist.github.com/simonmorley/7224897

[rashidkpc]

Closing this as inactive. Appears that the initial issue has been solved.