Custom Domains and Anonymous Access on Elastic Cloud

Two common requests for Elastic Cloud are:

Custom domain names: Rather than using https://<UUID>.<region>.<cloud-provider>.cloud.es.io:9243 you might want to access Kibana or Elasticsearch on https://mydomain.com.
Anonymous Kibana access: Read-only access to dashboards or canvas for simple sharing without logging in.

Neither are currently possible, but there are workarounds. This post walks you through the details plus the example repository contains a complete setup with nginx using Terraform and Ansible for the deployment.

Custom Domains #

Before jumping into the solution further down, you need to be aware of one thing: The endpoints that you get for Kibana and Elasticsearch on Elastic Cloud are proxies. Those route around unavailable nodes or allow no-downtime scaling and upgrading. Elastic Cloud Enterprise — the on-premise sibling of Elastic Cloud — explains the concepts in more detail, which are mostly the same for both products:

Proxies [are] mapping deployment IDs that are passed in request URLs for the container to the actual Elasticsearch cluster nodes and other instances.

Here is the DNS information from dig for the Elasticsearch endpoint. It shows what is happening behind the scene with some obfuscation (<UUID> and <XXX>) and shortening:

$ dig <UUID>.eu-west-1.aws.found.io

;; QUESTION SECTION:
;<UUID>.eu-west-1.aws.found.io. IN A

;; ANSWER SECTION:
<UUID>.eu-west-1.aws.found.io. 19 IN CNAME proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com.
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 52.30.133.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 34.251.157.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 52.215.23.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 54.171.35.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 52.213.135.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 52.19.192.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 34.254.88.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 19 IN A 52.208.205.<XXX>

;; Query time: 62 msec
;; MSG SIZE  rcvd: 283

The first idea you might have is to set up a CNAME for <UUID>.eu-west-1.aws.found.io, which is adding one more CNAME hop:

$ dig fail.xeraa.wtf

;; QUESTION SECTION:
;fail.xeraa.wtf. IN A

;; ANSWER SECTION:
fail.xeraa.wtf. 300 IN CNAME <UUID>.eu-west-1.aws.found.io.
<UUID>.eu-west-1.aws.found.io. 60 IN CNAME proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com.
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 52.50.143.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 34.252.164.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 54.77.147.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 54.72.42.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 52.213.233.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 34.249.171.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 52.30.120.<XXX>
proxy-production-v4-<XXX>.eu-west-1.elb.amazonaws.com. 60 IN A 52.19.30.<XXX>

;; Query time: 65 msec
;; MSG SIZE  rcvd: 311

Don’t forget to set the port, which is specific to Elastic Cloud, but connecting to the CNAME fails with the following error:

$ curl https://fail.xeraa.wtf:9243/
curl: (60) SSL certificate problem: Invalid certificate chain

But you can ignore that with the -k parameter. Adding authentication to the request and it works:

$ curl -k -u elastic https://fail.xeraa.wtf:9243/
{
  "name" : "instance-0000000038",
  "cluster_name" : "<UUID>",
  "cluster_uuid" : "teAtTAViTy6qmBV5nOpirA",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Well, it kind of works. Especially for Kibana and your users, you will need a better solution that avoids invalid certificates.

For that, you can use your own proxy to forward requests if you propagate the deployment ID. The proxy either needs to set the Host header with the full ID as a prefix or the custom X-Found-Cluster header with the ID; which one to use is a matter of preference.

If you are confused by the name Found: This was the company providing Elasticsearch as a Service before joining Elastic in 2015, and the header never changed.

Proxying with nginx #

I’m picking nginx as an example here, but if you prefer HAProxy or Apache HTTP, you can translate the configuration directly to those.

The only two things you need to configure are where the request should be redirected (proxy_pass) and the X-Found-Cluster header. Both are using the same UUID that is part of your cluster’s endpoint:

server {
    listen 443 ssl;
    server_name elasticsearch.xeraa.wtf;

    include /etc/nginx/tls.conf;

    location / {
        proxy_pass        https://<UUID>.eu-west-1.aws.found.io:9243/;
        proxy_set_header  X-Found-Cluster <UUID>;
    }
}

If the DNS and TLS settings are correct, this will now work without errors. Be sure to remove the port now since this is proxying from the default TLS port 443 to the cluster:

$ curl -u elastic https://elasticsearch.xeraa.wtf/
{
  "name" : "instance-0000000038",
  "cluster_name" : "<UUID>",
  "cluster_uuid" : "teAtTAViTy6qmBV5nOpirA",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

And the same from a browser: Proxied Elasticsearch

Speaking of TLS settings, I’m relying on ssl-config.mozilla.org, which does the hard work for you. Specifically, you have to use the current intermediate settings that include TLS 1.2, since TLS 1.3 only would make the curl requests fail with:

curl: (35) A bad protocol version was encountered

The same settings work for Kibana, but you must use the UUID of the Kibana endpoint:

server {
    listen 443 ssl;
    server_name kibana.xeraa.wtf;

    include /etc/nginx/tls.conf;

    location / {
        proxy_pass        https://<kibana-UUID>.eu-west-1.aws.found.io:9243/;
        proxy_set_header  X-Found-Cluster <kibana-UUID>;
    }
}

And with that, everything works as expected: Proxied Kibana

Including an A+ on the SSL Server Test: SSL Server Test result

Anonymous Access #

While Elastic Cloud doesn’t support anonymous access, sometimes you want to make a dashboard available without requiring a login. Proxies give you another nice workaround here. Passing the Authorization header, you can do the authentication on the fly. The full proxy setting is then:

server {
    listen 443 ssl;
    server_name dashboard.xeraa.wtf;

    include /etc/nginx/tls.conf;

    location / {
        proxy_pass        https://<kibana-UUID>.eu-west-1.aws.found.io:9243/;
        proxy_set_header  X-Found-Cluster <kibana-UUID>;
        proxy_set_header  Authorization "Basic YWRtaW46c2VjcmV0";
    }
}

Only what is YWRtaW46c2VjcmV0? It’s the Base64 representation of admin:secret — the username, colon, password of the user to log into Kibana. You can generate this on the command line:

$ echo -n admin:secret | base64
YWRtaW46c2VjcmV0

And now, to the challenging part of this section: Setting up security correctly and picking stronger credentials than me here. Unauthenticated access to Elasticsearch or Kibana is by far the most common security problem of Elasticsearch clusters. For this reason, there is no unauthenticated access on Elastic Cloud — no exceptions. Yet this is what this example is effectively doing. So it’s essential to keep security as much locked down as possible. The sample setting I’m using is in security_role_anonymous.json, which explicitly allows specific indices and limits the available Kibana features. Be sure that this is the data you want to share before setting up such a configuration.

Using a different browser or a new private window, you’re logged in automatically. Note the limited set of available Kibana apps in the menu on the left-hand side: Locked down Kibana

There is one final and rather minor point in terms of security. The current setup reveals headers from the proxied Elasticsearch or Kibana node:

$ curl -i https://dashboard.xeraa.wtf
HTTP/1.1 302 Found
Server: nginx
Date: Sat, 18 Apr 2020 11:42:32 GMT
Content-Length: 0
Connection: keep-alive
Cache-Control: no-cache
Kbn-License-Sig: 1c6f869f88b0168bc8c7cc5575a02d0f60b77386ed5d9a6677886b911afde3e5
Kbn-Name: 529de565a912
Kbn-Xpack-Sig: 2e2cefa1c7112a5bd2d3c4b785ae0c65
Location: /spaces/enter
X-Cloud-Request-Id: IeO3zv6PQKG1vHApVH9VZA
X-Found-Handling-Cluster: <kibana-UUID>
X-Found-Handling-Instance: instance-0000000011
X-Found-Handling-Server: <IP>
Strict-Transport-Security: max-age=63072000

Especially if you want to provide public access, you might want to strip out those headers from the response. That is again simple with nginx:

proxy_hide_header Kbn-License-Sig;
proxy_hide_header Kbn-Name;
proxy_hide_header Kbn-Xpack-Sig;
proxy_hide_header X-Cloud-Request-Id;
proxy_hide_header X-Found-Handling-Cluster;
proxy_hide_header X-Found-Handling-Instance;
proxy_hide_header X-Found-Handling-Server;

And now the response doesn’t leak any internal details:

$ curl -i https://dashboard.xeraa.wtf
HTTP/1.1 302 Found
Server: nginx
Date: Sat, 18 Apr 2020 12:11:03 GMT
Content-Length: 0
Connection: keep-alive
Cache-Control: no-cache
Location: /spaces/enter
Strict-Transport-Security: max-age=63072000

Conclusion #

Custom domains or aliases are in the works, but it will take some more time until they will be available. In the meantime, you can use a proxy, as described above. The same goes for anonymous access in Kibana, but please use it with care in terms of security. The full setup is available on GitHub.

The one big downside with this approach is that you have added a new single point of failure — the proxy. Unfortunately, most load balancers in the cloud only support redirecting to one of your own instances or IPs and a small subset of HTTP headers; that isn’t working for our requirements. The apparent solution could now be a load balancer with multiple nginx proxies behind it. But another load balancer and proxy layer on top of the one of Elastic Cloud feels like 🐢 all the way down. Especially for Kibana, I’m willing to accept a more straightforward yet imperfect solution.

PS: If you are running Elastic Cloud Enterprise or Elastic Cloud on Kubernetes and have control over the TLS certificate, you don’t need a proxy for a custom domain. Instead, add the domain or a wildcard to the certificate and point the CNAME to the cluster. You still need a proxy for anonymous access, though.