• / download-cloud Download (8 MB) hash elasticsearchkibanaelastic-stackupgrade

While you can find the full list of changes and especially the breaking changes in the release notes, we focus on some interesting ones and dive a little deeper into them.

If the slides alone are not providing enough context you can follow along with a more detailed description below.

Demo Environment

Before we get started, we need to cover the demo environment — a simple three node cluster with Docker Compose.

Live Upgrade

Starting with a 6.7 cluster we are upgrading to 7.0 and then even prepare for the 8.0 migration 😉.

First, we check that everything is up, we have the right version, and a green cluster:

GET /

GET _cat/nodes?h=id,version,master,name&v

GET _cluster/health

POST test/_doc
{
  "name": "Elasticsearch 6.7"
}

Next, we check the upgrade assistant in Kibana (local link) — since this is a fresh cluster nothing should need to change. We disable replica shard allocation because the upgraded nodes will come back up quickly and this would just add unnecessary work. But we allow new_primaries, because Kibana 7 needs to create a new index and will not start up otherwise. Finally, we do a sync flush to speed up recovery times of the nodes:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "new_primaries"
  }
}

POST _flush/synced

We change the Elastic Stack version in the .env file and upgrade the first node with docker-compose up -d --no-deps elasticsearch3. We can check the cluster during the upgrade and still keep querying documents:

GET _cat/nodes?h=id,version,master,name&v

GET test/_search

We follow the same procedure for elasticsearch2, elasticsearch1, and kibana. Once done we re-enable shard allocation, add a document, and see that everything is still working as expected:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": null
  }
}

POST test/_doc
{
  "name": "Elasticsearch 7.0"
}

GET test/_search

Finally, we check the upgrade assistant again, reindex the test index since data can only be read from one version back, and see that everything is ready for our (very) future 8.0 upgrade:

GET _cat/indices
GET _cat/aliases?v

Zen2

Cluster coordination rethought and rebuilt. Read the blog post about Zen2 if you want a more detailed description than the slides.

Types or Rather No Types

Both the past and the next steps in Elasticsearch’s journey of getting rid of types. We create a simple mapping and notice the absence of a type. But will a query like PUT no_type/2 work now?

# Types
PUT no_type
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

PUT no_type/_doc/1
{
  "@timestamp": "2019-05-08"
}

GET no_type/_search

PUT no_type/2
{
  "@timestamp": "2019-05-09"
}

No, this will not work and might be a surprise, but the documentation provides an answer:

In 7.0, _doc represents the endpoint name instead of the document type. The _doc component is a permanent part of the path for the document index, get, and delete APIs going forward, and will not be removed in 8.0.

Additional details are available on the Elastic blog about going typeless.

Optimizations

For more details than in the slides see the faster retrieval of top hits blog post, but the following demo shows how they work in practice. First, we add the flight demo data set in Kibana, so we have something to work with and then run through the following queries to see the differences in results:

GET kibana_sample_data_flights/_search

GET kibana_sample_data_flights/_search
{
  "track_total_hits": 10
}

GET kibana_sample_data_flights/_search
{
  "track_total_hits": true
}

index.search.idle.after should be relatively self explaining. The main caveat is that it will only work if you are using the default index.search.idle.after (one second). That is because skipping the refresh is cancelled on the next search and that search operation will block until the next refresh happens automatically. Blocking longer than one second is probably not what anybody wants, so that combination of settings is required.

New Defaults

A single primary shard is the new default. The motivation is that over-sharding is a very common problem and a single shard will normally make most sense for beginners. Experts will know how shards work and will reconfigure them as needed.

Adaptive Replica Selection is now on by default.

Less Heap

Frozen indices were first released in 6.6, but are still worth of being in the highlights. We run through the following queries to see frozen indices in action and while the force merge is not required it is recommended:

POST frozen/_doc
{
  "name": "Boaz"
}
POST frozen/_doc
{
  "name": "Pablo"
}

GET frozen/_search

POST frozen/_forcemerge?max_num_segments=1

POST frozen/_freeze

GET frozen/_search

Suddenly all our data seems to be gone, but this is for our protection. You need to be explicit about expensive queries:

GET frozen/_search?ignore_throttled=false

GET _cat/indices/frozen?v&h=health,status,index,pri,rep,docs.count,store.size

GET _cat/thread_pool/search_throttled?v&h=node_name,name,active,rejected,queue,completed&s=node_name

GET frozen/_search?ignore_throttled=false&pre_filter_shard_size=1

POST frozen/_doc
{
  "name": "Daliya"
}

Adding new records will fail as long as the index is frozen. This is how to check and change it:

GET frozen/_settings?flat_settings=true

POST frozen/_unfreeze

POST frozen/_doc
{
  "name": "Daliya"
}

GET frozen/_search

Coming back to the flight demo set we can check this on a larger scale. First, we look at the dashboard for that and see that we need to enable querying of frozen indices there as well under Advanced Settings (local link) called Search in frozen indices. Then we look at the thread staticstics and how the dedicated thread pool is at work:

POST kibana_sample_data_flights/_freeze

GET _cat/thread_pool/search_throttled?v&h=node_name,name,active,rejected,queue,completed&s=node_name

Additional details in the related blog post.

In addition, real-memory circuit breakers have landed and you can find more details than in the slides in the relevant blog post.

Kibana

The new look of Kibana is driven by the Elastic UI Framework while the de-Angularization is still in progress. While this is great for reuse, note the warning in the repository:

🚨 While open source, the intended consumers of this repository are Elastic products. Read the FAQ for details.