Tags: java, log, pattern, talk

Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.

This talk presents a simple Java application and multiple logging patterns with with their advantages and disadvantages, so you can pick the one that fits your organization best:

• Parse: Take the log files of your applications and extract the relevant pieces of information.
• Send: Add a log appender to send out your events directly without persisting them to a log file.
• Structure: Write your events in a structured file, which you can then centralize.
• Containerize: Keep track of short lived containers and configure their logging correctly.
• Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.

Each pattern has its own demo with the Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.

Demo: Try out the example code of the talk.

Tags: observability, devops, talk

You know the drill: DevOps is using tool(s) X. So obviously, observability can be solved by throwing some tools together as well; generally logs, metrics, and traces often called the pillars of observability.

But observability is not a tool — it is a property of a system. Moving from many small blackboxes to a more holistic view of your system. It includes tools, but not exactly three distinct features (especially if your solution happens to support those). For example, if half your user base cannot access your service because of some bad DNS settings and external health checks are not part of your pillars, you are none the wiser.

This is not (just) a rant, but a look at the actual value to be added and some approaches to it. Like turning your logs into richer events that align with your business. Which is not solved by fancy tools alone.

Demo: If you are looking for the actual tools and where the Elastic Stack can help you, take a look at the slides More Observable Systems with the Elastic Stack.

these slides are fucking phenomenal. i love how @xeraa draws the sharp comparison between observability and devops terminology, and how in both cases vendors are hawking what cannot be bought. 👏📈 https://twitter.com/dnsmichi/status/1339111875691356160

  Charity Majors, @mipsytipsy — 2020-12-18

@mipsytipsy It is a great learning curve to go beyond logs, traces and metrics for observability💡

Next to your & @lizthegrey ’s insights, I highly recommend @xeraa ’s talk: https://xeraa.net/talks/observability-is-more-than-logs-metrics-traces/ :)

  Michael Friedrich 🌈, @dnsmichi — 2020-12-16

Tags: search, elasticsearch, talk

Today’s applications are expected to provide powerful full-text search. But how does that work in general and how do I implement it on my site or in my application?

Actually, this is not as hard as it sounds at first. This talk covers:

• How full-text search works in general and what the differences to databases are.
• How the score or quality of a search result is calculated.
• How to handle languages, search for terms and phrases, run boolean queries, add suggestions, work with ngrams, and more with Elasticsearch.

We will run all the queries live and explore the possibilities for your use-case.

Tags: docker, kubernetes, helm, operator, talk

“Containers are the new ZIP format to distribute software” is a fitting description of today’s development world. However, it is not always that easy and this talk highlights the development of Elastic’s container strategy over time:

• Docker images: A new distribution model.
• Helm Chart: Going from demo to production.
• Kubernetes Operator: Full control with upgrades, scaling,…

Besides the strategy we are also discussing specific technical details and hurdles that appeared during the development. Or why the future will be a combination of Helm Chart and Operator (for now).

Tags: security, auditd, auditbeat, siem, talk

The Linux Audit daemon is responsible for writing audit records to the disk, which you can then access with ausearch and aureport. However, it turned out that parsing and centralizing these records is not as easy as you would hope. Elastic’s new Auditbeat fixes this by keeping the original configuration, but ships them to a centralized location where you can easily visualize all events. You can also use Auditbeat to detect changes to critical files, like binaries and configuration files, and identify potential security policy violations.

This talk shows you what can you do to discover changes, events, and potential security breaches as soon as possible on interactive dashboards. Additionally, we are combining Auditd events with logs, which are security relevant, and explore them in Elastic’s free SIEM.

Demo: Take a look at the example code for the talk.

Tags: kubernetes, debug, elastic-stack, talk

Your Kubernetes app is down. Your users start ranting on Twitter. Your boss is standing right behind you. What do you do?

This talk walks you through a live debugging session without panicking:

• Where does your monitoring point you?
• Is there anything helpful in the logs?
• Can you get more details from your application’s traces?
• What do your health checks say?
• What the heck is even deployed?

We are using the Elastic Stack in this demo with a special focus on its Kubernetes integration with metadata enrichment and autodiscovery in combination with APM / tracing, metrics, logs, and health checks.

Tags: elasticsearch, java, talk

Elasticsearch is one of the most widely used Java projects out there. While many people know the REST interface and how to use it, this talk takes a look under the hood and picks up some of the more interesting Java topics there:

• Why and how Elasticsearch is always bundling the latest JDK with the default binary; also what lessons can be learned from Docker to Java distributions issues.
• What Java features is Elasticsearch (not) using; also what are the requirements for building / testing and running.
• While the Java Security Manager served Elasticsearch extremely well for Log4Shell and other security issues, what are the next steps after its deprecation to replace it.
• Elasticsearch’s Gradle setup is a monster but a powerful one — how does it keep the developers productive and what are the most important changes around performance.

Tags: log4shell, log4j, talk

Looking behind the immediate pain of Java’s now infamous logging library Log4j:

• How do the vulnerabilities work and how to exploit them? Also, how critical are they in reality?
• Why is it relatively complex to detect and evaluate for both custom and off-the-shelf software?
• What does it mean for a widely used product like Elasticsearch?
• How can you monitor exploitation attempts?

PS: Full blog post for Log4Shell and Elasticsearch.

Tags: versioning, talk

Semantic Versioning seems to be THE solution every time you talk about releasing software. At least in theory. Because practice is often a lot more complex. Let’s cover the basics and complexities in five minutes!

• What is and isn’t covered by SemVer?
• What’s the asymmetry in it?
• What does the versioning schema of popular products like Java, Kubernetes, Terraform, Ansible, or MySQL look like?
• What are widespread alternatives?

Tags: architecture, talk

You only have one repository, build artifact, deployment step, process, and version that you can run locally but still scale horizontally. Sounds too good to be true and you think it must be a theoretical example? Far from it — you might even be running one yourself. Though there are of course some tradeoffs that we are covering as well.

Tags: opentelemetry, observability, talk

OpenTelemetry, or OTel for short, is taking over the observability world to uncover all kinds of performance problems or errors. Let’s look at that hands-on.

This talk dives into the background of OTel including the technologies it is building on (or replacing). It also takes a look at what auto-instrumentation can and cannot achieve. As well as the latest developments beyond tracing with metrics and the first steps toward logging.

This talk also takes a look at different approaches to integration — the Spring ecosystem is wrapping most of OTel into Micrometer while other frameworks are not. What are the pros and cons of these different integrations for the developers?

Tags: elasticsearch, log, talk

When you hear Elasticsearch and logs, you probably assume we want to store some application or infrastructure logs in the Elastic Stack. But this talk focusses on the logs that Elasticsearch produces itself: How to configure and use server, slow, GC, deprecation, and audit logs.

We discuss log levels and hierarchies, the options for changing log levels in Elasticsearch, and where you can access them. Along with some tricks and best practices — for example are you already using X-Opaque-Id?

And of course we are going to store the logs in the Elastic Stack itself to close the loop.

Tags: work, home, distributed, talk

You have never worked from home or only one day per week? Suddenly many of us are thrown into this and it is not without challenges. Elastic was founded in the “work from anywhere” style in 2012 and is still doing it with 2,000 employees; I have been working exclusively from home for four years now. Here are some tips and tricks and we will also have time for Q&A:

• How do you structure a workday to stay efficient and motivated?
• How do you handle coordination, meetings, and the social aspect of work?
• What are some recommended tools?
• How can you turn this into a strength for software projects?

Once you have worked from home for a while it is hard to imagine going back to an office — if done the right way.

PS: While this talk focuses on the perspective for individuals, Scaling Distributed Teams looks more into the challenges around company growth and Building Distributed Systems in Distributed Teams on how to get started as a company.

Tags: elasticsearch, intro, talk

So you want to get started with Elasticsearch? Let’s kick this off together. We are taking a quick look at:

• Architecture: Cluster, nodes, shards — how does this even work?
• Full-text search: The foundation of all other features — what makes it different to databases and how to work with text analysis, relevance,…
• Going beyond: Search with aggregations, geo features, and where to look next.

In the end you should have a general overview and enough pointers to dive further into specific areas with the help of the documentation.

Tags: observability, elastic-stack, talk

We are deepening the insights you can get from your application with the Elastic Stack with every release. Let’s discuss some of the latest developments:

• ECS and the new ECS logging libraries.
• Connecting your logs with APM / distributed tracing.
• Monitoring your cloud infrastructure.
• Better Kubernetes insights and health checks with a look into the future.

Tags: php, log, pattern, talk

Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.

This talk presents a simple PHP application and multiple logging patterns with with their advantages and disadvantages, so you can pick the one that fits your organization best:

• Parse: Take the log files of your applications and extract the relevant pieces of information.
• Send: Add a log appender to send out your events directly without persisting them to a log file.
• Structure: Write your events in a structured file, which you can then centralize.
• Containerize: Keep track of short lived containers and configure their logging correctly.
• Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.

Each pattern has its own demo with the Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.

Demo: Try out the example code of the talk.

Tags: elasticsearch, scale, cluster, talk

Elasticsearch is highly scalable, but some recent additions make it easier to tie everything together. In this talk we discuss:

• Lifecycle Management or how to automate having a multi-tier architecture as well as automated snapshots and how this actually works in the background.
• Cold Storage for much higher storage density per Elasticsearch node and beyond.
• Rollups or how to aggregate metrics over time.

The goal of this talk is to make your cluster management less complex and more cost effective.

Demo: The repository with the Docker Compose setup, configurations, and demo script for frozen indices are available on GitHub.

Tags: security, modsecurity, elastic-stack, talk

How do you protect against injections in your web application and how do you keep track of what attackers are trying to break in?

Security is a hard problem, especially when you are only running but not writing an application. The infamous comic “This is fine” is often the best description we have for this scenario. But it doesn’t have to be. This talks shows how to protect against injections and also how to monitor them.

This talk combines two of the OWASP top ten security risks:

• Injections (A1:2017): We are using a simple application exploitable by injection and will then secure it with the Web Application Firewall (WAF) ModSecurity.
• Insufficient Logging & Monitoring (A10:2017): We are logging and monitoring both the secured and the unsecured application with the open source Elastic Stack.

To make it more interactive, the audience has to do the injections, which we are then live monitoring and mitigating with ModSecurity.

Demo: Try out the example code of the talk or just look at the configuration of ModSecurity and logging.

PS: This is highly insecure and bad code — don’t use it for anything except demoing the security problem.

Tags: elasticsearch, query, talk

Elasticsearch and Kibana added a brand new query language: ES|QL — coming with a new endpoint (_query) and a simplified syntax. It lets you refine your results one step at a time and adds new features like JOINs (yes, JOINs!), data enrichment, and processing right in your query. And you can use it across the Elastic Stack — from the Elasticsearch API to Discover and Alerting in Kibana. But the biggest change is behind the scenes: Using a new compute engine that was built with performance in mind. Join us for an overview and a look at syntax and internals.

Kibana Console sample queries you can run on https://esql.demo.elastic.co.

/**
 * See the documentation for a complete list of
 * features; we'll focus on the concepts and some
 * interesting examples.
 * Some queries will (intentionally) result in errors.
 */


// Let's get started with the equivalent of GET /
// Cover output formats, case sensitivity, KEEP, LIMIT
POST _query
{
  "query": """
    SHOW INFO //Comment
  """
}
POST _query?format=json //But no comment here
{
  "query": """
    SHOW INFO
  """
}
POST _query?format=csv
{
  "query": """
    SHOW INFO
  """
}
POST _query?format=txt
{
  "query": """
    SHOW INFO
  """
}
POST _query?format=txt
{
  "query": """
    show info //Are keywords case sensitive?
  """
}
POST _query?format=txt
{
  "query": """
    SHOW INFO
    | LIMIT 1 //Removing the warning
  """
}
POST _query?format=txt
{
  "query": """
    SHOW INFO
    | LIMIT 1
    | KEEP version //Reducing the output
  """
}
POST _query?format=txt
{
  "query": """
    SHOW INFO
    | LIMIT 1
    | DROP hash //Reducing the output
  """
}
POST _query?format=txt
{
  "query": """
    /* Combining it all with a
     * multi-line comment */
    SHOW INFO
    | LIMIT 1
    | KEEP version
  """
}


// Where should you put the pipe?
POST _query?format=txt
{
  "query": """
    SHOW INFO | LIMIT 1 | KEEP version
  """
}
POST _query?format=txt
{
  "query": """
    SHOW INFO |
    LIMIT 1 |
    KEEP version
  """
}
POST _query?format=txt
{
  "query": """
    SHOW INFO
    | LIMIT 1
    | KEEP version
  """
}


// Use some actual data
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | LIMIT 5
  """
}
GET _query?format=txt
{
  "query": """ //Can you use GET as well?
    FROM sample_data
    | LIMIT 5
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data METADATA _index, _id, _version
    | LIMIT 5
  """
}
POST _query
{
  "query": """
    FROM sample_data
    | LIMIT 5
  """,
  "profile": true //This is undocumented for now but might come in handy; only works with JSON
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | KEEP @timestamp, message
    | LIMIT 10
  """
}


// A quick look at data types, comparisons and (optional) type-casting
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | WHERE client_ip = "172.21.2.162"
    | LIMIT 10
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | WHERE client_ip == "172.21.2.162"
    | LIMIT 10
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | WHERE client_ip == TO_IP("172.21.2.162")
    | LIMIT 10
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | WHERE client_ip IS NOT NULL
    | LIMIT 10
  """
}


// And finally aggregations (with a query DSL example for comparison)
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS COUNT_DISTINCT(client_ip), AVG(event_duration)
    | LIMIT 1
  """
}
GET sample_data/_search
{
  "size": 0,
  "aggs": {
    "distinct_client-ip": {
      "cardinality": {
        "field": "client_ip"
      }
    },
    "average_event-duration": {
      "avg": {
        "field": "event_duration"
      }
    }
  }
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS AVG(event_duration), COUNT(*) BY client_ip
    | LIMIT 10
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS AVG(event_duration), COUNT(*) BY client_ip
    | SORT COUNT(*)
    | LIMIT 10
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS AVG(event_duration), mycount = COUNT(*) BY client_ip
    | SORT mycount
    | LIMIT 10
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS AVG(event_duration), COUNT(*) BY client_ip
    | SORT `count(*)` //Case sensitivity revisited — will this work?
    | LIMIT 10
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS AVG(event_duration), COUNT(*) BY client_ip
    | SORT `COUNT(*)` DESC
    | LIMIT 10
  """
}


// Precedural queries as a core concept
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS AVG(event_duration), COUNT(*) BY client_ip
    | LIMIT 5
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | LIMIT 5
    | STATS AVG(event_duration), COUNT(*) BY client_ip
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | LIMIT 5
    | STATS AVG(event_duration), COUNT(*) BY client_ip
    | SORT `COUNT(*)` DESC
    | LIMIT 1
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | STATS average_duration = AVG(event_duration), count = COUNT(*) BY client_ip
    | STATS MAX(average_duration) BY count //A chained aggregation
    | LIMIT 5
  """
}


// More language features, using ROW for sample data
POST _query?format=txt
{
  "query": """
    ROW a = "philipp@elastic.co"
    | WHERE a RLIKE ".*@elastic.co"
    | KEEP a
  """
}
POST _query?format=txt
{
  "query": """
    ROW a = "2023-01-23T12:15:00.000Z - some text - 127.0.0.1"
    | DISSECT a "%{date} - %{msg} - %{ip}"
    | KEEP date, msg, ip
    | LIMIT 1
  """
}
POST _query?format=txt
{
  "query": """
    ROW a = "2023-01-23T12:15:00.000Z 127.0.0.1 some.email@foo.com 42"
    | GROK a "%{TIMESTAMP_ISO8601:date} %{IP:ip} %{EMAILADDRESS:email} %{NUMBER:num:int}"
    // Note the impact of switching KEEP and EVAL around
    | KEEP date, ip, email, num
    | EVAL date = DATE_EXTRACT("year", TO_DATETIME(date))
    | LIMIT 1
  """
}
POST _query?format=txt
{
  "query": """
    ROW first_name = "Philipp", last_name = "Krenn", height = 1.81
    | EVAL height_feet = height * 3.281, height_cm = height * 100
  """
}
POST _query?format=txt
{
  "query": """
    ROW height = 1.81, weight = 75
    | EVAL BMI = ROUND(weight / POW(height, 2), 1)
    | EVAL healthy = CASE(BMI < 18.5, false, BMI >= 25, false, true)
  """
}


// Multivalued (arrays) fields can be read but functions on them
// return null (unless otherwise noted in the docs)
POST _query?format=txt
{
  "query": """
    ROW a = 5, b = [ 1, 2 ]
    | EVAL a - 3, b + 2, a + b
    | LIMIT 1
  """
}


// Enrich your data with lookups and JOINs
GET _enrich/policy/clientip_policy
POST _query?format=txt
{
  "query": """
    FROM clientips
    | LIMIT 5
  """
}
POST _query?format=txt
{
  "query": """
    FROM sample_data
    | KEEP @timestamp, client_ip, event_duration
    | EVAL client_ip = TO_STRING(client_ip)
    | ENRICH clientip_policy ON client_ip WITH env
  """
}

POST _query?format=txt
{
  "query": """
    FROM sample_data
    | KEEP @timestamp, client_ip, event_duration
    | EVAL client_ip = TO_STRING(client_ip)
    | ENRICH clientip_policy ON client_ip WITH env
    | STATS median_duration = MEDIAN(event_duration) BY env
  """
}

POST _query?format=txt
{
  "query": """
    FROM kibana_sample_data_logs | WHERE response.keyword != "200"
    | LOOKUP JOIN envs_lkp ON clientip
    | STATS COUNT(*) by response, environment
  """
}

Tags: chatgpt, llm, talk

Of course not. But besides the provocative title, ChatGPT and other LLMs are changing — hopefully improving — our industry. Generative AI is massively hyped right now but the concept of copilots makes a lot of sense. We are the pilot but can rely on tooling to make our jobs faster and more efficient; like explaining a Kubernetes error message and listing the most common causes and potential solutions. So what is the state of the art of current copilots and what is not yet or soon° to be expected?

° to the best of our current knowledge

Tags: search, relevance, talk

Reciprocal Rank Fusion (RRF) combines multiple result sets with different relevance indicators — like lexical and dense or sparse vector search — into a single result set. One of the great attributes of RRF is that it requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results. If you have been using boosting to combine different types of searches, this is the talk for you. We’ll dive into the algorithm, how to use it, and how it might surprise you when coming from BM25 rankings.

Tags: elasticsearch, talk

A recent improvement in Elasticsearch is the synthetic source mapping that can cut your storage needs in half. Let’s dive into:

• What is the _source field, why do you need it, and what options did you have until now?
• How does synthetic source work under the hood and what are its limitations and tradeoffs?
• How much disk space can you save with the various mapping options?

On top of that, there is the new Time Series Data Stream (or DataStore), TSDS in short, index type that pushes the optimizations even further by making the most out of co-locating related data.

Tags: elasticsearch, ingest, talk

Make more out of your data by enriching it with metadata. But there are a couple of tradeoffs:

• When: Index- vs runtime — while Elasticsearch has historically always favored index-time, you can do both now. Which has an impact on performance and sometimes also correctness.
• Where: Edge vs central vs in-cluster — having an impact both on performance and operational aspects.
• How: Logstash vs Beats vs Agent (with Fleet) vs OpenTelemetry Collector vs Elasticsearch ingest pipeline vs Elasticsearch runtime fields — you are spoilt for choice.

AI / ML, “de-richment” for personal data, and more complex use-cases all also have their space in this talk.

Tags: observability, analytics, talk

Elastic — the company behind Elasticsearch and Kibana — is running an annual competition to reward contributions like pull requests, blog posts, talks, etc. Once we started giving away MacBooks, we got a massive influx of fraud. This talk tells the tongue-in-cheek story of how people cheated and also how we caught them:

• Observability: Find the bots and trace everyone’s actions to figure out what is a coincidence and what is not.
• Analytics: See how people are trying to exploit the system through fake accounts, shady content, or bending the rules.

While we initially hadn’t planned for this scenario, having the power of search available across observability and analytics data let us do many interesting correlations to get a complete picture of the monster we had created.

Tags: elasticsearch, talk

Search is not just traditional TF/IDF anymore but the current trend of machine learning and models has opened another dimension for search. This talk gives an overview of:

• “Classic” search and its limitation
• What is a model and how can you use it
• How to use vector search or hybrid search in Elasticsearch
• Where OpenAI’s ChatGPT or similar LLMs come into play with Elastic

Tags: datastore, serverless, talk

The architecture of datastores has recently added a new trick: relying on blob stores. It offers new possibilities in scale, cost, and operational simplicity by building on the current storage standard of our industry — AWS S3; and everyone has a compatible implementation. But of course, it comes with tradeoffs. This talk gives an overview of:

• How do blob stores fit into the trend of stateless, serverless, and cloud-native for datastores?
• What options for S3 compatible blob stores do you have?
• Who is using them and what are their tradeoffs or sweet spots?

Tags: scale, talk

Do you still remember the infamous “web-scale” video about MongoDB? It’s been a while, so what is web-scale (not), how can it apply today, and is it still useful? Or will it always be stuck on its ironic use?

Tags: security, opa, talk

One of the emerging standards for cloud (native) security is OPA, the Open Policy Agent; an open source standard under the Cloud Native Computing Foundation. This talk gives an overview of what OPA can do for you and how you can write declarative policies to check your APIs, Kubernetes, or applications. It’s structured into three segments:

1. Why do you want to add a continuous runtime checker to your APIs or applications and what gaps is it covering?
2. How do you write declarative policies with OPA?
3. What does it look like in hands-on examples against APIs, Kubernetes, and applications?

TIL, there are three acronymous for security issues:

FYI: Low WTF: Medium OMG: High

#cfgmgmtcamp

  Willian Braga ❁, @neverping — 2023-02-06

Tags: performance, talk

After getting a bit of a bad reputation (“benchmarketing”) it looks as if vendor benchmarks are very much in fashion again. Let’s take a quick look at common mistakes and how to do them … poorly.

Tags: elastic, terraform, talk

Elastic, the company behind Elasticsearch, has grown in popularity over the years and so have the configuration options. As of now, we are mostly focusing on Terraform. This talk dives into the “why” and how we are splitting up our providers to give our large user-base the best possible experience.

Tags: elasticsearch, foundation, talk

Join for a deep-dive into the engine of Elasticsearch:

• Cluster, cluster state, and node types
• Shards, primaries, and replicas
• Lucene all the way down to segments
• Node balancing and lifecycle management
• Common settings like refresh_interval or heap size
• Benchmarking and performance

Tags: opentelemetry, spring-boot, talk

OpenTelemetry, or OTel for short, is taking over the observability world. And with Spring Boot’s widespread success, they work together very well to uncover all kinds of performance problems or errors. Let’s look at that hands-on.

This talk dives into the background of OTel including the technologies its building on (or replacing). It also takes a look at what auto-instrumentation can and cannot achieve. As well as the latest developments beyond tracing with metrics and the first steps towards logging.

Tags: health, uptime, availability, talk

Sometimes it feels a bit like Harry Potter: “Why is it always the three of you” and then you end up hunting a Horcrux. Let’s take a look at them and how we can counter them. This talk dives into how well-structured health checks can detect our common foes early on. Looking at the failure patterns you can detect some of the signs straight away.

Tags: logs, tradeoffs, talk

Logs are everywhere. But they have gone through an interesting development over the years:

• grep: This works well as long as you have a single instance to search on. Once you need to SSH into many machines and try to piece together the results of multiple grep commands, things tend not to work that well anymore.
• Splunk: Centralizing those logs and letting users search through them with a piped language in Splunk is the logical step to fix that issue. However, the more data you centralize, the slower this will get.
• ELK: The solution to that idleness is using full-text search. Elasticsearch, in combination with Logstash and Kibana (plus Beats), gave logs a major performance boost. But at what cost?
• Loki: Reducing the scope and going back to a smart data structure combined with grep gives Loki the possibility to reduce costs while still providing good performance.
• Closing the gap: So what are the tradeoffs between the different systems, and are they potentially closing some gaps between performance, features, and cost?

Tags: kubernetes, operator, talk

The goal of this live debugging session is to better understand how to work with a failing Kubernetes Operator and get used to some helpful Kubernetes commands.

Each of the three examples follows the same structure:

• Apply an invalid YAML manifest.
• Figure out what is wrong and how to fix it.
• Hints that may help solve the problem.
• A detailed walkthrough to understand and solve the problem.

Demo: The broken YAML manifests — don’t use them unless you want to debug 😅

Tags: nosql, security, talk

New systems are always attractive targets since their security model couldn’t mature yet. NoSQL databases are no exception and had some bad press about their security, but how does their protection look like? We will take a look at three widely used systems and their unique approaches:

• MongoDB: Widely criticized for publicly accessible databases and a frequent victim of ransomware. In reality, it provides an elaborate authentication and authorization system.
• Redis: Security through obscurity or how you can rename commands. And it features a unique tradeoff for binding to publicly accessible interfaces plus brand-new security features.
• Elasticsearch: Groovy scripting has been a constant headache, but the new, custom-built scripting language Painless tries to take the pain away literally. Plus, free TLS and role-based authentication to combat ransomware attacks.

Tags: monitor, java, talk

“With microservices every outage is like a murder mystery” is a common complaint. But it doesn’t have to be! This talk gives an overview on how to monitor (distributed) applications. We dive into:

• Uptime monitoring: Ping services and actively monitor their health and response time.
• Application logs: Collect structured logs in a central location.
• Network monitoring: Find out what is happening on the network.
• System metrics: Keep track of network traffic and system load.
• Security events: Collect who is doing what when to find malicious actors.
• Application metrics: Get the information from the application’s metrics and health endpoints via REST or JMX to see how your heap is filling up.
• Request tracing: Trace requests through a distributed system and show how long each call takes and where errors are happening.

And we do all of that live, since it is so easy and much more interactive that way using Elasticsearch Service on Elastic Cloud.

Tags: spring-boot, log, talk

Let’s look at the good parts of logging:

• How can you bring logging to production quickly?
• How does this work with Spring Boot in as a practical example?
• Do you want to add tracing and make the most of the combination?
• How do you manage the data that you are collecting?

Tags: test, elasticsearch, talk

How are you testing with your database?

• Mocking is not an option since you want to test the actual system.
• In-memory databases, like H2 or HSQLDB, have subtle differences and not all datastores have in-memory cousins.
• Managing and running tests in parallel against the actual datastore is a pain.

So what is the solution? There are some very neat solutions based on containers, namely the Docker-Maven-Plugin and Testcontainers. From your tests you can start a lightweight, throwaway instance of your datastore and this talk will walk you through how to do that.

Demo: The repository with the example tests are available on GitHub.

Tags: seccomp, security, talk

Why should you allow all possible system calls from your application when you know that you only need some? If you have ever wondered the same then this is the right talk for you. We are covering:

• What is seccomp in a nutshell and where could you use it.
• Practical example with Elasticsearch and Beats.
• How to collect seccomp violations with Auditd.

Because your security approach can always use an additional layer of protection.

Video of the talk at BalCCon.

PS: This talk is based on Alexander Reelsen’s blog post Seccomp in the Elastic Stack.

Tags: elasticsearch, coordination, cluster, talk

Distributed systems are hard and unfortunately often less perfect than we would hope. This talk shows a concrete example: Elasticsearch’s cluster coordination, called Zen Discovery, got a rewrite in version 7.0. Starting from a formal model, the coordination layer was rebuilt to address multiple issues discovered over the years. We discuss the main improvements of the new implementation: Master elections are much faster, the infamous minimum_master_nodes setting has been removed, growing and shrinking clusters becomes safer and easier, and leaves less room to misconfigure the system. Let us join the new, more Zen way of cluster coordination.

Demo: Take a look at the demo code for the talk.

Tags: devrel, talk, talk

Virtual talks are the new norm. In theory, they are great — speakers and attendees can join from around the world, infrastructure is cheap and scalable, recordings are simple. But what is happening in reality with problems like Zoom fatigue? What is and what isn’t working? At Elastic, we have done more than 500 — by now more than 1,000 — virtual talks and meetups since the pandemic’s start.

This talk gives an overview of the development over time, specifics in regions, differences in tooling,… and adds some of my personal anecdotes.

Tags: security, tradeoff, talk

The NoSQL ecosystem thrived on combining scalability and simplicity. This talk focuses on some assumptions we built Elasticsearch on, which helped the ease of use initially, but turned out to be less than perfect for security in the long run:

• Binding to all interfaces and broadcasting join requests to the whole subnet makes clustering simple.
• Running as root is the straightforward option.
• Guessing the content-type of a request is fine.
• Default passwords and clear-text password files are a reasonable tradeoff.
• Docker and distributed systems play well with your security efforts.
• Generating TLS certificates is easy.
• Everyone will turn on security and defaults are easy.

Hypothesis: You need ease of use to grow initially, but switch to more secure settings for critical workloads over time.

Tags: oss, business, talk

How do you build a lasting and successful company that also stays true to its open source roots? This talk takes a look at why open source is important to business and three essential elements of this path:

• Strategy: How can you monetize your open source product? Is it support, an open core approach, cloud services, or a combination of the three? And which ones are the features you can even commercialize without alienating your community?
• Struggle: “You received a 100 million dollars in venture capital and yet you have so many open issues?!” Once money is involved the dynamics often change. How can you manage expectations and still build on a flourishing open source community?
• Success: How do you balance open source and commercial success? How do you align engineering, product, and sales decisions?

This talk takes the perspective of Elastic, the company behind Elasticsearch, Kibana, Beats, and Logstash, which makes its money with support, the commercial extensions, and cloud offerings. But we are also taking a look at how others are approaching this challenge, what worked, and what failed.

Tags: tradeoff, elasticsearch, cap, talk

The CAP theorem is widely known for distributed systems, but it’s not the only tradeoff you should be aware of. For datastores there is also the FAB theory and just like with the CAP theorem you can only pick two:

• Fast: Results are real-time or near real-time instead of batch oriented.
• Accurate: Answers are exact and don’t have a margin of error.
• Big: You require horizontal scaling and need to distribute your data.

While Fast and Big are relatively easy to understand, Accurate is a bit harder to picture. This talk shows some concrete examples of accuracy tradeoffs Elasticsearch can take for terms aggregations, cardinality aggregations with HyperLogLog++, and the IDF part of full-text search. Or how to trade some speed or the distribution for more accuracy.

Tags: yaml, harmful, talk

We have all become YAML engineers, because everybody loves it, right?! Sometimes it can also burn you pretty badly and this is what this talk is about.

We dive into some common errors people encounter when working with YAML. These seem to be commonly accepted, which might be a form of the Stockholm Syndrome. And we are also taking a quick look at alternative formats and what they do (not) have to offer. Will you still love your YAML files after this talk? Let’s see…

Tags: data, visualization, dashboard, kibana, talk

Kibana can do more than just logs and metrics — it can also be a tool for business analytics and your management. Especially when you want to integrate and correlate as many different data-sources as possible.

Based on a built-in sample dataset we look into:

• See the raw data in Discover.
• Two kinds of data — entity-centric and timeseries.
• Transform timeseries into entity-centric data with the (continuous) ML Transformations.
• Visualizations of timeseries with line charts, tag clouds, or the Visual Builder (including annotations).
• Adding a more user-friendly and discoverable approach to visualizations with Lens.
• Dashboards and Reports.
• The new features in Maps.
• Pixel perfect presentations in Canvas.

Tags: distributed, team, talk

Working from home in distributed teams isn’t exactly new. What is rarer though is how to scale it. At Elastic we are now more than 2,000 and we have been in the work from anywhere mode when we were founded in 2012. This talk shares some things that have and have not worked out while we learned how to scale over the years:

• Responsibilities from each one of us.
• How to work with meetings — 1:1, team, and company-wide.
• How your managers are essential in keeping the team together and everybody’s sanity.
• Why adding more teams and people will create bumps in your growth and how to overcome them.

PS: While this talk focuses on the scaling side, Work from Home: Challenges & Tips dives into the perspective for individuals and Building Distributed Systems in Distributed Teams on how to get started as a company.

Tags: gdpr, datastore, talk

The General Data Protection Regulation (GDPR) is changing how you can handle data in Europe. But what does this actually mean? The first part of this talk gives an overview about the implications of GDPR, which affects every software project with a European relation. That includes users’ right to see, edit, and export their data, the right to be forgotten,… The second part takes a look at what this means for actual software projects with the specific use-case of logging. The main focus here is how to stay GDPR compliant while still being able to use the data for security and operation aspects.

PS: This talk does not replace legal advice or a deeper examination of the topic. It gives you an overview and pointers to relevant techniques, but you need to discuss the implementation for your project with your own legal counsel.

Tags: team, distributed, elastic, talk

Building distributed systems is notoriously hard. Building a distributed team, even more so. At Elastic — the company behind Elasticsearch, Kibana, Beats, and Logstash — everything is distributed; the company and our products. This talk dives into the details of how Elastic is thriving on its distributed model:

• How Elastic started to be distributed by design.
• What our shared values are and what we have learned as we progressed and grew to our current size.
• What the day of an Elastician can look like.
• What we are using in terms of tooling.
• How we hire and onboard new employees.

PS: While this talk focuses on how to build a distributed company, Work from Home: Challenges & Tips dives into the perspective for individuals and Scaling Distributed Teams looks more into the challenges around company growth.

Tags: elasticsearch, kibana, elastic-stack, upgrade, talk

While you can find the full list of changes and especially the breaking changes in the release notes, we focus on some interesting ones and dive a little deeper into them.

If the slides alone are not providing enough context you can follow along with a more detailed description below.

Demo Environment #

Before we get started, we need to cover the demo environment — a simple three node cluster with Docker Compose.

Live Upgrade #

Starting with a 6.7 cluster we are upgrading to 7.0 and then even prepare for the 8.0 migration 😉.

First, we check that everything is up, we have the right version, and a green cluster:

GET /

GET _cat/nodes?h=id,version,master,name&v

GET _cluster/health

POST test/_doc
{
  "name": "Elasticsearch 6.7"
}

Next, we check the upgrade assistant in Kibana (/app/kibana#/management/elasticsearch/upgrade_assistant) — since this is a fresh cluster nothing should need to change. We disable replica shard allocation because the upgraded nodes will come back up quickly and this would just add unnecessary work. But we allow new_primaries, because Kibana 7 needs to create a new index and will not start up otherwise. Finally, we do a sync flush to speed up recovery times of the nodes:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "new_primaries"
  }
}

POST _flush/synced

We change the Elastic Stack version in the .env file and upgrade the first node with docker-compose up -d --no-deps elasticsearch3. We can check the cluster during the upgrade and still keep querying documents:

GET _cat/nodes?h=id,version,master,name&v

GET test/_search

We follow the same procedure for elasticsearch2, elasticsearch1, and kibana. Once done we re-enable shard allocation, add a document, and see that everything is still working as expected:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": null
  }
}

POST test/_doc
{
  "name": "Elasticsearch 7.0"
}

GET test/_search

Finally, we check the upgrade assistant again, reindex the test index since data can only be read from one version back, and see that everything is ready for our (very) future 8.0 upgrade:

GET _cat/indices
GET _cat/aliases?v

Zen2 #

Cluster coordination rethought and rebuilt. Read the blog post about Zen2 if you want a more detailed description than the slides.

Types or Rather No Types #

Both the past and the next steps in Elasticsearch’s journey of getting rid of types. We create a simple mapping and notice the absence of a type. But will a query like PUT no_type/2 work now?

# Types
PUT no_type
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

PUT no_type/_doc/1
{
  "@timestamp": "2019-05-08"
}

GET no_type/_search

PUT no_type/2
{
  "@timestamp": "2019-05-09"
}

No, this will not work and might be a surprise, but the documentation provides an answer:

In 7.0, _doc represents the endpoint name instead of the document type. The _doc component is a permanent part of the path for the document index, get, and delete APIs going forward, and will not be removed in 8.0.

Additional details are available on the Elastic blog about going typeless.

Optimizations #

For more details than in the slides see the faster retrieval of top hits blog post, but the following demo shows how they work in practice. First, we add the flight demo data set in Kibana, so we have something to work with and then run through the following queries to see the differences in results:

GET kibana_sample_data_flights/_search

GET kibana_sample_data_flights/_search
{
  "track_total_hits": 10
}

GET kibana_sample_data_flights/_search
{
  "track_total_hits": true
}

index.search.idle.after should be relatively self explaining. The main caveat is that it will only work if you are using the default index.search.idle.after (one second). That is because skipping the refresh is cancelled on the next search and that search operation will block until the next refresh happens automatically. Blocking longer than one second is probably not what anybody wants, so that combination of settings is required.

New Defaults #

A single primary shard is the new default. The motivation is that over-sharding is a very common problem and a single shard will normally make most sense for beginners. Experts will know how shards work and will reconfigure them as needed.

Adaptive Replica Selection is now on by default.

Less Heap #

Frozen indices were first released in 6.6, but are still worth of being in the highlights. We run through the following queries to see frozen indices in action and while the force merge is not required it is recommended:

POST frozen/_doc
{
  "name": "Boaz"
}
POST frozen/_doc
{
  "name": "Pablo"
}

GET frozen/_search

POST frozen/_forcemerge?max_num_segments=1

POST frozen/_freeze

GET frozen/_search

Suddenly all our data seems to be gone, but this is for our protection. You need to be explicit about expensive queries:

GET frozen/_search?ignore_throttled=false

GET _cat/indices/frozen?v&h=health,status,index,pri,rep,docs.count,store.size

GET _cat/thread_pool/search_throttled?v&h=node_name,name,active,rejected,queue,completed&s=node_name

GET frozen/_search?ignore_throttled=false&pre_filter_shard_size=1

POST frozen/_doc
{
  "name": "Daliya"
}

Adding new records will fail as long as the index is frozen. This is how to check and change it:

GET frozen/_settings?flat_settings=true

POST frozen/_unfreeze

POST frozen/_doc
{
  "name": "Daliya"
}

GET frozen/_search

Coming back to the flight demo set we can check this on a larger scale. First, we look at the dashboard for that and see that we need to enable querying of frozen indices there as well under Advanced Settings (/app/kibana#/management/kibana/settings/?_g=()) called Search in frozen indices. Then we look at the thread staticstics and how the dedicated thread pool is at work:

POST kibana_sample_data_flights/_freeze

GET _cat/thread_pool/search_throttled?v&h=node_name,name,active,rejected,queue,completed&s=node_name

Additional details in the related blog post.

In addition, real-memory circuit breakers have landed and you can find more details than in the slides in the relevant blog post.

Kibana #

The new look of Kibana is driven by the Elastic UI Framework while the de-Angularization is still in progress. While this is great for reuse, note the warning in the repository:

🚨 While open source, the intended consumers of this repository are Elastic products. Read the FAQ for details.

Tags: elasticsearch, scale, metrics, talk

“Only accept features that scale” is one of Elasticsearch’s engineering principles. So how do we scale metrics stored in Elasticsearch? And is that even possible on a full-text search engine?

This talk explores:

• How are metrics stored in Elasticsearch? And how does this translate to disk use as well as query performance?
• What does an efficient, multi-tier architecture look like that balances speed for today’s data against density for older one?
• How can you compress metrics and what does the mathematical model look like for that?

Demo: We are trying this hands-on during the talk since this has become much simpler recently. Follow along in the GitHub project.

Tags: docker, image, experience, talk

If you want to be taken seriously, you need to provide containers to your users. It’s easy — everybody is uploading containers to Docker Hub, right? Unfortunately, reality is never as easy as it sounds at first. This talk gives an overview of Elastic’s ongoing journey to providing official Docker images:

• Docker Hub: What “official” really means and why we are using our own registry.
• Base image: Just use Alpine — it is small and the perfect fit for containers. We tried that and reconsidered…
• Release policy: What do you actually get in a specific tag and how are we releasing our images?
• Support: Combine two complex systems like Elasticsearch and Docker — and you will get a lot of questions.
• Orchestration: Our current approach for orchestration and how we are treating feature requests.