How Does Elasticsearch Store a float
Value into an integer
Field?
Today there was a Discuss post on “Elasticsearch data type” that demonstrates one of the more confusing features in Elasticsearch. But if you are familiar with Elasticsearch it is an excellent puzzle — so follow along and test your knowledge.
First, add a document:
PUT users/_doc/1
{
"user_id": 1
}
This index uses a dynamic mapping, which defaults to what data type for the user_id
field?
Default Numeric Type #
GET users/_mapping
shows the answer:
{
"users" : {
"mappings" : {
"properties" : {
"user_id" : {
"type" : "long"
}
}
}
}
}
So your user_id
field is a long
. Next, you try to add four more documents:
PUT users/_doc/2
{
"user_id" : 2
}
PUT users/_doc/3
{
"user_id" : "3"
}
PUT users/_doc/4
{
"user_id" : 4.5
}
PUT users/_doc/5
{
"user_id" : "5.1"
}
The Document with ID two is the same as our first one, so that will work. But what happens if you try to store a string
, a float
, or even a stringified float
value into a long
field?
Handling Dirty Data #
It still works. But why?
By default, Elasticsearch will coerce
data to clean it up. Quoting from its documentation:
Coercion attempts to clean up dirty values to fit the datatype of a field. For instance:
- Strings will be coerced to numbers.
- Floating points will be truncated for integer values.
Especially for quoted numbers, this makes sense, since some systems err on the side of quoting too much rather than too little. Perl used to be one of the well-known offenders there, and coercing would helpfully clean this up.
Sounds reasonable, but you want to verify this by retrieving the documents with GET users/_search
and expect the user_id
values 1 2 3 4 5, right? But you actually get the result — focus on the array in hits.hits
:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "users",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"user_id" : 1
}
},
{
"_index" : "users",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"user_id" : 2
}
},
{
"_index" : "users",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"user_id" : "3"
}
},
{
"_index" : "users",
"_type" : "_doc",
"_id" : "4",
"_score" : 1.0,
"_source" : {
"user_id" : 4.5
}
},
{
"_index" : "users",
"_type" : "_doc",
"_id" : "5",
"_score" : 1.0,
"_source" : {
"user_id" : "5.1"
}
}
]
}
}
Is this a bug? How could you store 4.5 in a long
? If you recheck the mapping with GET users/_mapping
, it’s still returning "type": "long"
.
_source
Is Only an Illusion #
The final piece in this puzzle is that Elasticsearch never changes the _source
. But the stored field user_id
is a long
as you would expect. You can verify this by running an aggregation on the field:
GET users/_search
{
"size": 0,
"aggs": {
"my_sum": {
"sum": {
"field": "user_id"
}
}
}
}
Which gives the correct result for 1 + 2 + 3 + 4 + 5 = 15:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"my_sum" : {
"value" : 15.0
}
}
}
Which, by the way, now defaults to a floating representation for value
which you could change with the extra parameter "format": "0"
. That would add a "value_as_string" : "10"
to the result. But let’s leave it at that.
Conclusion #
I hope you are less confused than before or at least enjoyed the puzzle. As a parting note, be aware that coerce
might be removed in the future since it is a trappy feature — especially around truncating floating-point numbers 😄.