Elasticsearch, as the name suggests, is primarily used for its fast search capabilities. But have you ever wondered how scoring happens in the background and why certain documents are given more scoring compared to the rest? In this two-part series, we will be focusing on how to write simple to complex search queries, followed by which we will be explaining the calculations that happen under the hood using the explain API.
Search Results Ranking Basics:
Elasticsearch used to have TF-IDF as its default similarity algorithm and has shifted to BM25 (Best Matching) ever since the introduction of Lucene 6.
A simple explanation of how search results are assigned a score is the function of the
Below criteria:
- Term Frequency (TF): No of times the search term appears in the document.
- Length of the document containing the search term vs the average length of the document in the search result.
- Inverse Document Frequency (IDF): Number of documents that contain a value for the search field vs Number of documents that contain the search term in the field that we are searching. In simpler terms, how rare is this search term in a document. Higher the score if rare search terms are present.
- Elasticsearch will sum up the score computed by all search terms.
Learning with a scenario:
Let us assume we have a books catalogue (books index) where category, a short description and a long description are primarily used for finding relevant books. We start by writing a simple search query to find the relevant books from the catalogue.
Note: Step by step guide on how to reproduce the queries have been provided at the end of the article.
Let’s begin our search journey:
Let us start our search journey by trying to find the word “Open Search” in the books index.
Scenario 1: Simple search:
Search for the word “Open Search” in the field named title.
The search would give 100+ hits for the keyword “Open Search”. It might seem relevant at an initial glance but none of them is relevant since they don’t talk about Lucene (which is a famous Open Source based search engine).
In reality, the word “Open source” can also be referenced in the fields short description and long description and this was not even considered in the search query. In the upcoming section, we will be taking a look at how different fields influence the search results.
Scenario 2: Adding more fields to the query:
Extending Scenario 1, now let us say we want to search the same word phrase “Open Search” in multiple fields: categories, longDescription, shortDescription.
You may notice a considerable change in results with a total of 36 matched documents having a max_score of 8.118633. To our surprise, the document that was on top in Scenario 1 would be placed somewhere in the middle.
Note: The type “phrase” takes the max from every field and returns the document with the highest score.
Scenario 3: Boosting selected fields
In some use cases, selected fields should contribute more to the result score. Use field boosts to assign that weight.
Note: boosting uses the caret (^) followed by the boost value.
Scenario 4: Extending it further:
Now let us say we also want to return partial matches for the word “Open Search” in addition to the exact match.
If you notice, the max_score would be 530.05334 for the document with ISBN 1933988177 (Lucene in Action, Second Edition) for the topmost document.
The results would vary depending on the type selected and it is important to understand every type before starting to write complex search queries. For example, if we had chosen the “best_fields” instead of “most_fields” the document with ISBN 1933988673 would have returned. This is because “best_fields” makes use of the max of the search results whereas “most_fields” makes use of the max of the search results. (Other Supported types)
Summary:
To summarize, the more we go in-depth about the tuning of ES queries and improving of fields, the more we will notice the importance of each parameter. It is important to consider the following while developing your search queries:
- Keywords field types are not intended for partial matches, they can hurt the search results. It is recommended to use keyword field types only when a single word is to be searched so that an exact word match can be performed.
- Keywords fields are supposed to be unique word limits and overutilization of keyword fields can lead to improper search results.
- It is important to only improve fields when there is an absolute necessity. Overboosting of fields can lead to undesired results.
- The use of field types in match queries should be based on the need and requirement.
- Sometimes it is better to filter first and then perform a search operation.
- Function score can be used for decaying the score and relevance of older documents
The follow-up article explains how to use the Explain API to inspect scoring behaviour.
Exercise:
Step 1: Get the sample books dataset from the reference link.
Step 2: Create an index called books with the following mappings as given below. For article sake, we have created mappings only for fields against which search will be performed.
Note: It is important to create explicit mapping for better performance and less overhead.
Step 3: Upload the JSON to Elasticsearch with the Bulk API, then run the query.
To continue learning about Elastic Search, read part 2