编程知识 cdmana.com

Step by step disassembly to solve the problem of elasticsearch retrieval template

1、 Online practical questions

Elasticsearch When doing template query , In the use of terms When batch query, it failed to put the array into the template , It is similar to how the template passes in the array ?

Source of problem : screwing Elasticsearch Knowledge of the planet

#  Define index 
PUT uint-2020-08-17
{
    "mappings": {
      "properties": {
        "clock": {
          "type": "date",
          "format": "epoch_second"
        },
        "itemid": {
          "type": "long"
        },
        "ns": {
          "type": "long"
        },
        "ttl": {
          "type": "long"
        },
        "value": {
          "type": "long"
        }
      }
    }
}

#  Add content 
PUT uint-2020-08-17/_bulk
{ "index" : {  "_id" : "1" } }
{"itemid":1,"ns":643214179,"clock":1597752311,"value":"1123","ttl":604800}
{ "index" : {  "_id" : "2" } }
{"itemid":2,"ns":643214179,"clock":1597752311,"value":"123555","ttl":604800}
{ "index" : {  "_id" : "3" } }
{"itemid":3,"ns":643214179,"clock":1597752311,"value":"1","ttl":604800}
{ "index" : {  "_id" : "4" } }
{"itemid":4,"ns":643214179,"clock":1597752311,"value":"134","ttl":604800}
{ "index" : {  "_id" : "5" } }
{"itemid":2,"ns":643214179,"clock":1597752311,"value":"123556","ttl":604800}

Query statement :

PUT _scripts/item_agg
{
  "script": {
    "lang": "mustache",
    "source": {
      "_source": [
        "value"
      ],
      "size": 0,
      "query": {
        "bool": {
          "filter": [
            {
              "terms": "{
    {#toJson}}statuses{
    {/toJson}}"
            },
            {
              "range": {
                "clock": {
                  "gte": "{
    {startTime}}",
                  "lte": "{
    {endTime}}"
                }
              }
            }
          ]
        }
      },
      "aggs": {
        "group_terms": {
          "terms": {
            "field": "itemid"
          },
          "aggs": {
            "avg_value": {
              "avg": {
                "field": "value"
              }
            },
            "max_value": {
              "max": {
                "field": "value"
              }
            }
          }
        }
      }
    }
  }
}

Query template parameters :

POST uint-*/_search/template
{
  "id": "item_agg",
  "params": {
    "itemid":{
      "statuses":[1,2]
    },
    "startTime":1597752309,
    "endTime":1597752333
    
  }
}

The above content looks very long , According to the notes, it is disassembled as :

  • Define index 、

  • insert data 、

  • Create a template 、

  • Construct parameter retrieval

Four sub parts, you won't panic .

2、 Interpretation of knowledge points —— Search template

2.1 What is a search template ?

Many people have heard of using Index template index template, The benefits of index templates :

  • Easy to model across indexes ;

  • Especially suitable for large amount of data 、 Index fields are similar to business systems ;

  • Flexible and convenient .

Search template (search template) People use it less , In a real business scenario : Every business request is constructed DSL, such as : This time title、 Next time check content, Beyond that DSL part Are all the same , But two requests : The back-end code should be modified and adapted accordingly . There is no modification 、 Splicing DSL Using a search scheme ? This leads to the search template .

Search templates are very similar to stored procedures in relational databases . Common queries can be defined as templates , And use Elasticsearch The application can be simply passed through its ID Reference query .

Templates accept parameters specified at run time . Search templates are stored on the server side , It can be modified without changing the client code .

Templates use Mustache The template engine represents . About Mustache You can visit :

http://mustache.github.io/mustache.5.html.

2.2 Search template example

According to the first part of the actual combat data , The following templates are defined .

PUT _scripts/cur_search_template
{
  "script": {
    "lang": "mustache",
    "source": {
      "query": {
      "match": {
        "{
    {cur_field}}": "{
    {cur_value}}"
      }
    },
    "size": "{
    {cur_size}}"
    }
  }
}

POST uint-*/_search/template
{
  "id": "cur_search_template",
  "params": {
    "cur_field":"itemid",
    "cur_value":1,
    "cur_size":50
    
  }
}

The template : Support user-defined dynamic setting of search fields and search parameter fields .

In actual combat, we can pass the following _scripts The way , Define the retrieval template to the server side .

If you want to retrieve other fields : The client or the requester can pass different parameters .

The real thing happened : Separation of retrieval and request parameters .

For more principles and foundations, see the official documentation :

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html

2.3 search template The grammar is very mind blowing

The following is excerpted from :Wood The uncle ——Elastic Certification examination experience .

Write one as required search template

be familiar with search template Of mustache Template language can easily write , But unfortunately , Never used it search template, Although I know something about , But when I wrote it , I don't know where the grammar is wrong ,PUT template Always unsuccessful . Guess which position of the character may not be translated to produce illegal json character , Or which level of nesting is problematic . In a word, failure to debug is just debugging , And wasted a lot of time .

https://elasticsearch.cn/article/6133

What I want to illustrate is that :search template The grammar of is more complicated , If you haven't used it , It's easy to have a big head .

3、 Problem disassembly

3.1 The original DSL Anything wrong? ?

A real fight , An error is as follows :

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "[terms] query malformed, no start_object after query name",
        "line": 1,
        "col": 67
      }
    ],
    "type": "parsing_exception",
    "reason": "[terms] query malformed, no start_object after query name",
    "line": 1,
    "col": 67
  },
  "status": 400
}

3.2 What's wrong ?

Take it apart .

  • script The part includes nothing but : The retrieval and aggregation section .

  • The search part is to define search template At the heart of , The aggregation part needs no attention .

This is the time , You can write a search DSL Check to see if ok, as follows :

POST /_search
{
  "_source": [
    "value"
  ],
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "itemid": [
              1,
              2
            ]
          }
        },
        {
          "range": {
            "clock": {
              "gte": 1597752309,
              "lte": 1597752333
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "group_terms": {
      "terms": {
        "field": "itemid"
      },
      "aggs": {
        "avg_value": {
          "avg": {
            "field": "value"
          }
        },
        "max_value": {
          "max": {
            "field": "value"
          }
        }
      }
    }
  }
}
  • And retrieval and aggregation are right , That's mostly the definition search template Part of it went wrong .

The problem is just a little bit taken apart .

Come up and change this directly DSL I don't know how to start , Let's do it : Minimize it .

Throw away all :_source、size、aggs、range query part , Only keep terms How to write scripts correctly ?

Come on , A real fight :

  • First step : To minimize the terms Search template .

GET _search/template
{
  "source": "{ \"query\": { \"terms\": {
    {#toJson}}statuses{
    {/toJson}} }}",
  "params": {
    "statuses" : {
        "itemid": [ 1, 2 ]
    }
  }
}

Compare the mistakes in the first part with the correct ones now , Two mistakes can be found :

  • error 1:source The contents should be added :"\" .

  • error 2: Query template parameters statuses and itemid The position is wrong .

The official document says : 

The { {#toJson}}parameter{ {/toJson}} function can be used to convert parameters like maps and array to their JSON representation:

statuses It's an auxiliary parameter , Our core parameter is itemid.

  • The second step : Turn the first step into script form .

POST _scripts/test_script_01
{
  "script": {
    "lang": "mustache",
    "source": "{ \"query\": { \"terms\": {
    {#toJson}}statuses{
    {/toJson}} }}"
  }
}

POST uint-*/_search/template 
{
  "id": "test_script_01",
  "params": {
    "statuses": {
      "itemid": [
        1,
        2
      ]
    },
    "startTime": 1597752309,
    "endTime": 1597752333
  }
}
  • The third step : Complete the parameters according to the actual combat requirements .

Pay attention to the completion , I suggest : Copy DSL( Format one line version ) To third-party text tools such as :Nodepad++, Global replacement .

Remember not to knock , It's easy to make a mistake .

Replace to template source part , And then according to the first step 、 The second step is to modify the content .

  • The answer to the actual question

GET _search/template
{
  "source": "{\"_source\":[\"value\"],\"size\":0,\"query\":{\"bool\":{\"filter\":[{\"terms\":{
    {#toJson}}statuses{
    {/toJson}}},{\"range\":{\"clock\":{\"gte\":{
    {startTime}},\"lte\":{
    {endTime}}}}}]}},\"aggs\":{\"group_terms\":{\"terms\":{\"field\":\"itemid\"},\"aggs\":{\"avg_value\":{\"avg\":{\"field\":\"value\"}},\"max_value\":{\"max\":{\"field\":\"value\"}}}}}}",
    "params": {
    "statuses" : {
        "itemid": [ 1, 2 ]
    },
       "startTime":1597752309,
    "endTime":1597752333
  }
}

Copy source Part of it can be converted to script format , The space problem , I won't repeat .

4、 Summary

It seems complicated , After dismantling, it's not complicated .

It seems very difficult , It's easy to disassemble .

The search template works well , Less wrangle at the front and back 、 It's a lot more efficient !

Your little problem , My big problem .

Be with you , screwing Elasticsearch!

Reference resources :

https://elastic-search-in-action.medcl.com/3.site_search/3.3.search_box/search_template/

https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781787128453/7/ch07lvl1sec61/search-templates

recommend :

blockbuster | screwing Elasticsearch Methodological cognitive checklist (2020 National day update )

If you can get a driver's license, you can pass Elastic Certification examination !

more short time more Learn quickly more More dry !

China  40%+ Elastic Certified engineers come from !

And global  800+ Elastic Enthusiasts fight together Elasticsearch!

Add WeChat :elastic6, Ask for value 18 Tickets to the original planet

版权声明
本文为[Mingyi world]所创,转载请带上原文链接,感谢

Scroll to Top