Elasticsearch Upgrade

Owned by Dustin Lo
Last updated: Jun 20, 2019
55 min read

Version: 1.7 → 7.1

Big Changes

  • Only 1 type allowed in each index: _doc

  • Need to manually enable all field text searches

  • Removal of filtered since ES 5.0

  • Changes in the geo coordinates query

    • Note: {"type": "geo_shape","tree": "quadtree","tree_levels": 26} makes uploading documents slow, specifically "tree_levels”: 26

    • { "query": { "bool": { "filter": { "geo_shape": { "location": { "shape": { "type": "polygon", "coordinates": [[<coordinates>]] }, "relation": "within" } } } } } }

  • Removal of _all: { "enabled": true } type in indices so we cannot search for all fields

 

Running Elasticsearch 7 on EC2 instance

In order to expose port 0.0.0.0 properly, we need to edit the config/elasticsearch.yml file

network.host: 0.0.0.0 cluster.name: grq_cluster node.name: ESNODE_CYR node.master: true node.data: true transport.host: localhost transport.tcp.port: 9300 http.port: 9200 discovery.zen.minimum_master_nodes: 2 # allows UI to talk to elasticsearch (in production we would put the actual hostname of the uI) http.cors.enabled : true http.cors.allow-origin: "*"

 

Running Kibana on EC2 instance

Install Kibana in command line

curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.1.1-darwin-x86_64.tar.gz tar -xzf kibana-7.1.1-darwin-x86_64.tar.gz cd kibana-7.1.1-darwin-x86_64/

Edit the config/kibana.yml file to expose host 0.0.0.0

Index Mapping

Index: grq_v1.1_s1-iw_slc

What I did to update the mapping

  • create mapping called grq_v1.1_s1-iw_slc

  • copied example data into new ES index, using built in dynamic mapping to build initial mapping

  • mapping changes:

    • metadata.context to {"type": "object", "enabled": false}

    • properties.location to {"type": "geo_shape","tree": "quadtree"}

    • use type keyword to be able to use msearch:

      • "reason": "Fielddata is disabled on text fields by default. ... Alternatively use a keyword field instead."

 

  • PUT grq_v1.1_s1-iw_slc/_mapping

 

 

Index: grq_v3.0_area_of_interest

  • PUT grq_v3.0_area_of_interest/_mapping

 

Index: grq_v2.0.2_s1-gunw

PUT grq_v2.0.2_s1-gunw/_mapping