Elasticsearch 用户名检索方案

2年前

安装es

实验目的

主要用于项目的用户名检索

实验对比

实验样本

用户名
༺ཎཽཾ࿄࿆大࿈宝࿄࿆ནཽཾ༻
西瓜买的土豆
Yvonne✨
💗不想💗錯過你💗
算了就算麻辣烫
你好高，我怕够不着
Sophie
🤪
Hermione🐯
panda
aka. 橘子
中华人民共和国国歌

elasticsearch-analysis-ik

创建数据库

curl -X PUT \
  http://localhost:9200/user_ik \
  -H 'Content-Type: application/json' \
  -d '{
  "mappings": {
    "user": {
      "properties": {
        "nickname": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        }
      }
    }
  }
}'

写入数据

curl -XPOST http://localhost:9200/user_ik/user/1 -H 'Content-Type:application/json' -d' {"nickname":"༺ཎཽཾ࿄࿆大࿈宝࿄࿆ནཽཾ༻"}'
curl -XPOST http://localhost:9200/user_ik/user/2 -H 'Content-Type:application/json' -d' {"nickname":"西瓜买的土豆"}'
curl -XPOST http://localhost:9200/user_ik/user/3 -H 'Content-Type:application/json' -d' {"nickname":"Yvonne✨"}'
curl -XPOST http://localhost:9200/user_ik/user/4 -H 'Content-Type:application/json' -d' {"nickname":"💗不想💗錯過你💗"}'
curl -XPOST http://localhost:9200/user_ik/user/5 -H 'Content-Type:application/json' -d' {"nickname":"算了就算麻辣烫"}'
curl -XPOST http://localhost:9200/user_ik/user/6 -H 'Content-Type:application/json' -d' {"nickname":"你好高，我怕够不着"}'
curl -XPOST http://localhost:9200/user_ik/user/7 -H 'Content-Type:application/json' -d' {"nickname":"Sophie"}'
curl -XPOST http://localhost:9200/user_ik/user/8 -H 'Content-Type:application/json' -d' {"nickname":"🤪"}'
curl -XPOST http://localhost:9200/user_ik/user/9 -H 'Content-Type:application/json' -d' {"nickname":"Hermione🐯"}'
curl -XPOST http://localhost:9200/user_ik/user/10 -H 'Content-Type:application/json' -d' {"nickname":"panda"}'
curl -XPOST http://localhost:9200/user_ik/user/11 -H 'Content-Type:application/json' -d' {"nickname":"aka. 橘子"}'
curl -XPOST http://localhost:9200/user_ik/user/12 -H 'Content-Type:application/json' -d' {"nickname":"中华人民共和国国歌"}'

检索

这里我们检索 aka. 橘子 这个条数据

curl -X GET \
  http://localhost:9200/user_ik/_search \
  -H 'Content-Type: application/json' \
  -d '{
    "query" : { "match" : { "nickname" : "ak" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "nickname" : {}
        }
    }
}'

最后检索不到

利用_analyze分析为什么检索不到

curl -X GET \
  http://localhost:9200/_analyze \
  -H 'Content-Type: application/json' \
  -d '{
  "tokenizer": "ik_max_word",
  "text": "aka. 橘子"
}'
{
    "tokens": [
        {
            "token": "aka.",
            "start_offset": 0,
            "end_offset": 4,
            "type": "LETTER",
            "position": 0
        },
        {
            "token": "aka",
            "start_offset": 0,
            "end_offset": 3,
            "type": "ENGLISH",
            "position": 1
        },
        {
            "token": "橘子",
            "start_offset": 5,
            "end_offset": 7,
            "type": "CN_WORD",
            "position": 2
        }
    ]
}

出来的分词结果并没有ak 所以我们自然是检索不到的

analysis-ngram-tokenizer

创建数据库

curl -X PUT \
  http://localhost:9200/user_ngram \
  -H 'Content-Type: application/json' \
  -d '{
  "settings": {
    "analysis": {
      "analyzer": {
        "ngram_analyzer": {
          "tokenizer": "ngram_tokenizer"
        }
      },
      "tokenizer": {
        "ngram_tokenizer": {
          "type": "ngram",
          "min_gram": 1,
          "max_gram": 1,
          "token_chars": [
            "letter",
            "digit",
            "symbol"
          ]
        }
      }
    }
  },
  "mappings": {
    "user": {
      "properties": {
        "nickname": {
          "type": "text",
          "analyzer": "ngram_analyzer"
        }
      }
    }
  }
}'

录入数据

curl -XPOST http://localhost:9200/user_ngram/user/1 -H 'Content-Type:application/json' -d' {"nickname":"༺ཎཽཾ࿄࿆大࿈宝࿄࿆ནཽཾ༻"}'
curl -XPOST http://localhost:9200/user_ngram/user/2 -H 'Content-Type:application/json' -d' {"nickname":"西瓜买的土豆"}'
curl -XPOST http://localhost:9200/user_ngram/user/3 -H 'Content-Type:application/json' -d' {"nickname":"Yvonne✨"}'
curl -XPOST http://localhost:9200/user_ngram/user/4 -H 'Content-Type:application/json' -d' {"nickname":"💗不想💗錯過你💗"}'
curl -XPOST http://localhost:9200/user_ngram/user/5 -H 'Content-Type:application/json' -d' {"nickname":"算了就算麻辣烫"}'
curl -XPOST http://localhost:9200/user_ngram/user/6 -H 'Content-Type:application/json' -d' {"nickname":"你好高，我怕够不着"}'
curl -XPOST http://localhost:9200/user_ngram/user/7 -H 'Content-Type:application/json' -d' {"nickname":"Sophie"}'
curl -XPOST http://localhost:9200/user_ngram/user/8 -H 'Content-Type:application/json' -d' {"nickname":"🤪"}'
curl -XPOST http://localhost:9200/user_ngram/user/9 -H 'Content-Type:application/json' -d' {"nickname":"Hermione🐯"}'
curl -XPOST http://localhost:9200/user_ngram/user/10 -H 'Content-Type:application/json' -d' {"nickname":"panda"}'
curl -XPOST http://localhost:9200/user_ngram/user/11 -H 'Content-Type:application/json' -d' {"nickname":"aka. 橘子"}'
curl -XPOST http://localhost:9200/user_ngram/user/12 -H 'Content-Type:application/json' -d' {"nickname":"中华人民共和国国歌"}'

检索

curl -X GET \
  http://localhost:9200/user_ngram/_search \
  -H 'Content-Type: application/json' \
  -d '{
    "query" : { "match" : { "nickname" : "ak" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "nickname" : {}
        }
    }
}'

检索结果

{
    "took": 9,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 2,
        "max_score": 2.7314157,
        "hits": [
            {
                "_index": "user_ngram",
                "_type": "user",
                "_id": "11",
                "_score": 2.7314157,
                "_source": {
                    "nickname": "aka. 橘子"
                },
                "highlight": {
                    "nickname": [
                        "<tag1>a</tag1><tag1>k</tag1><tag1>a</tag1>. 橘子"
                    ]
                }
            },
            {
                "_index": "user_ngram",
                "_type": "user",
                "_id": "10",
                "_score": 1.936579,
                "_source": {
                    "nickname": "panda"
                },
                "highlight": {
                    "nickname": [
                        "p<tag1>a</tag1>nd<tag1>a</tag1>"
                    ]
                }
            }
        ]
    }
}

_analyze拆词分析

curl -X GET \
  http://localhost:9200/_analyze \
  -H 'Content-Type: application/json' \
  -d '{
  "tokenizer": "ngram",
  "text": "aka. 橘子"
}'
token start_offset end_offset type position
a 0 1 word 0
ak 0 2 word 1
k 1 2 word 2
ka 1 2 word 2
a 2 3 word 4
a. 2 4 word 5
. 3 4 word 6
. 3 5 word 7
4 5 word 8
橘 4 6 word 9
橘 5 6 word 10
橘子 5 7 word 11
子 6 7 word 12