系列文章索引

Elasticsearch實(shí)戰(zhàn)（一）：Springboot實(shí)現(xiàn)Elasticsearch統(tǒng)一檢索功能
Elasticsearch實(shí)戰(zhàn)（二）：Springboot實(shí)現(xiàn)Elasticsearch自動(dòng)漢字、拼音補(bǔ)全，Springboot實(shí)現(xiàn)自動(dòng)拼寫糾錯(cuò)
Elasticsearch實(shí)戰(zhàn)（三）：Springboot實(shí)現(xiàn)Elasticsearch搜索推薦
Elasticsearch實(shí)戰(zhàn)（四）：Springboot實(shí)現(xiàn)Elasticsearch指標(biāo)聚合與下鉆分析
Elasticsearch實(shí)戰(zhàn)（五）：Springboot實(shí)現(xiàn)Elasticsearch電商平臺(tái)日志埋點(diǎn)與搜索熱詞

一、指標(biāo)聚合與分類

1、什么是指標(biāo)聚合（Metric）

聚合分析是數(shù)據(jù)庫(kù)中重要的功能特性，完成對(duì)某個(gè)查詢的數(shù)據(jù)集中數(shù)據(jù)的聚合計(jì)算，
如：找出某字段（或計(jì)算表達(dá)式的結(jié)果）的最大值、最小值，計(jì)算和、平均值等。
ES作為搜索引擎兼數(shù)據(jù)庫(kù)，同樣提供了強(qiáng)大的聚合分析能力。
對(duì)一個(gè)數(shù)據(jù)集求最大值、最小值，計(jì)算和、平均值等指標(biāo)的聚合，在ES中稱為指標(biāo)聚合。

2、Metric聚合分析分為單值分析和多值分析兩類

1、單值分析，只輸出一個(gè)分析結(jié)果
min,max,avg,sum,cardinality（cardinality 求唯一值，即不重復(fù)的字段有多少（相當(dāng)于mysql中的distinct）
2、多值分析，輸出多個(gè)分析結(jié)果
stats,extended_stats,percentile,percentile_rank

3、概述

官網(wǎng)：https://www.elastic.co/guide/en/elasticsearch/reference/7.4/search-aggregations-metrics.html
語(yǔ)法：

"aggregations" : {"<aggregation_name>" : { <!--聚合的名字 -->"<aggregation_type>" : { <!--聚合的類型 --><aggregation_body> <!--聚合體：對(duì)哪些字段進(jìn)行聚合 -->}[,"meta" : { [<meta_data_body>] } ]? <!--元 -->[,"aggregations" : { [<sub_aggregation>]+ } ]? <!--在聚合里面在定義子聚合-->}[,"<aggregation_name_2>" : { ... } ]* <!--聚合的名字 -->
}

openAPI設(shè)計(jì)目標(biāo)與原則：
1、DSL調(diào)用與語(yǔ)法進(jìn)行高度抽象，參數(shù)動(dòng)態(tài)設(shè)計(jì)
2、Open API通過(guò)結(jié)果轉(zhuǎn)換器支持上百種組合調(diào)用qurey,constant_score,match/matchall/filter/sort/size/frm/higthlight/_source/includes
3、邏輯處理公共調(diào)用，提升API業(yè)務(wù)處理能力
4、保留原生API與參數(shù)的用法

二、單值分析API設(shè)計(jì)

1、Avg(平均值)

從聚合文檔中提取的價(jià)格的平均值。

（1）對(duì)所有文檔進(jìn)行avg聚合（DSL）

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"avg": {"field": "price"}}}
}

以上匯總計(jì)算了所有文檔的平均值。
“size”: 0, 表示只查詢文檔聚合數(shù)量，不查文檔，如查詢50，size=50
aggs：表示是一個(gè)聚合
result：可自定義，聚合后的數(shù)據(jù)將顯示在自定義字段中

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"avg": {"field": "price"}}}}
}

（2）對(duì)篩選后的文檔聚合

POST product_list_info/_search
{"size": 0,"query": {"term": {"onelevel": "手機(jī)通訊"}},"aggs": {"result": {"avg": {"field": "price"}}}
}

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"query": {"term": {"onelevel": "手機(jī)通訊"}},"aggs": {"result": {"avg": {"field": "price"}}}}
}

（3）根據(jù)Script計(jì)算平均值

es所使用的腳本語(yǔ)言是painless這是一門安全-高效的腳本語(yǔ)言,基于jvm的

#統(tǒng)計(jì)所有
POST product_list_info/_search?size=0
{"aggs": {"result": {"avg": {"script": {"source": "doc.evalcount.value"}}}}
}
結(jié)果："value" : 599929.2282791147
"source": "doc['evalcount']"
"source": "doc.evalcount"

#有條件
POST product_list_info/_search?size=0
{"query": {"term": {"onelevel": "手機(jī)通訊"}},"aggs": {"czbk": {"avg": {"script": {"source": "doc.evalcount"}}}}
}
結(jié)果："value" : 600055.6935087288

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"czbk": {"avg": {"script": {"source": "doc.evalcount"}}}}}
}

（4）總結(jié)

avg平均
1、統(tǒng)一avg（所有文檔）
2、有條件avg（部分文檔）
3、腳本統(tǒng)計(jì)（所有）
4、腳本統(tǒng)計(jì)（部分）

2、Max(最大值)

計(jì)算從聚合文檔中提取的數(shù)值的最大值。

（1）統(tǒng)計(jì)所有文檔

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"max": {"field": "price"}}}
}

結(jié)果： “value” : 9.9999999E7

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"max": {"field": "price"}}}}
}

（2）統(tǒng)計(jì)過(guò)濾后的文檔

POST product_list_info/_search
{"size": 0,"query": {"term": {"onelevel": "手機(jī)通訊"}},"aggs": {"result": {"max": {"field": "price"}}}
}

結(jié)果： “value” : 2474000.0

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"query": {"term": {"onelevel": "手機(jī)通訊"}},"aggs": {"czbk": {"max": {"field": "price"}}}}
}

結(jié)果： “value” : 2474000.0

3、Min(最小值)

計(jì)算從聚合文檔中提取的數(shù)值的最小值。

（1）統(tǒng)計(jì)所有文檔

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"min": {"field": "price"}}}
}

結(jié)果：“value”: 0.0

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"min": {"field": "price"}}}}
}

（2）統(tǒng)計(jì)篩選后的文檔

POST product_list_info/_search
{"size": 1,"query": {"term": {"onelevel": "手機(jī)通訊"}},"aggs": {"czbk": {"min": {"field": "price"}}}
}

結(jié)果：“value”: 0.0

參數(shù)size=1；可查詢出金額為0的數(shù)據(jù)

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 1,"query": {"term": {"onelevel": "手機(jī)通訊"}},"aggs": {"result": {"min": {"field": "price"}}}}
}

4、Sum(總和)

（1）統(tǒng)計(jì)所有文檔匯總

POST product_list_info/_search
{"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"sum": {"field": "price"}}}
}

結(jié)果：“value” : 3.433611809E7

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"sum": {"field": "price"}}}}
}

5、Cardinality(唯一值)

Cardinality Aggregation，基數(shù)聚合。它屬于multi-value，基于文檔的某個(gè)值（可以是特定的字段，也可以通過(guò)腳本計(jì)算而來(lái)），計(jì)算文檔非重復(fù)的個(gè)數(shù)（去重計(jì)數(shù)），相當(dāng)于sql中的distinct。

cardinality 求唯一值，即不重復(fù)的字段有多少（相當(dāng)于mysql中的distinct）

（1）統(tǒng)計(jì)所有文檔

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"cardinality": {"field": "storename"}}}
}

結(jié)果：“value” : 103169

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"cardinality": {"field": "storename"}}}}
}

（2）統(tǒng)計(jì)篩選后的文檔

POST product_list_info/_search
{"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"cardinality": {"field": "storename"}}}
}

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"cardinality": {"field": "storename"}}}}
}

三、多值分析API設(shè)計(jì)

1、Stats Aggregation

Stats Aggregation，統(tǒng)計(jì)聚合。它屬于multi-value，基于文檔的某個(gè)值（可以是特定的數(shù)值型字段，也可以通過(guò)腳本計(jì)算而來(lái)），計(jì)算出一些統(tǒng)計(jì)信息（min、max、sum、count、avg 5個(gè)值）

（1）統(tǒng)計(jì)所有文檔

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"stats": {"field": "price"}}}
}返回
"aggregations" : {"result" : {"count" : 5072447,"min" : 0.0,"max" : 9.9999999E7,"avg" : 920.1537270512633,"sum" : 4.66743101232E9

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"stats": {"field": "price"}}}}
}

（2）統(tǒng)計(jì)篩選文檔

POST product_list_info/_search
{"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"stats": {"field": "price"}}}
}

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"stats": {"field": "price"}}}}
}

2、擴(kuò)展?fàn)顟B(tài)統(tǒng)計(jì)

Extended Stats Aggregation，擴(kuò)展統(tǒng)計(jì)聚合。它屬于multi-value，比stats多4個(gè)統(tǒng)計(jì)結(jié)果： 平方和、方差、標(biāo)準(zhǔn)差、平均值加/減兩個(gè)標(biāo)準(zhǔn)差的區(qū)間

（1）統(tǒng)計(jì)所有文檔

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"extended_stats": {"field": "price"}}}
}
返回：
aggregations" : {"result" : {"count" : 5072447,"min" : 0.0,"max" : 9.9999999E7,"avg" : 920.1537270512633,"sum" : 4.66743101232E9,"sum_of_squares" : 2.0182209054045464E16,"variance" : 3.9779448262354884E9,"std_deviation" : 63070.950731977144,"std_deviation_bounds" : {"upper" : 127062.05519100555,"lower" : -125221.74773690302}

sum_of_squares:平方和
variance：方差
std_deviation：標(biāo)準(zhǔn)差
std_deviation_bounds：標(biāo)準(zhǔn)差的區(qū)間

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"extended_stats": {"field": "price"}}}}
}

（2）統(tǒng)計(jì)篩選后的文檔

POST product_list_info/_search
{"size": 1,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"extended_stats": {"field": "price"}}}
}結(jié)果;
aggregations" : {"result" : {"count" : 12402,"min" : 0.0,"max" : 2474000.0,"avg" : 2768.595233833253,"sum" : 3.433611809E7,"sum_of_squares" : 6.445447222627729E12,"variance" : 5.120451870452684E8,"std_deviation" : 22628.41547800615,"std_deviation_bounds" : {"upper" : 48025.42618984555,"lower" : -42488.23572217905

sum_of_squares:平方和
variance：方差
std_deviation：標(biāo)準(zhǔn)差
std_deviation_bounds：標(biāo)準(zhǔn)差的區(qū)間

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 1,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"czbk": {"extended_stats": {"field": "price"}}}}
}

3、百分位度量/百分比統(tǒng)計(jì)

Percentiles Aggregation，百分比聚合。它屬于multi-value，對(duì)指定字段（腳本）的值按從小到大累計(jì)每個(gè)值對(duì)應(yīng)的文檔數(shù)的占比（占所有命中文檔數(shù)的百分比），返回指定占比比例對(duì)應(yīng)的值。默認(rèn)返回[1, 5, 25, 50, 75, 95, 99 ]分位上的值。

它們表示了人們感興趣的常用百分位數(shù)值。

（1）統(tǒng)計(jì)所有文檔

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"percentiles": {"field": "price"}}}
}返回:
aggregations" : {"result" : {"values" : {"1.0" : 0.0,"5.0" : 15.021825109603165,"25.0" : 58.669333121791,"50.0" : 139.7398105623917,"75.0" : 388.2363222057536,"95.0" : 3630.78148822216,"99.0" : 12561.562823894474}}

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"percentiles": {"field": "price"}}}}
}

（2）統(tǒng)計(jì)篩選后的文檔

POST product_list_info/_search
{"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"percentiles": {"field": "price"}}}
}

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"percentiles": {"field": "price"}}}}
}

4、百分位等級(jí)/百分比排名聚合

百分比排名聚合：這里有另外一個(gè)緊密相關(guān)的度量叫 percentile_ranks 。 percentiles 度量告訴我們落在某個(gè)百分比以下的所有文檔的最小值。

（1）統(tǒng)計(jì)所有文檔

統(tǒng)計(jì)價(jià)格在15元之內(nèi)統(tǒng)計(jì)價(jià)格在30元之內(nèi)文檔數(shù)據(jù)占有的百分比

tips：
統(tǒng)計(jì)數(shù)據(jù)會(huì)變化
這里的15和30；完全可以理解萬(wàn)SLA的200；比較字段不一樣而已

POST product_list_info/_search
{"size": 0,"aggs": {"result": {"percentile_ranks": {"field": "price","values": [15,30]}}}
}返回：
價(jià)格在15元之內(nèi)的文檔數(shù)據(jù)占比是4.92%
價(jià)格在30元之內(nèi)的文檔數(shù)據(jù)占比是12.72%
aggregations" : {"result" : {"values" : {"15.0" : 4.92128378837021,"30.0" : 12.724827959646579}}
}

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"aggs": {"result": {"percentile_ranks": {"field": "price","values": [15,30]}}}}
}

（2）統(tǒng)計(jì)過(guò)濾后的文檔

POST product_list_info/_search
{"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"percentile_ranks": {"field": "price","values": [15,30]}}}
}

OpenAPI查詢參數(shù)設(shè)計(jì)：

{"indexName": "product_list_info","map": {"size": 0,"query": {"constant_score": {"filter": {"match": {"threelevel": "手機(jī)"}}}},"aggs": {"result": {"percentile_ranks": {"field": "price","values": [15,30]}}}}
}

四、JavaAPI實(shí)現(xiàn)

調(diào)用metricAgg方法，傳參CommonEntity 。

/** @Description: 指標(biāo)聚合(Open)* @Method: metricAgg* @Param: [commonEntity]* @Update:* @since: 1.0.0* @Return: java.util.Map<java.lang.String,java.lang.Long>**/
public Map<Object, Object> metricAgg(CommonEntity commonEntity) throws Exception {//查詢公共調(diào)用,將參數(shù)模板化SearchResponse response = getSearchResponse(commonEntity);//定義返回?cái)?shù)據(jù)Map<Object, Object> map = new HashMap<Object, Object>();// 此處完全可以返回ParsedAggregation ，不用instance，弊端是返回的數(shù)據(jù)字段多、get的時(shí)候需要寫死，下面循環(huán)map為的是動(dòng)態(tài)獲取keyMap<String, Aggregation> aggregationMap = response.getAggregations().asMap();// 將查詢出來(lái)的數(shù)據(jù)放到本地局部線程變量中SearchTools.setResponseThreadLocal(response);//此處循環(huán)一次，目的是動(dòng)態(tài)獲取client端傳來(lái)的【result】for (Map.Entry<String, Aggregation> m : aggregationMap.entrySet()) {//處理指標(biāo)聚合metricResultConverter(map, m);}//公共數(shù)據(jù)處理mbCommonConverter(map);return map;
}
/** @Description: 查詢公共調(diào)用,參數(shù)模板化* @Method: getSearchResponse* @Param: [commonEntity]* @Update:* @since: 1.0.0* @Return: org.elasticsearch.action.search.SearchResponse**/
private SearchResponse getSearchResponse(CommonEntity commonEntity) throws Exception {//定義查詢請(qǐng)求SearchRequest searchRequest = new SearchRequest();//指定去哪個(gè)索引查詢searchRequest.indices(commonEntity.getIndexName());//構(gòu)建資源查詢構(gòu)建器，主要用于拼接查詢條件SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();//將前端的dsl查詢轉(zhuǎn)化為XContentParserXContentParser parser = SearchTools.getXContentParser(commonEntity);//將parser解析成功查詢APIsourceBuilder.parseXContent(parser);//將sourceBuilder賦給searchRequestsearchRequest.source(sourceBuilder);//執(zhí)行查詢SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);return response;
}
/** @Description: 指標(biāo)聚合結(jié)果轉(zhuǎn)化器* @Method: metricResultConverter* @Param: [map, m]* @Update:* @since: 1.0.0* @Return: void**/
private void metricResultConverter(Map<Object, Object> map, Map.Entry<String, Aggregation> m) {//平均值if (m.getValue() instanceof ParsedAvg) {map.put("value", ((ParsedAvg) m.getValue()).getValue());}//最大值else if (m.getValue() instanceof ParsedMax) {map.put("value", ((ParsedMax) m.getValue()).getValue());}//最小值else if (m.getValue() instanceof ParsedMin) {map.put("value", ((ParsedMin) m.getValue()).getValue());}//求和else if (m.getValue() instanceof ParsedSum) {map.put("value", ((ParsedSum) m.getValue()).getValue());}//不重復(fù)的值else if (m.getValue() instanceof ParsedCardinality) {map.put("value", ((ParsedCardinality) m.getValue()).getValue());}//擴(kuò)展?fàn)顟B(tài)統(tǒng)計(jì)else if (m.getValue() instanceof ParsedExtendedStats) {map.put("count", ((ParsedExtendedStats) m.getValue()).getCount());map.put("min", ((ParsedExtendedStats) m.getValue()).getMin());map.put("max", ((ParsedExtendedStats) m.getValue()).getMax());map.put("avg", ((ParsedExtendedStats) m.getValue()).getAvg());map.put("sum", ((ParsedExtendedStats) m.getValue()).getSum());map.put("sum_of_squares", ((ParsedExtendedStats) m.getValue()).getSumOfSquares());map.put("variance", ((ParsedExtendedStats) m.getValue()).getVariance());map.put("std_deviation", ((ParsedExtendedStats) m.getValue()).getStdDeviation());map.put("lower", ((ParsedExtendedStats) m.getValue()).getStdDeviationBound(ExtendedStats.Bounds.LOWER));map.put("upper", ((ParsedExtendedStats) m.getValue()).getStdDeviationBound(ExtendedStats.Bounds.UPPER));}//狀態(tài)統(tǒng)計(jì)else if (m.getValue() instanceof ParsedStats) {map.put("count", ((ParsedStats) m.getValue()).getCount());map.put("min", ((ParsedStats) m.getValue()).getMin());map.put("max", ((ParsedStats) m.getValue()).getMax());map.put("avg", ((ParsedStats) m.getValue()).getAvg());map.put("sum", ((ParsedStats) m.getValue()).getSum());}//百分位等級(jí)else if (m.getValue() instanceof ParsedTDigestPercentileRanks) {for (Iterator<Percentile> iterator = ((ParsedTDigestPercentileRanks) m.getValue()).iterator(); iterator.hasNext(); ) {Percentile p = (Percentile) iterator.next();map.put(p.getValue(), p.getPercent());}}//百分位度量else if (m.getValue() instanceof ParsedTDigestPercentiles) {for (Iterator<Percentile> iterator = ((ParsedTDigestPercentiles) m.getValue()).iterator(); iterator.hasNext(); ) {Percentile p = (Percentile) iterator.next();map.put(p.getPercent(), p.getValue());}}}/** @Description: 公共數(shù)據(jù)處理(指標(biāo)聚合、桶聚合)* @Method: mbCommonConverter* @Param: []* @Update:* @since: 1.0.0* @Return: void**/
private void mbCommonConverter(Map<Object, Object> map) {if (!CollectionUtils.isEmpty(ResponseThreadLocal.get())) {//從線程中取出數(shù)據(jù)map.put("list", ResponseThreadLocal.get());//清空本地線程局部變量中的數(shù)據(jù)，防止內(nèi)存泄露ResponseThreadLocal.clear();}}