Skip to content

Commit 3168a8c

Browse files
github-actions[bot]lcawl
github-actions[bot]
and
lcawl
authored
Add more examples (#3487) (#3493)
(cherry picked from commit 10e3a0a) Co-authored-by: Lisa Cawley <lcawley@elastic.co>
1 parent bbae345 commit 3168a8c

File tree

14 files changed

+616
-77
lines changed

14 files changed

+616
-77
lines changed

‎output/openapi/elasticsearch-openapi.json

Lines changed: 34 additions & 23 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎output/schema/schema.json

Lines changed: 47 additions & 31 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎specification/text_structure/find_field_structure/FindFieldStructureRequest.ts

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,22 @@ import { EcsCompatibilityType, FormatType } from '../_types/Structure'
2626
/**
2727
* Find the structure of a text field.
2828
* Find the structure of a text field in an Elasticsearch index.
29+
*
30+
* This API provides a starting point for extracting further information from log messages already ingested into Elasticsearch.
31+
* For example, if you have ingested data into a very simple index that has just `@timestamp` and message fields, you can use this API to see what common structure exists in the message field.
32+
*
33+
* The response from the API contains:
34+
*
35+
* * Sample messages.
36+
* * Statistics that reveal the most common values for all fields detected within the text and basic numeric statistics for numeric fields.
37+
* * Information about the structure of the text, which is useful when you write ingest configurations to index it or similarly formatted text.
38+
* * Appropriate mappings for an Elasticsearch index, which you could use to ingest the text.
39+
*
40+
* All this information can be calculated by the structure finder with no guidance.
41+
* However, you can optionally override some of the decisions about the text structure by specifying one or more query parameters.
42+
*
43+
* If the structure finder produces unexpected results, specify the `explain` query parameter and an explanation will appear in the response.
44+
* It helps determine why the returned structure was chosen.
2945
* @rest_spec_name text_structure.find_field_structure
3046
* @availability stack stability=stable visibility=public
3147
* @cluster_privileges monitor_text_structure
@@ -63,7 +79,7 @@ interface Request extends RequestBase {
6379
*/
6480
ecs_compatibility?: EcsCompatibilityType
6581
/**
66-
* If true, the response includes a field named `explanation`, which is an array of strings that indicate how the structure finder produced its result.
82+
* If `true`, the response includes a field named `explanation`, which is an array of strings that indicate how the structure finder produced its result.
6783
* @server_default false
6884
*/
6985
explain?: boolean
@@ -99,7 +115,7 @@ interface Request extends RequestBase {
99115
/**
100116
* If the format is `delimited`, you can specify whether values between delimiters should have whitespace trimmed from them.
101117
* If this parameter is not specified and the delimiter is pipe (`|`), the default value is true.
102-
* Otherwise, the default value is false.
118+
* Otherwise, the default value is `false`.
103119
*/
104120
should_trim_fields?: boolean
105121
/**

‎specification/text_structure/find_field_structure/examples/response/FindFieldStructureResponseExample1.yaml

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# summary:
2+
description: A successful response from `GET _text_structure/find_field_structure?index=test-logs&field=message`.
3+
# type: response
4+
# response_code: ''
5+
value:
6+
"{\n \"num_lines_analyzed\" : 22,\n \"num_messages_analyzed\" : 22,\n \"\
7+
sample_start\" : \"[2024-03-05T10:52:36,256][INFO ][o.a.l.u.VectorUtilPanamaProvider]\
8+
\ [laptop] Java vector incubator API enabled; uses preferredBitSize=128\\n[2024-03-05T10:52:41,038][INFO\
9+
\ ][o.e.p.PluginsService ] [laptop] loaded module [repository-url]\\n\",\n \
10+
\ \"charset\" : \"UTF-8\",\n \"format\" : \"semi_structured_text\",\n \"multiline_start_pattern\"\
11+
\ : \"^\\\\[\\\\b\\\\d{4}-\\\\d{2}-\\\\d{2}[T ]\\\\d{2}:\\\\d{2}\",\n \"grok_pattern\"\
12+
\ : \"\\\\[%{TIMESTAMP_ISO8601:timestamp}\\\\]\\\\[%{LOGLEVEL:loglevel} \\\\]\\\\\
13+
[.*\",\n \"ecs_compatibility\" : \"disabled\",\n \"timestamp_field\" : \"timestamp\"\
14+
,\n \"joda_timestamp_formats\" : [\n \"ISO8601\"\n ],\n \"java_timestamp_formats\"\
15+
\ : [\n \"ISO8601\"\n ],\n \"need_client_timezone\" : true,\n \"mappings\"\
16+
\ : {\n \"properties\" : {\n \"@timestamp\" : {\n \"type\" : \"date\"\
17+
\n },\n \"loglevel\" : {\n \"type\" : \"keyword\"\n },\n \
18+
\ \"message\" : {\n \"type\" : \"text\"\n }\n }\n },\n \"ingest_pipeline\"\
19+
\ : {\n \"description\" : \"Ingest pipeline created by text structure finder\"\
20+
,\n \"processors\" : [\n {\n \"grok\" : {\n \"field\" :\
21+
\ \"message\",\n \"patterns\" : [\n \"\\\\[%{TIMESTAMP_ISO8601:timestamp}\\\
22+
\\]\\\\[%{LOGLEVEL:loglevel} \\\\]\\\\[.*\"\n ],\n \"ecs_compatibility\"\
23+
\ : \"disabled\"\n }\n },\n {\n \"date\" : {\n \
24+
\ \"field\" : \"timestamp\",\n \"timezone\" : \"{{ event.timezone }}\"\
25+
,\n \"formats\" : [\n \"ISO8601\"\n ]\n }\n\
26+
\ },\n {\n \"remove\" : {\n \"field\" : \"timestamp\"\n\
27+
\ }\n }\n ]\n },\n \"field_stats\" : {\n \"loglevel\" : {\n\
28+
\ \"count\" : 22,\n \"cardinality\" : 1,\n \"top_hits\" : [\n \
29+
\ {\n \"value\" : \"INFO\",\n \"count\" : 22\n }\n\
30+
\ ]\n },\n \"message\" : {\n \"count\" : 22,\n \"cardinality\"\
31+
\ : 22,\n \"top_hits\" : [\n {\n \"value\" : \"[2024-03-05T10:52:36,256][INFO\
32+
\ ][o.a.l.u.VectorUtilPanamaProvider] [laptop] Java vector incubator API enabled;\
33+
\ uses preferredBitSize=128\",\n \"count\" : 1\n },\n {\n\
34+
\ \"value\" : \"[2024-03-05T10:52:41,038][INFO ][o.e.p.PluginsService \
35+
\ ] [laptop] loaded module [repository-url]\",\n \"count\" : 1\n \
36+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,042][INFO ][o.e.p.PluginsService\
37+
\ ] [laptop] loaded module [rest-root]\",\n \"count\" : 1\n \
38+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,043][INFO ][o.e.p.PluginsService\
39+
\ ] [laptop] loaded module [ingest-user-agent]\",\n \"count\" : 1\n\
40+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,043][INFO ][o.e.p.PluginsService\
41+
\ ] [laptop] loaded module [x-pack-core]\",\n \"count\" : 1\n \
42+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,043][INFO ][o.e.p.PluginsService\
43+
\ ] [laptop] loaded module [x-pack-redact]\",\n \"count\" : 1\n \
44+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
45+
\ ] [laptop] loaded module [lang-painless]]\",\n \"count\" : 1\n \
46+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
47+
\ ] [laptop] loaded module [repository-s3]\",\n \"count\" : 1\n \
48+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
49+
\ ] [laptop] loaded module [x-pack-analytics]\",\n \"count\" : 1\n\
50+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
51+
\ ] [laptop] loaded module [x-pack-autoscaling]\",\n \"count\" : 1\n\
52+
\ }\n ]\n },\n \"timestamp\" : {\n \"count\" : 22,\n \
53+
\ \"cardinality\" : 14,\n \"earliest\" : \"2024-03-05T10:52:36,256\",\n \
54+
\ \"latest\" : \"2024-03-05T10:52:49,199\",\n \"top_hits\" : [\n \
55+
\ {\n \"value\" : \"2024-03-05T10:52:41,044\",\n \"count\" : 6\n\
56+
\ },\n {\n \"value\" : \"2024-03-05T10:52:41,043\",\n \
57+
\ \"count\" : 3\n },\n {\n \"value\" : \"2024-03-05T10:52:41,059\"\
58+
,\n \"count\" : 2\n },\n {\n \"value\" : \"2024-03-05T10:52:36,256\"\
59+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:41,038\"\
60+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:41,042\"\
61+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:43,291\"\
62+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:46,098\"\
63+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:47,227\"\
64+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:47,259\"\
65+
,\n \"count\" : 1\n }\n ]\n }\n }\n}"

‎specification/text_structure/find_message_structure/FindMessageStructureRequest.ts

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ import { EcsCompatibilityType, FormatType } from '../_types/Structure'
2929
*
3030
* This API provides a starting point for ingesting data into Elasticsearch in a format that is suitable for subsequent use with other Elastic Stack functionality.
3131
* Use this API rather than the find text structure API if your input text has already been split up into separate messages by some other process.
32+
*
3233
* The response from the API contains:
3334
*
3435
* * Sample messages.
@@ -38,6 +39,9 @@ import { EcsCompatibilityType, FormatType } from '../_types/Structure'
3839
*
3940
* All this information can be calculated by the structure finder with no guidance.
4041
* However, you can optionally override some of the decisions about the text structure by specifying one or more query parameters.
42+
*
43+
* If the structure finder produces unexpected results, specify the `explain` query parameter and an explanation will appear in the response.
44+
* It helps determine why the returned structure was chosen.
4145
* @rest_spec_name text_structure.find_message_structure
4246
* @availability stack stability=stable visibility=public
4347
* @cluster_privileges monitor_text_structure
@@ -71,7 +75,8 @@ interface Request extends RequestBase {
7175
* @server_default false
7276
*/
7377
explain?: boolean
74-
/** The high level structure of the text.
78+
/**
79+
* The high level structure of the text.
7580
* By default, the API chooses the format.
7681
* In this default scenario, all rows must have the same number of fields for a delimited format to be detected.
7782
* If the format is `delimited` and the delimiter is not set, however, the API tolerates up to 5% of rows that have a different number of columns than the first row.
@@ -94,7 +99,7 @@ interface Request extends RequestBase {
9499
/**
95100
* If the format is `delimited`, you can specify whether values between delimiters should have whitespace trimmed from them.
96101
* If this parameter is not specified and the delimiter is pipe (`|`), the default value is true.
97-
* Otherwise, the default value is false.
102+
* Otherwise, the default value is `false`.
98103
*/
99104
should_trim_fields?: boolean
100105
/**

‎specification/text_structure/find_message_structure/examples/request/FindMessageStructureRequestExample1.yaml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# summary:
2+
# method_request: POST _text_structure/find_message_structure
3+
description: >
4+
Run `POST _text_structure/find_message_structure` to analyze Elasticsearch log files.
5+
# type: request
6+
value:
7+
"{\n \"messages\": [\n \"[2024-03-05T10:52:36,256][INFO ][o.a.l.u.VectorUtilPanamaProvider]\
8+
\ [laptop] Java vector incubator API enabled; uses preferredBitSize=128\",\n \
9+
\ \"[2024-03-05T10:52:41,038][INFO ][o.e.p.PluginsService ] [laptop] loaded\
10+
\ module [repository-url]\",\n \"[2024-03-05T10:52:41,042][INFO ][o.e.p.PluginsService\
11+
\ ] [laptop] loaded module [rest-root]\",\n \"[2024-03-05T10:52:41,043][INFO\
12+
\ ][o.e.p.PluginsService ] [laptop] loaded module [x-pack-core]\",\n \"[2024-03-05T10:52:41,043][INFO\
13+
\ ][o.e.p.PluginsService ] [laptop] loaded module [x-pack-redact]\",\n \"\
14+
[2024-03-05T10:52:41,043][INFO ][o.e.p.PluginsService ] [laptop] loaded module\
15+
\ [ingest-user-agent]\",\n \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
16+
\ ] [laptop] loaded module [x-pack-monitoring]\",\n \"[2024-03-05T10:52:41,044][INFO\
17+
\ ][o.e.p.PluginsService ] [laptop] loaded module [repository-s3]\",\n \"\
18+
[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService ] [laptop] loaded module\
19+
\ [x-pack-analytics]\",\n \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
20+
\ ] [laptop] loaded module [x-pack-ent-search]\",\n \"[2024-03-05T10:52:41,044][INFO\
21+
\ ][o.e.p.PluginsService ] [laptop] loaded module [x-pack-autoscaling]\",\n\
22+
\ \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService ] [laptop] loaded\
23+
\ module [lang-painless]]\",\n \"[2024-03-05T10:52:41,059][INFO ][o.e.p.PluginsService\
24+
\ ] [laptop] loaded module [lang-expression]\",\n \"[2024-03-05T10:52:41,059][INFO\
25+
\ ][o.e.p.PluginsService ] [laptop] loaded module [x-pack-eql]\",\n \"[2024-03-05T10:52:43,291][INFO\
26+
\ ][o.e.e.NodeEnvironment ] [laptop] heap size [16gb], compressed ordinary object\
27+
\ pointers [true]\",\n \"[2024-03-05T10:52:46,098][INFO ][o.e.x.s.Security \
28+
\ ] [laptop] Security is enabled\",\n \"[2024-03-05T10:52:47,227][INFO\
29+
\ ][o.e.x.p.ProfilingPlugin ] [laptop] Profiling is enabled\",\n \"[2024-03-05T10:52:47,259][INFO\
30+
\ ][o.e.x.p.ProfilingPlugin ] [laptop] profiling index templates will not be installed\
31+
\ or reinstalled\",\n \"[2024-03-05T10:52:47,755][INFO ][o.e.i.r.RecoverySettings\
32+
\ ] [laptop] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]\"\
33+
,\n \"[2024-03-05T10:52:47,787][INFO ][o.e.d.DiscoveryModule ] [laptop] using\
34+
\ discovery type [multi-node] and seed hosts providers [settings]\",\n \"[2024-03-05T10:52:49,188][INFO\
35+
\ ][o.e.n.Node ] [laptop] initialized\",\n \"[2024-03-05T10:52:49,199][INFO\
36+
\ ][o.e.n.Node ] [laptop] starting ...\"\n ]\n}"

‎specification/text_structure/find_message_structure/examples/response/FindMessageStructureResponseExample1.yaml

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# summary:
2+
description: A successful response from `POST _text_structure/find_message_structure`.
3+
# type: response
4+
# response_code: ''
5+
value:
6+
"{\n \"num_lines_analyzed\" : 22,\n \"num_messages_analyzed\" : 22,\n \"\
7+
sample_start\" : \"[2024-03-05T10:52:36,256][INFO ][o.a.l.u.VectorUtilPanamaProvider]\
8+
\ [laptop] Java vector incubator API enabled; uses preferredBitSize=128\\n[2024-03-05T10:52:41,038][INFO\
9+
\ ][o.e.p.PluginsService ] [laptop] loaded module [repository-url]\\n\",\n \
10+
\ \"charset\" : \"UTF-8\",\n \"format\" : \"semi_structured_text\",\n \"multiline_start_pattern\"\
11+
\ : \"^\\\\[\\\\b\\\\d{4}-\\\\d{2}-\\\\d{2}[T ]\\\\d{2}:\\\\d{2}\",\n \"grok_pattern\"\
12+
\ : \"\\\\[%{TIMESTAMP_ISO8601:timestamp}\\\\]\\\\[%{LOGLEVEL:loglevel} \\\\]\\\\\
13+
[.*\",\n \"ecs_compatibility\" : \"disabled\",\n \"timestamp_field\" : \"timestamp\"\
14+
,\n \"joda_timestamp_formats\" : [\n \"ISO8601\"\n ],\n \"java_timestamp_formats\"\
15+
\ : [\n \"ISO8601\"\n ],\n \"need_client_timezone\" : true,\n \"mappings\"\
16+
\ : {\n \"properties\" : {\n \"@timestamp\" : {\n \"type\" : \"date\"\
17+
\n },\n \"loglevel\" : {\n \"type\" : \"keyword\"\n },\n \
18+
\ \"message\" : {\n \"type\" : \"text\"\n }\n }\n },\n \"ingest_pipeline\"\
19+
\ : {\n \"description\" : \"Ingest pipeline created by text structure finder\"\
20+
,\n \"processors\" : [\n {\n \"grok\" : {\n \"field\" :\
21+
\ \"message\",\n \"patterns\" : [\n \"\\\\[%{TIMESTAMP_ISO8601:timestamp}\\\
22+
\\]\\\\[%{LOGLEVEL:loglevel} \\\\]\\\\[.*\"\n ],\n \"ecs_compatibility\"\
23+
\ : \"disabled\"\n }\n },\n {\n \"date\" : {\n \
24+
\ \"field\" : \"timestamp\",\n \"timezone\" : \"{{ event.timezone }}\"\
25+
,\n \"formats\" : [\n \"ISO8601\"\n ]\n }\n\
26+
\ },\n {\n \"remove\" : {\n \"field\" : \"timestamp\"\n\
27+
\ }\n }\n ]\n },\n \"field_stats\" : {\n \"loglevel\" : {\n\
28+
\ \"count\" : 22,\n \"cardinality\" : 1,\n \"top_hits\" : [\n \
29+
\ {\n \"value\" : \"INFO\",\n \"count\" : 22\n }\n\
30+
\ ]\n },\n \"message\" : {\n \"count\" : 22,\n \"cardinality\"\
31+
\ : 22,\n \"top_hits\" : [\n {\n \"value\" : \"[2024-03-05T10:52:36,256][INFO\
32+
\ ][o.a.l.u.VectorUtilPanamaProvider] [laptop] Java vector incubator API enabled;\
33+
\ uses preferredBitSize=128\",\n \"count\" : 1\n },\n {\n\
34+
\ \"value\" : \"[2024-03-05T10:52:41,038][INFO ][o.e.p.PluginsService \
35+
\ ] [laptop] loaded module [repository-url]\",\n \"count\" : 1\n \
36+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,042][INFO ][o.e.p.PluginsService\
37+
\ ] [laptop] loaded module [rest-root]\",\n \"count\" : 1\n \
38+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,043][INFO ][o.e.p.PluginsService\
39+
\ ] [laptop] loaded module [ingest-user-agent]\",\n \"count\" : 1\n\
40+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,043][INFO ][o.e.p.PluginsService\
41+
\ ] [laptop] loaded module [x-pack-core]\",\n \"count\" : 1\n \
42+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,043][INFO ][o.e.p.PluginsService\
43+
\ ] [laptop] loaded module [x-pack-redact]\",\n \"count\" : 1\n \
44+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
45+
\ ] [laptop] loaded module [lang-painless]]\",\n \"count\" : 1\n \
46+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
47+
\ ] [laptop] loaded module [repository-s3]\",\n \"count\" : 1\n \
48+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
49+
\ ] [laptop] loaded module [x-pack-analytics]\",\n \"count\" : 1\n\
50+
\ },\n {\n \"value\" : \"[2024-03-05T10:52:41,044][INFO ][o.e.p.PluginsService\
51+
\ ] [laptop] loaded module [x-pack-autoscaling]\",\n \"count\" : 1\n\
52+
\ }\n ]\n },\n \"timestamp\" : {\n \"count\" : 22,\n \
53+
\ \"cardinality\" : 14,\n \"earliest\" : \"2024-03-05T10:52:36,256\",\n \
54+
\ \"latest\" : \"2024-03-05T10:52:49,199\",\n \"top_hits\" : [\n \
55+
\ {\n \"value\" : \"2024-03-05T10:52:41,044\",\n \"count\" : 6\n\
56+
\ },\n {\n \"value\" : \"2024-03-05T10:52:41,043\",\n \
57+
\ \"count\" : 3\n },\n {\n \"value\" : \"2024-03-05T10:52:41,059\"\
58+
,\n \"count\" : 2\n },\n {\n \"value\" : \"2024-03-05T10:52:36,256\"\
59+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:41,038\"\
60+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:41,042\"\
61+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:43,291\"\
62+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:46,098\"\
63+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:47,227\"\
64+
,\n \"count\" : 1\n },\n {\n \"value\" : \"2024-03-05T10:52:47,259\"\
65+
,\n \"count\" : 1\n }\n ]\n }\n }\n}"

0 commit comments

Comments
 (0)