You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/extend/add-mapping.md
+31-35Lines changed: 31 additions & 35 deletions
Original file line number
Diff line number
Diff line change
@@ -19,47 +19,45 @@ In the integration, the `fields` directory serves as the blueprint used to creat
19
19
20
20
Like ingest pipelines, mappings only apply to the data stream dataset, for our example the `apache.access` dataset.
21
21
22
-
+ NOTE: The names of these files are conventions, any file name with a `.yml` extension will work.
22
+
:::{note}
23
+
The names of these files are conventions, any file name with a `.yml` extension will work.
24
+
:::
23
25
24
26
Integrations have had significant enhancements in how ECS fields are defined. Below is a guide on which approach to use, based on the version of Elastic your integration will support.
25
27
26
-
+. ECS mappings component template (>=8.13.0) Integrations **only** supporting version 8.13.0 and up, can use the [ecs@mappings](https://github.com/elastic/elasticsearch/blob/c2a3ec42632b0339387121efdef13f52c6c66848/x-pack/plugin/core/template-resources/src/main/resources/ecs%40mappings.json) component template installed by Fleet. This makes explicitly declaring ECS fields unnecessary; the `ecs@mappings` component template in Elasticsearch will automatically detect and configure them. However, should ECS fields be explicitly defined, they will overwrite the dynamic mapping provided by the `ecs@mappings` component template. They can also be imported with an `external` declaration, as seen in the example below.
28
+
1. ECS mappings component template (>=8.13.0) Integrations **only** supporting version 8.13.0 and up, can use the [ecs@mappings](https://github.com/elastic/elasticsearch/blob/c2a3ec42632b0339387121efdef13f52c6c66848/x-pack/plugin/core/template-resources/src/main/resources/ecs%40mappings.json) component template installed by Fleet. This makes explicitly declaring ECS fields unnecessary; the `ecs@mappings` component template in Elasticsearch will automatically detect and configure them. However, should ECS fields be explicitly defined, they will overwrite the dynamic mapping provided by the `ecs@mappings` component template. They can also be imported with an `external` declaration, as seen in the example below.
27
29
28
-
+. Dynamic mappings imports (<8.13.0 & >=8.13.0) Integrations supporting the Elastic stack below version 8.13.0 can still dynamically import ECS field mappings by defining `import_mappings: true` in the ECS section of the `_dev/build/build.yml` file in the root of the package directory. This introduces a [dynamic mapping](https://github.com/elastic/elastic-package/blob/f439b96a74c27c5adfc3e7810ad584204bfaf85d/internal/builder/_static/ecs_mappings.yaml) with most of the ECS definitions. Using this method means that, just like the previous approach, ECS fields don’t need to be defined in your integration, they are dynamically integrated into the package at build time. Explicitly defined ECS fields can be used and will also overwrite this mechanism.
30
+
1. Dynamic mappings imports (<8.13.0 & >=8.13.0) Integrations supporting the Elastic stack below version 8.13.0 can still dynamically import ECS field mappings by defining `import_mappings: true` in the ECS section of the `_dev/build/build.yml` file in the root of the package directory. This introduces a [dynamic mapping](https://github.com/elastic/elastic-package/blob/f439b96a74c27c5adfc3e7810ad584204bfaf85d/internal/builder/_static/ecs_mappings.yaml) with most of the ECS definitions. Using this method means that, just like the previous approach, ECS fields don’t need to be defined in your integration, they are dynamically integrated into the package at build time. Explicitly defined ECS fields can be used and will also overwrite this mechanism.
29
31
30
-
An example of the aformentioned `build.yml` file for this method:
32
+
An example of the aformentioned `build.yml` file for this method:
31
33
32
-
+
33
-
34
-
```yaml
35
-
dependencies:
36
-
ecs:
37
-
reference: git@v8.6.0
38
-
import_mappings: true
39
-
```
40
-
41
-
+ . Explicit ECS mappings As mentioned in the previous two approaches, ECS mappings can still be set explicitly and will overwrite the dynamic mappings. This can be done in two ways: - Using an `external: ecs` reference to import the definition of a specific field. - Literally defining the ECS field.
34
+
```yaml
35
+
dependencies:
36
+
ecs:
37
+
reference: git@v8.6.0
38
+
import_mappings: true
39
+
```
42
40
43
-
The `external: ecs` definition instructs the `elastic-package` command line tool to refer to an external ECS reference to resolve specific fields. By default it looks at the [ECS reference](https://raw.githubusercontent.com/elastic/ecs/v8.6.0/generated/ecs/ecs_nested.yml) file hosted on Github. This external reference file is determined by a Git reference found in the `_dev/build/build.yml` file, in the root of the package directory. The `build.yml` file set up for external references:
41
+
1. Explicit ECS mappings As mentioned in the previous two approaches, ECS mappings can still be set explicitly and will overwrite the dynamic mappings. This can be done in two ways: - Using an `external: ecs` reference to import the definition of a specific field. - Literally defining the ECS field.
44
42
45
-
+
43
+
The `external: ecs` definition instructs the `elastic-package` command line tool to refer to an external ECS reference to resolve specific fields. By default it looks at the [ECS reference](https://raw.githubusercontent.com/elastic/ecs/v8.6.0/generated/ecs/ecs_nested.yml) file hosted on Github. This external reference file is determined by a Git reference found in the `_dev/build/build.yml` file, in the root of the package directory. The `build.yml` file set up for external references:
46
44
47
-
```yaml
48
-
dependencies:
49
-
ecs:
50
-
reference: git@v8.6.0
51
-
```
45
+
```yaml
46
+
dependencies:
47
+
ecs:
48
+
reference: git@v8.6.0
49
+
```
52
50
53
-
Literal definition a ECS field:
51
+
Literal definition a ECS field:
54
52
55
-
```yaml
56
-
- name: cloud.acount.id
57
-
level: extended
58
-
type: keyword
59
-
ignore_above: 1024
60
-
description: 'The cloud account or organ....'
61
-
example: 43434343
62
-
```
53
+
```yaml
54
+
- name: cloud.acount.id
55
+
level: extended
56
+
type: keyword
57
+
ignore_above: 1024
58
+
description: 'The cloud account or organ....'
59
+
example: 43434343
60
+
```
63
61
64
62
1. Local ECS reference file (air-gapped setup) By changing the Git reference in in `_dev/build/build.yml` to the path of the downloaded [ECS reference](https://raw.githubusercontent.com/elastic/ecs/v8.6.0/generated/ecs/ecs_nested.yml) file, it is possible for the `elastic-package` command line tool to look for this file locally. Note that the path should be the full path to the reference file. Doing this, our `build.yml` file looks like:
65
63
@@ -72,9 +70,9 @@ Literal definition a ECS field:
72
70
73
71
The `access` data stream dataset of the Apache integration has four different field definitions:
74
72
75
-
+ NOTE: The `apache` integration below has not yet been updated to use the dynamic ECS field definition and uses `external` references to define ECS fields in `ecs.yml`.
76
-
77
-
+
73
+
:::{note}
74
+
The `apache` integration below has not yet been updated to use the dynamic ECS field definition and uses `external` references to define ECS fields in `ecs.yml`.
75
+
:::
78
76
79
77
```text
80
78
apache
@@ -112,8 +110,6 @@ In this file, the `data_stream` subfields `type`, `dataset` and `namespace` are
112
110
113
111
Here we define fields that we need in our integration and are not found in the ECS. The example below defines field `apache.access.ssl.protocol` in the Apache integration.
Copy file name to clipboardExpand all lines: docs/extend/pipeline-testing.md
+3-1Lines changed: 3 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -20,7 +20,9 @@ Conceptually, running a pipeline test involves the following steps:
20
20
21
21
## Limitations [pipeline-limitations]
22
22
23
-
At the moment, pipeline tests have limitations. The main ones are: * As you’re only testing the ingest pipeline, you can prepare mocked documents with imaginary fields, different from ones collected in {{beats}}. Also, the other way round, you can skip most of the example fields and use tiny documents with a minimal set of fields just to satisfy the pipeline validation. * There might be integrations that transform data mainly using {{beats}} processors instead of ingest pipelines. In such cases, ingest pipelines are rather plain.
23
+
At the moment, pipeline tests have limitations. The main ones are:
24
+
* As you’re only testing the ingest pipeline, you can prepare mocked documents with imaginary fields, different from ones collected in {{beats}}. Also, the other way round, you can skip most of the example fields and use tiny documents with a minimal set of fields just to satisfy the pipeline validation.
25
+
* There might be integrations that transform data mainly using {{beats}} processors instead of ingest pipelines. In such cases, ingest pipelines are rather plain.
24
26
25
27
26
28
## Defining a pipeline test [pipeline-defining-test]
Copy file name to clipboardExpand all lines: docs/extend/quick-start.md
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -136,7 +136,7 @@ You’ve now built an integration package, but it does not contain any assets. F
136
136
137
137
The command creates the required data in the `/data_stream/log` directory. If you pick `log` as data stream name, the dataset is called `sample.log` and the final data stream created will be `logs-sample.log-default` as an example.
138
138
139
-
3. To not have to worry about mappings, you can pull in all [Elastic Common Schema (ECS) fields][Elastic Common Schema (ECS)](ecs://reference/index.md)). To do this, create the file `_dev/build/build.yml` under the root directory and add the following content:
139
+
3. To not have to worry about mappings, you can pull in all [Elastic Common Schema (ECS) fields](ecs://reference/index.md). To do this, create the file `_dev/build/build.yml` under the root directory and add the following content:
Copy file name to clipboardExpand all lines: docs/extend/testing-validation.md
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -105,8 +105,8 @@ The CI job runner collects coverage data and stores them together with build art
105
105
106
106
As the Cobertura report format refers to packages, classes, methods, and such, unfortunately it doesn’t map easily onto the packages domain. We have decided to make a few assumptions for the Cobertura classification:
107
107
108
-
* **Package**: `integration``
109
-
***File**: `data stream``
108
+
* **Package**: `integration`
109
+
* **File**: `data stream`
110
110
* **Class**: test type (`pipeline tests`, `system tests`, etc.)
111
111
* **Method**: "OK" if there are any tests present.
0 commit comments