Skip to content
This repository was archived by the owner on Dec 14, 2022. It is now read-only.

Commit 28768e1

Browse files
author
Chris Wiechmann
committed
Added new parameter IGNORE_API_LOOKUP_ENABLED
1 parent c90640f commit 28768e1

File tree

7 files changed

+84
-43
lines changed

7 files changed

+84
-43
lines changed

CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
77
## [4.4.0] 2022-03-31
88
### Added
99
- Now the parameter: `CACHE_API_PATHS` is also used to cache the API-Ignore lookup more efficient
10+
- New parameter: `APPLICATION_LOOKUP_ENABLED` to disable application lookup for API-Gateway Traffic Monitoring improvement
11+
- New parameter: `IGNORE_API_LOOKUP_ENABLED` to disable the API-Ignore lookup, when not used anyway
1012

1113
## [4.4.0] 2022-03-29
1214
### Changed

UPDATE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ On the other hand, the API builder Docker image, as a central component of the s
3737

3838
| Ver | API-Builder | Logstash | Memcached | Filebeat | ANM-Config | Dashboards | Params |Elastic-Config | ELK-Ver. | Notes |
3939
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |:---: | :---: | :--- |
40+
| 4.5.0 | - | [X](#api-builderlogstashmemcached) | - | - | - | - | [X](#parameters)|- | [7.17.1](#update-elastic-stack-version) | |
4041
| 4.4.0 | [X](#api-builderlogstashmemcached) | [X](#api-builderlogstashmemcached) | - | - | - | - | [X](#parameters)|- | [7.17.1](#update-elastic-stack-version) | |
4142
| 4.3.0 | [X](#api-builderlogstashmemcached) | [X](#api-builderlogstashmemcached) | - | - | - | [X](#dashboards)| [X](#parameters)|[X](#elastic-config)| [7.17.1](#update-elastic-stack-version) | |
4243
| 4.2.0 | [X](#api-builderlogstashmemcached) | - | - | - | - | [X](#dashboards)| [X](#parameters)|- | [7.17.0](#update-elastic-stack-version) | |

docker-compose.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@ services:
3131
- xpack.geoip.download.endpoint=${GEOIP_DOWNLOAD_ENDPOINT}
3232
- EVENTLOG_CUSTOM_ATTR=${EVENTLOG_CUSTOM_ATTR}
3333
- CACHE_API_PATHS=${CACHE_API_PATHS}
34+
- APPLICATION_LOOKUP_ENABLED=${APPLICATION_LOOKUP_ENABLED:-true}
35+
- IGNORE_API_LOOKUP_ENABLED=${IGNORE_API_LOOKUP_ENABLED:-true}
3436
ports:
3537
- 5044:5044
3638
volumes:

env-sample

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -430,6 +430,30 @@ ELASTIC_VERSION=7.17.1
430430
# Defaults to all levels are forwarded.
431431
# DROP_TRACE_MESSAGE_LEVELS=DEBUG,DATA
432432

433+
# ----------------------------------------------------------------------------------------------
434+
# If an API is called by a registered API-Manager application and no user is authenticated, then
435+
# this application ID appears in the Subject column in the traffic monitor.
436+
# Now, by default, the solution tries to convert the unhelpful application IDs passed in the
437+
# authentication.subject.id attribute to the actual application name. This is done whenever the
438+
# authentication subject ID is a UUID.
439+
# However, if other UUIDs are used in your configuration in the authentication subject
440+
# (for example, customer IDs), then the lookup is actually always incorrect and an unnecessary
441+
# number of API requests are made. In this case, you should disable the function, since you have
442+
# no use for it anyway.
443+
# By the way, this has nothing to do with the application details, which are passed separately.
444+
# Used-By: Logstash
445+
# Defaults to true
446+
# APPLICATION_LOOKUP_ENABLED=false
447+
448+
# ----------------------------------------------------------------------------------------------
449+
# There is a possibility to exclude APIs from indexing in Elasticsearch, i.e. to ignore them.
450+
# If you do not want to make use of this, i.e. do not want to ignore APIs, then you can
451+
# completely deactivate the necessary Loookup, which is executed as part of the OpenTraffic
452+
# pipeline, in order to further reduce the ingest latency, for example.
453+
# Used-By: Logstash
454+
# Defaults to true
455+
# IGNORE_API_LOOKUP_ENABLED=false
456+
433457
# ----------------------------------------------------------------------------------------------
434458
# Disables the setup flows in API-Builder that are used to configure Elasticsearch.
435459
# If you run more than one API builder, you can set this parameter to true, as it is not

helm/templates/elasticApimLogstash/logstash-config.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ data:
3838
LOOKUP_CACHE_TTL: {{ default "" .Values.logstash.lookupCacheTTL | quote }}
3939
{{- end }}
4040
CACHE_API_PATHS: {{ default "" .Values.logstash.cacheAPIPaths | quote }}
41+
APPLICATION_LOOKUP_ENABLED: {{ default "true" .Values.logstash.applicationLookupEnabled | quote }}
42+
IGNORE_API_LOOKUP_ENABLED: {{ default "true" .Values.logstash.ignoreApiLookupEnabled | quote }}
4143
GEOIP_ENABLED: {{ default "true" .Values.logstash.geoip.enabled | quote }}
4244
GEOIP_CACHE_SIZE: {{ default "1000" .Values.logstash.geoip.cacheSize | quote }}
4345
GEOIP_CUSTOM_ATTRIBUTE: {{ default "true" .Values.logstash.geoip.customAttribute | quote }}

helm/values.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,12 @@ logstash:
274274
# Check the env-sample parameter: CACHE_API_PATHS for more details.
275275
# cacheAPIPaths: "/api/v2/petstore, /api/v1/user, ..."
276276

277+
# Disable application lookup. For more details please check parameter APPLICATION_LOOKUP_ENABLED
278+
# applicationLookupEnabled: "false"
279+
280+
# Disable the lookup if an API should be ignored. Makes sense if you dont ignore any API
281+
# ignoreApiLookupEnabled: "false"
282+
277283
# Injects the environment variables from the ConfigMaps and Secrets into the
278284
# Logstash container. Specify your own ConfigMaps or Secrets if you don't
279285
# provide Configuration and Secrets as part of this values.yaml.

logstash/pipelines/OpenTrafficPipeline.conf

Lines changed: 47 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ filter {
2525
add_field => { "[@metadata][ignoreField][policyName]" => "" } # and policyName used for ingoreLookup
2626
add_field => { "[@metadata][apiCacheKeyPrefix]" => "" } # Create a default for the cacheKeyPrefix
2727
add_field => { "[@metadata][cacheAPIPaths]" => "${CACHE_API_PATHS:''}" }
28+
add_field => { "[@metadata][applicationLookupEnabled]" => "${APPLICATION_LOOKUP_ENABLED:'true'}" }
29+
add_field => { "[@metadata][ignoreApiLookupEnabled]" => "${IGNORE_API_LOOKUP_ENABLED:'true'}" }
2830
}
2931

3032
# Check if the document should be ignored/dropped (configured based on the apiPath or PolicyName)
@@ -70,54 +72,56 @@ filter {
7072
'
7173
}
7274
}
73-
# Create a cache key for the event, either on API-Path or Policy-Name
74-
mutate {
75-
add_field => { "[@metadata][ignoreLookupCacheKey]" => "%{[@metadata][apiCacheKeyPrefix]}###%{[@metadata][ignoreField][policyName]}" }
76-
}
77-
# Lookup the cache is the document should be ignored
78-
memcached {
79-
hosts => "${MEMCACHED}"
80-
namespace => "ignoredAPIs"
81-
get => { "%{[@metadata][ignoreLookupCacheKey]}" => "[isIgnoreAPI]" }
82-
}
83-
# If we don't know yet, consult the API-Builder API
84-
if !([isIgnoreAPI]) {
85-
http {
86-
id => "Ignore API check"
87-
url => "${API_BUILDER_URL}/api/elk/v1/api/lookup/api/ignore"
88-
query => {
89-
"policyName" => "%{[@metadata][ignoreField][policyName]}"
90-
"apiPath" => "%{[@metadata][ignoreField][apiPath]}"
91-
"groupId" => "%{[processInfo][groupId]}"
92-
"region" => "%{[processInfo][gatewayRegion]}"
93-
"correlationId" => "%{[correlationId]}"
94-
}
95-
automatic_retries => 1 # 1 retry only, as this is not super important and should not block the entire processing
96-
cacert => "${API_BUILDER_SSL_CERT}"
97-
target_body => "isIgnoreAPI"
98-
add_field => { "[@metadata][updateIsIgnoreCache]" => "true" }
99-
}
100-
}
101-
# Drop the document if it should be ignored
102-
if ([isIgnoreAPI][ignore]) {
103-
ruby {
104-
code => 'logger.debug("Drop API/event as it should be ignored: ", "isIgnoreAPI" => event.get("[isIgnoreAPI]"), "ignoreLookupCacheKey" => event.get("[@metadata][ignoreLookupCacheKey]") );'
105-
}
106-
drop {}
107-
}
108-
# If the PolicyName or API-Path has been looked up right now add it to the cache
109-
if([@metadata][updateIsIgnoreCache]=="true") {
110-
ruby {
111-
code => 'logger.debug("Adding ignore status to cache: ", "isIgnoreAPI" => event.get("[isIgnoreAPI]"), "ignoreLookupCacheKey" => event.get("[@metadata][ignoreLookupCacheKey]") );'
75+
if([@metadata][ignoreApiLookupEnabled]) {
76+
# Create a cache key for the event, either on API-Path or Policy-Name
77+
mutate {
78+
add_field => { "[@metadata][ignoreLookupCacheKey]" => "%{[@metadata][apiCacheKeyPrefix]}###%{[@metadata][ignoreField][policyName]}" }
11279
}
80+
# Lookup the cache is the document should be ignored
11381
memcached {
11482
hosts => "${MEMCACHED}"
11583
namespace => "ignoredAPIs"
116-
ttl => "${LOOKUP_CACHE_TTL:600}"
117-
set => { "[isIgnoreAPI]" => "%{[@metadata][ignoreLookupCacheKey]}" }
84+
get => { "%{[@metadata][ignoreLookupCacheKey]}" => "[isIgnoreAPI]" }
85+
}
86+
# If we don't know yet, consult the API-Builder API
87+
if !([isIgnoreAPI]) {
88+
http {
89+
id => "Ignore API check"
90+
url => "${API_BUILDER_URL}/api/elk/v1/api/lookup/api/ignore"
91+
query => {
92+
"policyName" => "%{[@metadata][ignoreField][policyName]}"
93+
"apiPath" => "%{[@metadata][ignoreField][apiPath]}"
94+
"groupId" => "%{[processInfo][groupId]}"
95+
"region" => "%{[processInfo][gatewayRegion]}"
96+
"correlationId" => "%{[correlationId]}"
97+
}
98+
automatic_retries => 1 # 1 retry only, as this is not super important and should not block the entire processing
99+
cacert => "${API_BUILDER_SSL_CERT}"
100+
target_body => "isIgnoreAPI"
101+
add_field => { "[@metadata][updateIsIgnoreCache]" => "true" }
102+
}
103+
}
104+
# Drop the document if it should be ignored
105+
if ([isIgnoreAPI][ignore]) {
106+
ruby {
107+
code => 'logger.debug("Drop API/event as it should be ignored: ", "isIgnoreAPI" => event.get("[isIgnoreAPI]"), "ignoreLookupCacheKey" => event.get("[@metadata][ignoreLookupCacheKey]") );'
108+
}
109+
drop {}
110+
}
111+
# If the PolicyName or API-Path has been looked up right now add it to the cache
112+
if([@metadata][updateIsIgnoreCache]=="true") {
113+
ruby {
114+
code => 'logger.info("Adding ignore status to cache: ", "ignoreLookupCacheKey" => event.get("[@metadata][ignoreLookupCacheKey]"), "isIgnoreAPI" => event.get("[isIgnoreAPI]") );'
115+
}
116+
memcached {
117+
hosts => "${MEMCACHED}"
118+
namespace => "ignoredAPIs"
119+
ttl => "${LOOKUP_CACHE_TTL:600}"
120+
set => { "[isIgnoreAPI]" => "%{[@metadata][ignoreLookupCacheKey]}" }
121+
}
118122
}
123+
mutate { remove_field => "[isIgnoreAPI]" }
119124
}
120-
mutate { remove_field => "[isIgnoreAPI]" }
121125

122126
# Check, if event is a Scheduled-Policy - Most of the properties are NULL
123127
if([correlationId]!="000000000000000000000000" and ![circuitPath] and ![transactionElement] and ![transactionSummary][path] and ![transactionSummary][protocol] and ![transactionSummary][protocolSrc]) {
@@ -348,7 +352,7 @@ filter {
348352
}
349353
}
350354
# Try to translate the applicationId (e.g. 180b1f32-d72f-40f4-949a-fc3f3f7dec2c) into a meaningful application-name
351-
if ([http]) { # Translation only supported for HTTP-Requests
355+
if ([http] and [@metadata][applicationLookupEnabled]) { # Translation only supported for HTTP-Requests
352356
grok {
353357
match => { "[http][authSubjectId]" => "^.{8}-.{4}-.{4}-.{4}-.{12}$" }
354358
tag_on_failure => ["_authNSubjectNoUUID"]

0 commit comments

Comments
 (0)