eBPF Agent - general questions

[johnhtodd]

Sorry for n00b questions here, but I've just found netobserv and it's really interesting for some of our use cases. We're using Vector for some of our other telemetry flows, so this has a similar shape but I'm still not clear on some syntax issues. We'll probably aggregate/average

1. I'm trying to get some minimal configs running with very simple filters just for testing. I've got my "flp-config.json" as decribed on the main page, and I've tried setting my environment variables as described in this example (but with different IP data) and also setting "ENABLE_FLOW_FILTER" to "true". I get:

"FATA[0000] can't instantiate NetObserv eBPF Agent        error="unexpected end of JSON input"

Any hints on what I'm doing incorrectly here? If I turn "ENABLE_FLOW_FILTER" to "false" then no error is produced, but no filter seems to be applied, either. What JSON input is this error referencing?

2. We are really interested in just the events with TCP round-trip data. I assume I can just strip that out in the flowlogs config via the "remove_entry_if_doesnt_exist" action looking for TimeFlowRttNs?

3. I'd like to trigger on traffic that is received on two non-sequential TCP ports. It doesn't appear that is possible, since FLOW_FILTER_DESTINATION_PORT seems to only be able to be defined once - do I need to run two instances of the Agent to get two ports? That is unexpected. Perhaps I am not understanding some really basic idea.

[jotak]

Hi @johnhtodd
oops, looks like the doc hasn't been updated - I'll work on that asap
The single-rule env was removed and replace with a multi-rules config in json
Here's what you can do:

Create a filters.json file such as:

[
	{"ip_cidr":"0.0.0.0/0", "action": "Accept"}
]

You can use several rules there. Until I update the doc, you can look here for the available config; code won't lie :-)
You can also read this blog post about filtering with the agent: https://netobserv.io/posts/enhancing-netobserv-by-introducing-multi-rules-flow-filtering-capability-in-ebpf/ , but note that the configs listed in the blog are for the netobserv kubernetes operator, it differs in structure from the agent config but that's a 1-1 mapping so that can still be useful to you, just convert the field names with what's in my first link.

Then, simply export that:

export FLOW_FILTER_RULES=$(cat filters.json)

and run the agent, hopefully it should work

[jotak]

2. We are really interested in just the events with TCP round-trip data. I assume I can just strip that out in the flowlogs config via the "remove_entry_if_doesnt_exist" action looking for TimeFlowRttNs?

yes, you can do that. There's probably more optimization you can do, to filter at the earliest possible stage.. such as looking only for TCP traffic, to start with.

3. I'd like to trigger on traffic that is received on two non-sequential TCP ports.  It doesn't appear that is possible, since FLOW_FILTER_DESTINATION_PORT seems to only be able to be defined once - do I need to run two instances of the Agent to get two ports?  That is unexpected. Perhaps I am not understanding some really basic idea.

You can provide several ports, see https://github.com/netobserv/netobserv-ebpf-agent/blob/main/pkg/config/config.go#L52-L54
Does that answer the question?

[johnhtodd]

3. I'd like to trigger on traffic that is received on two non-sequential TCP ports.  It doesn't appear that is possible, since FLOW_FILTER_DESTINATION_PORT seems to only be able to be defined once - do I need to run two instances of the Agent to get two ports?  That is unexpected. Perhaps I am not understanding some really basic idea.

You can provide several ports, see https://github.com/netobserv/netobserv-ebpf-agent/blob/main/pkg/config/config.go#L52-L54 Does that answer the question?

After a bit of head-scratching and re-reading the patch wording closely, I see that the "ports" specifier only accepts two comma-separated ports as src or dst. This seems a bit odd - why just two? I tried adding a third, and it then ignores that directive entirely as if the "ports" line was not in the file.

This works:

[
{
    "ip_cidr": "0.0.0.0/0",
    "action": "Accept",
    "protocol": "TCP",
    "ports": "443,853"
}
]

but this does not:

[
{
    "ip_cidr": "0.0.0.0/0",
    "action": "Accept",
    "protocol": "TCP",
    "ports": "53,443,853"
}
]

[jotak]

I know that's a bit odd, we can't use an unbound/arbitrary size for the list of ports, it has to be a fixed number. Currently it's just 2 because we haven't had the need for more, but we could increase it.

[johnhtodd]

OK, thanks for reply. Is it possible to add other rules in the same "flow" that examine the data set or do I have to launch two instances of the agent to monitor four ports? My particular use case is looking at DNS, and we are on 53 (TCP), 853, and 443. We also operate on other alternate ports but they get far less traffic (https://docs.quad9.net/services/) and would probably not be super-illuminating to track (except for edge cases like "What people in countries are most likely to use non-common ports to access the DNS, and what is the typical latency they are willing to put up with?") Ideally with the set of ports on which we operate production traffic, the number is "6", so in the current model would that mean we would have to run 3 instances of the agent?

[jotak]

yeah that's correct, you can run several agents, they won't conflict.
Also fyi since you're interested in DNS, maybe you've seen already that you can enable a DNS feature (ENABLE_DNS_TRACKING) which basically will tell you the response code, the dns id (so you can correlate requests and responses), and will compute the latency, for traffic on port 53, or (and this is undocumented because currently is more like a hack) an additional port that you can specify via DNS_TRACKING_PORT env. We will surely refactor this part at some point to offer more flexibility, so don't hesitate to give us feedback!

[johnhtodd]

1. OK, I'm running with just more items in the filter array and that seems to work. Can I just extend this model with more objects in the array that specify more subnets/ports? It seems the answer is "Yes" so I don't have to run multiple agents (unless this is in fact running multiple agents in the background as a result of creating the filter list like this, but I don't see them in the process table.)

[
{
    "ip_cidr": "0.0.0.0/0",
    "action": "Accept",
    "direction": "Ingress",
    "protocol": "TCP",
    "ports": "443,853"
},
{
    "ip_cidr": "::0/0",
    "action": "Accept",
    "direction": "Ingress",
    "protocol": "TCP",
    "ports": "443,853"
},
{
    "ip_cidr": "::0/0",
    "action": "Accept",
    "direction": "Ingress",
    "protocol": "TCP",
    "ports": "53"
},
{
    "ip_cidr": "0.0.0.0/0",
    "action": "Accept",
    "direction": "Ingress",
    "protocol": "TCP",
    "ports": "53"
}
]

2. The DNS tracking stuff is interesting, but it's not really applicable right now for us because we collect that information from dnstap data (relayed to Vector) coming from the recursive resolvers that we operate. We get latency at the application layer for all DNS protocols, not just port 53 and we needed a "universal" solution for measuring outbound traffic. In any case, I'd rather trust the application in this instance than the layer 3 data since the application data will probably be much more pessimistic. You'll note that I'm tracking 853 and 443 (and later, 53) above, which is DOT and DOH respectively since what I'm interested in is latency to clients, not to auth servers. In short: we measure DNS latency for how we need to via a different way already.

3. Unrelated question: I note that there doesn't seem to be an "interval" at which I can specify OTLP data being sent out - am I missing something? Here's why I ask: My current pipeline that I've created is netobserv-agent(resolvers) ->[grpc]-> flowlogs-pipeline(telemetry system in POP) ->[grpc/OTLP]-> Vector ->[prometheus scrapes,kafka] -> Prometheus,Clickhouse. My concern is in the step between flowlogs-pipeline and the output to the OTLP write stage, or indeed any write stage. It seems that there is a 2 or 5 second window between full state transmission there.

I have OTLP and stdout writes happening on the same paths, so I can see what's happening and looking at the network dumps and stdout they seem to match, so I'll assume for convenience that is the case. I have an aggregate that produces these lines (spacing added by me to show 'clusters' of events):

netobserv# flowlogs-pipeline --config flowlog-config.yaml |grep 192.168.132.179
Jun  4 18:27:42.234: map[SrcAddr:192.168.132.179 name:avg operation_key:TimeFlowRttNs operation_type:avg total_count:33 total_value:1.4786787878787879e+07]
Jun  4 18:27:42.236: map[SrcAddr:192.168.132.179 name:min operation_key:TimeFlowRttNs operation_type:min total_count:33 total_value:7.768e+06]
Jun  4 18:27:42.239: map[SrcAddr:192.168.132.179 name:max operation_key:TimeFlowRttNs operation_type:max total_count:33 total_value:3.7607e+07]

Jun  4 18:27:47.236: map[SrcAddr:192.168.132.179 name:avg operation_key:TimeFlowRttNs operation_type:avg total_count:37 total_value:1.4250567567567568e+07]
Jun  4 18:27:47.240: map[SrcAddr:192.168.132.179 name:min operation_key:TimeFlowRttNs operation_type:min total_count:37 total_value:7.768e+06]
Jun  4 18:27:47.242: map[SrcAddr:192.168.132.179 name:max operation_key:TimeFlowRttNs operation_type:max total_count:37 total_value:3.7607e+07]

Jun  4 18:27:51.343: map[SrcAddr:192.168.132.179 name:min operation_key:TimeFlowRttNs operation_type:min total_count:41 total_value:7.768e+06]
Jun  4 18:27:51.346: map[SrcAddr:192.168.132.179 name:max operation_key:TimeFlowRttNs operation_type:max total_count:41 total_value:3.7607e+07]
Jun  4 18:27:51.352: map[SrcAddr:192.168.132.179 name:avg operation_key:TimeFlowRttNs operation_type:avg total_count:41 total_value:1.3710487804878049e+07]
Jun  4 18:27:52.225: map[SrcAddr:192.168.132.179 name:max operation_key:TimeFlowRttNs operation_type:max total_count:41 total_value:3.7607e+07]
Jun  4 18:27:52.226: map[SrcAddr:192.168.132.179 name:min operation_key:TimeFlowRttNs operation_type:min total_count:41 total_value:7.768e+06]
Jun  4 18:27:52.235: map[SrcAddr:192.168.132.179 name:avg operation_key:TimeFlowRttNs operation_type:avg total_count:41 total_value:1.3710487804878049e+07]

Jun  4 18:27:56.341: map[SrcAddr:192.168.132.179 name:min operation_key:TimeFlowRttNs operation_type:min total_count:45 total_value:7.768e+06]
Jun  4 18:27:56.347: map[SrcAddr:192.168.132.179 name:avg operation_key:TimeFlowRttNs operation_type:avg total_count:45 total_value:1.3785355555555556e+07]
Jun  4 18:27:56.360: map[SrcAddr:192.168.132.179 name:max operation_key:TimeFlowRttNs operation_type:max total_count:45 total_value:3.7607e+07]
Jun  4 18:27:57.234: map[SrcAddr:192.168.132.179 name:avg operation_key:TimeFlowRttNs operation_type:avg total_count:45 total_value:1.3785355555555556e+07]
Jun  4 18:27:57.235: map[SrcAddr:192.168.132.179 name:max operation_key:TimeFlowRttNs operation_type:max total_count:45 total_value:3.7607e+07]
Jun  4 18:27:57.246: map[SrcAddr:192.168.132.179 name:min operation_key:TimeFlowRttNs operation_type:min total_count:45 total_value:7.768e+06]

Looking at this data, some of those lines seem to (confusingly) be produced twice, roughly 2 seconds apart, with the same data (always the same data- same total count, etc.) But sometimes it's 5 seconds. I think I have found a bug? But I have no idea.

Back to my original discussion: My reason for looking at this was to increase the interval that the data is sent on the OTLP writes. I need that interval between full state dumps to be much much larger than every two or five seconds. I will have hundreds of thousands (or millions?) of individual IP addresses which will be tracked in this fashion in any one location - the example above just shows a single address that I'm tracking. To handle that volume of records by downstream processors in a regularly pulsed fashion, I might need to increase the interval of full state transmission to every minute, or two minutes, or even 3 minutes. It would be even MORE ideal to have each entry have its own timer of when it gets sent, rather than ALL items in the list being sent all at once - this is currently a flood risk for downstream components and I'd much rather have individual elements keep their own timers rather than a single, universal timer that dumps everything to the downstream components all at once.

Am I missing something here where I can already do this with the config options in flowlogs-pipelinie? Or do I need to start looking at the code to submit a PR?

[jotak]

Doc update PR: netobserv/netobserv-ebpf-agent#710

[johnhtodd]

Thanks for the hints - digging in and trying to make this work now.

[johnhtodd]

Thanks @jotak - got this to work for my next increment, but now on to the next step, which again may be my thickheadedness or a doc problem. I am now trying to make an aggregate (average) using the subnet of each entry. I have achieved a step where I have slow-stepped my way through things and have objects passing through the system that look like this:

May 30 19:28:33.695: map[DstAddr:10.10.3.3 DstPort:853 SrcAddr:192.168.58.30 SrcPort:11944 SrcSubnet:192.168.58.0/24 TimeFlowRttNs:6240000]

So far, so good. Now, I try feeding it into an aggregate, using the example provided as my template. Here is my conversion:

  - name: SubnetRTTAverage
    extract:
      type: aggregates
      aggregates:
        - name: "Average by round trip time per subnet"
          by:
            - "SrcSubnet"
          operation: "avg"
          operationKey: "TimeFlowRttNs"

But I get this error:

FATA[0000] can't instantiate NetObserv eBPF Agent        error="failed to read config: yaml: unmarshal errors:\n  line 94: cannot unmarshal !!seq into api.Aggregates"

I have tried many different permutations of the lines, but I'll admit I'm cargo-culting this a bit with the YAML. I have the "pipeline:" statements correct; line 94 is the " - name: "Average by round trip time per subnet"" line. I've minimized this example all the way down to just a config file with the example text for "Aggregates" in it and running directly as a config file ("flowlogs-pipeline --config minimal.yaml") and it still fails with the unmarshal error. $ crapbot has also been entirely stumped as to the syntax issues I'm running into.

Is this me, or the documentation, or a bug?

A follow-up question: are aggregates available inside of netobserv-ebpf-agent or are they only available if I pipe the output to flowlogs-pipeline in some fashion?

[jotak]

Hmm where did you get the documentation? I don't see where that could come from.
See here: https://github.com/netobserv/flowlogs-pipeline/blob/main/docs/api.md#aggregate-metrics-api
There's a missing rules field in your config below aggregates, by should be groupByKeys and operation should be operationType.

[jotak]

A follow-up question: are aggregates available inside of netobserv-ebpf-agent or are they only available if I pipe the output to flowlogs-pipeline in some fashion?

If you use the direct-flp export mode as explained here, you don't need to run FLP separately, in short, in this mode the agent embeds FLP and uses it directly within the same process.

[jotak]

btw I have no idea what is "crapbot" 🫣 a friendly nickname for chatgpt? :D

[johnhtodd]

Thanks - I think I've grokked most of this particular sub-thread and have things working as I need to. Yes, "crapbot" is <$LLM-of-your-choice> - sorry for the localized slang. :-)