Skip to content

NETOBSERV-2307: NETOBSERV-2315: fix several IPFIX issues #1019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jul 9, 2025
Merged

NETOBSERV-2307: NETOBSERV-2315: fix several IPFIX issues #1019

merged 6 commits into from
Jul 9, 2025

Conversation

jotak
Copy link
Member

@jotak jotak commented Jul 3, 2025 •
edited
Loading

Description

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
    • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
    • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
    • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
    • Standard QE validation, with pre-merge tests unless stated otherwise.
    • Regression tests only (e.g. refactoring with no user-facing change).
    • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

To run a perfscale test, comment with: /test flp-node-density-heavy-25nodes

@jotak
NETOBSERV-2307: NETOBSERV-2315: fix several IPFIX issues
b044453 
- Fixed flows without ports that generated errors in logs, and were not exported. It
  shouldn't matter that ports are missing (e.g. ICMP)
- More generally, any missing field won't trigger an error anymore
- Some fields were missing: icmp type/code, tcp flags
- Fix resending templates in case of collector being restarted
@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Jul 3, 2025 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-2307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Description

  • Fixed flows without ports that generated errors in logs, and were not exported. It shouldn't matter that ports are missing (e.g. ICMP)
  • More generally, any missing field won't trigger an error anymore
  • Some fields were missing: icmp type/code, tcp flags
  • Fix resending templates in case of collector being restarted

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

To run a perfscale test, comment with: /test flp-node-density-heavy-25nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Jul 3, 2025 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-2307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Description

  • Fixed flows without ports that generated errors in logs, and were not exported. It shouldn't matter that ports are missing (e.g. ICMP)
  • More generally, any missing field won't trigger an error anymore
  • Some fields were missing: icmp type/code, tcp flags
  • Fix resending templates in case of collector being restarted (See also: https://github.com/orgs/netobserv/discussions/1649)

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

To run a perfscale test, comment with: /test flp-node-density-heavy-25nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jotak jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 3, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

New image:
quay.io/netobserv/flowlogs-pipeline:9269049

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=9269049 make set-flp-image

jotak added 2 commits July 4, 2025 12:41
@jotak
fix sending decoded tcpflags on ipfix
d56febe
@jotak
Use patched go-ipfix, add write+read test...
743583a 
The go-ipfix patch regularly recheck the udp connection, e.g. to account
for restarted host with a different resolved IP

... also:
- allow to provide data mapping for ingesting with goflow2
- rename API collector => ipfix
- simplify ingest API with more default values
- write ipfix: do not resend templates every second! Instead, use
  configurable periodicity
@github-actions github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 4, 2025
@jotak jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 4, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 4, 2025

New image:
quay.io/netobserv/flowlogs-pipeline:f992bcc

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=f992bcc make set-flp-image

@github-actions github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 4, 2025
@jotak jotak added the needs-review Tells that the PR needs a review label Jul 7, 2025
jpinsonneau
jpinsonneau reviewed Jul 7, 2025
View reviewed changes
pkg/api/api.go
@@ -53,7 +53,7 @@ type API struct {
PromEncode PromEncode `yaml:"prom" doc:"## Prometheus encode API\nFollowing is the supported API format for prometheus encode:\n"`
KafkaEncode EncodeKafka `yaml:"kafka" doc:"## Kafka encode API\nFollowing is the supported API format for kafka encode:\n"`
S3Encode EncodeS3 `yaml:"s3" doc:"## S3 encode API\nFollowing is the supported API format for S3 encode:\n"`
IngestCollector IngestCollector `yaml:"collector" doc:"## Ingest collector API\nFollowing is the supported API format for the NetFlow / IPFIX collector:\n"`
IngestIpfix IngestIpfix `yaml:"ipfix" doc:"## Ingest NetFlow/IPFIX API\nFollowing is the supported API format for the NetFlow / IPFIX collector:\n"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be carefull since this is still used in multiple places such in:
https://github.com/netobserv/netobserv-ebpf-agent/blob/ba5c7c6ffaea7dfaaf61aeb3111bdd14a86c871d/e2e/ipfix/manifests/20-flp-transformer.yml#L53

flowlogs-pipeline/network_definitions/config.yaml

Lines 7 to 10 in 19fb27b

collector:
port: 2055
portLegacy: 2056
hostName: 0.0.0.0

flowlogs-pipeline/contrib/kubernetes/flowlogs-pipeline.conf.yaml

Lines 24 to 27 in 19fb27b

collector:
hostName: 0.0.0.0
port: 2055
portLegacy: 2056

flowlogs-pipeline/hack/examples/docker-ipfix-config.yaml

Lines 23 to 26 in 19fb27b

collector:
hostName: localhost
port: 4739 # Use this for IPFIX / netflow v9
portLegacy: 2055 # Use this for legacy v5 netflow

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done (except the first which is a different thing, it's the conf-generator tool, which I didn't touch here)
Also added a warn log for deprecated API usage

@jotak jotak requested a review from jpinsonneau July 8, 2025 06:04
@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Jul 8, 2025 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-2307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Description

  • Fixed flows without ports that generated errors in logs, and were not exported. It shouldn't matter that ports are missing (e.g. ICMP)
  • More generally, any missing field won't trigger an error anymore
  • Some fields were missing: icmp type/code, tcp flags
  • Fix recreating a connection to the collector in case of collector being restarted (See also: https://github.com/orgs/netobserv/discussions/1649)

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

To run a perfscale test, comment with: /test flp-node-density-heavy-25nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Jul 8, 2025 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-2307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Description

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

To run a perfscale test, comment with: /test flp-node-density-heavy-25nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the lgtm label Jul 8, 2025
@jotak jotak removed the needs-review Tells that the PR needs a review label Jul 8, 2025
@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Jul 8, 2025 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-2307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Description

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

To run a perfscale test, comment with: /test flp-node-density-heavy-25nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot removed the lgtm label Jul 8, 2025
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Jul 8, 2025

New changes are detected. LGTM label has been removed.

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Jul 8, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jpinsonneau. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Jul 8, 2025

@jotak: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/qe-e2e-tests 0d88640 link false /test qe-e2e-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@memodi
Copy link
Member

memodi commented Jul 8, 2025 •
edited
Loading

even though qe-e2e-tests failed, I see the IPFIX exporter test is passing:

started: (2/21/27) "[sig-netobserv] Network_Observability Author:aramesha-High-64156-Verify IPFIX-exporter [Serial]"
passed: (2m18s) 2025-07-08T15:45:23 "[sig-netobserv] Network_Observability Author:aramesha-High-64156-Verify IPFIX-exporter [Serial]"

Need to investigate here why the run failed here though, unsure if it was infra issue or the test issue.

/label qe-approved

@jotak let me know if full scale regression would be relevant here.

@openshift-ci openshift-ci bot added the qe-approved QE has approved this pull request label Jul 8, 2025
@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Jul 8, 2025 •
edited by openshift-ci bot
Loading

@jotak: This pull request references NETOBSERV-2307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Description

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
  • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
  • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
  • Standard QE validation, with pre-merge tests unless stated otherwise.
  • Regression tests only (e.g. refactoring with no user-facing change).
  • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

To run a perfscale test, comment with: /test flp-node-density-heavy-25nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jotak jotak merged commit 5895dcb into netobserv:main Jul 9, 2025
7 of 9 checks passed
@jotak
Copy link
Member Author

jotak commented Jul 9, 2025 •
edited
Loading

@memodi thanks! sounds good enough to me

On the error to investigate, I see "curl: command not found" which apparently comes from there: https://github.com/openshift/openshift-tests-private/blob/master/test/extended/netobserv/test_exporters.go#L235 cc @Amoghrd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@OlivierCazade OlivierCazade OlivierCazade approved these changes

@jpinsonneau jpinsonneau jpinsonneau approved these changes

Assignees

@OlivierCazade OlivierCazade

@jpinsonneau jpinsonneau

Labels
jira/valid-reference qe-approved QE has approved this pull request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jotak @openshift-ci-robot @memodi @OlivierCazade @jpinsonneau