move context from record to pipeline level

[divyanshu-tiwari]

This pull request introduces a significant refactor to how metadata is handled in pipeline records, replacing the previous usage of Go's context.Context with a custom Meta map. It also updates the pipeline and task interfaces to use context.Context for cancellation and control, improving robustness and clarity. The changes touch core pipeline logic, task implementations, and configuration handling.

Metadata handling refactor:

• Replaced the Context field in the Record struct with a Meta map for storing metadata, ensuring backward compatibility by keeping the JSON key as context.
• Removed SetContextValue and GetContextValue methods in favor of new SetMetaValue and GetMetaValue methods for managing record metadata. [1] [2]
• Updated all task implementations (archive, compress, converter, delay, echo, file, flatten) to use Meta for metadata storage and retrieval instead of Context. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

Pipeline and task interface improvements:

• Added a pipeline-level context.Context and cancel function for supporting cancellation and control flow, and updated task concurrency handling to pass context to tasks. [1] [2] [3] [4]
• Refactored all task Run methods to accept a context.Context parameter, enabling cancellation and consistent interface usage. [1] [2] [3] [4] [5] [6] [7] [8]

Configuration and macro/context handling:

• Updated configuration macro/context handling to use new meta methods, including renaming placeholder and evaluation functions and updating template function registration for backward compatibility. [1] [2] [3] [4]

These changes modernize metadata handling, improve cancellation support, and ensure backward compatibility for configuration templates.- storing metadata in context at record level was creating chain of references preventing garbage collection of records, which was leading to OOM issues.

Description

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation and I have updated the documentation accordingly.
• I have added tests to cover my changes.

[Copilot]

Pull request overview

This pull request refactors metadata handling in pipeline records by replacing Go's context.Context with a custom Meta map, while introducing pipeline-level context for cancellation support. The change addresses memory leaks caused by context reference chains preventing garbage collection of records.

Changes:

• Replaced Record.Context field with Record.Meta map for metadata storage, maintaining JSON backward compatibility
• Updated all task Run methods to accept context.Context parameter for cancellation support
• Modified configuration handling to use new meta methods while preserving template function names for backward compatibility

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
internal/pkg/pipeline/record/record.go	Replaced Context field with Meta map in Record struct
internal/pkg/pipeline/record/meta.go	Added new SetMetaValue/GetMetaValue methods for metadata management
internal/pkg/pipeline/record/context.go	Removed old context-based metadata methods
internal/pkg/pipeline/pipeline.go	Added pipeline-level context with cancellation support
internal/pkg/pipeline/task/task.go	Updated Task interface and base methods to use context parameter and Meta map
internal/pkg/pipeline/task/*/*.go	Updated all task implementations to use new Run signature and Meta instead of Context
internal/pkg/config/context.go	Renamed functions to use Meta terminology while maintaining backward compatibility
internal/pkg/config/config.go	Updated template function registration to use renamed meta functions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.