[META] Testing strategy for auxiliary transports

[finnegancarroll]

OpenSearch allows for pluggable client server transports to be implemented by plugins and exposed to clients. As these transports mature and become adopted by users they must integrate functionality provided by the security plugin such as encryption in transit, authentication, and authorization. For these features we require a robust and maintainable testing framework which comes with some complications.

Auxiliary transports are implemented as OpenSearch plugins. Due to this, security plugin is agnostic as to their implementation and existence. To the security plugin, auxiliary transports are represented only by a "type" identifier ("grpc-transport", "arrow-flight-transport", ect) as well as a port range. Requiring the security plugin to integration test each new auxiliary transport breaks this framework and is potentially unsustainable as new transports are introduced in third party plugins.

Some options to consider:

• Implement aux transport ITs in opensearch-build smoke tests which are executed daily on the bundled distribution.
• Implement aux transport ITs in security plugin with security plugin taking aux transports as test only dependencies.
• Require aux transports be third party plugins such that they can take a dependency on the security plugin and implement their own ITs.
• Negotiate some split where security plugin bears the responsibility of ITs for native plugin and module auxiliary transports, but third party plugin aux transports must take a dependency on security plugin and implement ITs themselves.

[finnegancarroll]

Previous discussion on this topic: #5375 (comment)
cc @rishabhmaurya @andrross @karenyrx @reta @cwperks

[rishabhmaurya]

Is it possible that aux transport take a dependency (on snapshot version)and install security plugin as part of their integ test setup? Just like we can install any other plugin available in the core?

[finnegancarroll]

Is it possible that aux transport take a dependency (on snapshot version)and install security plugin as part of their integ test setup? Just like we can install any other plugin available in the core?

How frequently do we publish snapshots from main? In this case if I go to fix a bug in security plugin gRPC logic I would:
-> Raise a bugfix PR in security plugin
-> Raise a PR in core to cover this case
-> Run core gRPC ITs with my security plugin feature branch and provide proof the regression test is passing?
-> Once security plugin fix is merged, wait until a new snapshot is published to merge new test in core.

[cwperks]

[Triage] IMO if this is a core module than the security plugin is an appropriate repo for the tests, but the aux transport has a separate plugin repo then any tests with security should live in that repo.