Scalability issues

[snosratiershad]

As channels are stored in engine struct (in memory) bean can not to be deployed with multiple replicas causing issues with ha and scalability of platform.
I think a pub/sub based solution to handle send or receive of chunks will help, there will be two channels receiver channel and sender channel every replica will receive all chunks and will beam them if owns the receiver's channel, and every replica received a chunk from sender channel will also publish it to the topic/etc.
Also I'm thinking to a shard based p2p messaging, which channels are sharded and messages will be routed based on a shard key to the replica that has the receiver channel.
What do you think about them?

[ksdme]

Are you planning on deploying beam at scale?

I think before we jump into implementing such a solution, it would be more productive to validate the need for it. We could start by running a load test on a beam instance. And, if possible, tune beam to be a bit more resource friendly first.

[snosratiershad]

@ksdme , the load test plan sounds good, but I think we'll have to do scalability things at the end of the day.
However, I'll try to send a patch of load testing capabilities.