[BUG]: Improve movement of frame data to/from Python

[miguelaeh]

Moving the frames into Python and getting them back is adding a few milliseconds of latency that we can optimize.More specifically, it adds between 1 and 3 ms on each operation.