Downsampling in sRGB isn't correct, so there should be an option to do gamma correction automatically.
True sRGB <-> linear is quite expensive, and gamma 2.3 isn't much cheaper. Maybe we could get away with using gamma 2? Then sRGB -> linear is just squaring. This is huge, because the ration of sRGB->linear to linear->sRGB operations is 3:1. So sRGB->linear better be cheap.