Store smaller objects in mongo vs minio #717
Description
Now that the workspace stores objects in minio vs shock and the object storage location is definitely invisible to users, we could update the workspace to store smaller objects in mongo, which would reduce the small files problem in minio and make fetching the objects faster as they can be batched in mongo and a separate minio call isn't needed.
Possibilities:
- Just store as a normal doc -
$and.in keys will need to be looked into. Also means will need an extra JSON conversion vs storing in JSON. - Store in JSON in GFS.
Drawbacks - if stored in Minio, S3 has json parsing built in which could be used for pulling sub objects. If the files are relatively small this probably doesn't really matter. Also if the files are stored as normal docs there's libraries for doing this.
Once the workspace is updated, we'd want to run a script to move small objects from minio -> mongo