-
Notifications
You must be signed in to change notification settings - Fork 86
Commit 8f665cf
fix for list_objects
* for postgres queries, there is no need to make an extra DB query,
after processing the results of the first query.
* in Mongo map-reduce, we could get to a situation where the query
returned less than the limit, even though there are still relevant
entries.
* in Postgres, this is not happening, since the map_common_prefixes,
iterates until it finds enough records (or until it reaches the end).
* Performing the extra DB query in postgres, can cause a very long query
that eventually returns 0 entries. One scenario is where we have a
directory that is the last one returned in the previous query, and it
has a large number of objects under it (.e.g /folder/file0 ..
/folder/file9999999). when returning for the extra query, the marker
that is used is 'folder/', so map_common_prefixes starts iterating from
'folder/', looking for all objects that do not start with 'folder/', but
ends with a '/' (the delimiters). for postgres there is no smart way to
do it, so it just going over all objects under 'folder/' and filetering
them out. this can take over several minutes to complete.
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
(cherry picked from commit 219699d)1 parent a417731 commit 8f665cfCopy full SHA for 8f665cf
File tree
Expand file treeCollapse file tree
0 file changed
+0
-0
lines changedFilter options
Expand file treeCollapse file tree
0 file changed
+0
-0
lines changed
0 commit comments