Skip to content

Commit 8f665cf

Browse files
dannyzakenliranmauda
authored andcommitted
fix for list_objects
* for postgres queries, there is no need to make an extra DB query, after processing the results of the first query. * in Mongo map-reduce, we could get to a situation where the query returned less than the limit, even though there are still relevant entries. * in Postgres, this is not happening, since the map_common_prefixes, iterates until it finds enough records (or until it reaches the end). * Performing the extra DB query in postgres, can cause a very long query that eventually returns 0 entries. One scenario is where we have a directory that is the last one returned in the previous query, and it has a large number of objects under it (.e.g /folder/file0 .. /folder/file9999999). when returning for the extra query, the marker that is used is 'folder/', so map_common_prefixes starts iterating from 'folder/', looking for all objects that do not start with 'folder/', but ends with a '/' (the delimiters). for postgres there is no smart way to do it, so it just going over all objects under 'folder/' and filetering them out. this can take over several minutes to complete. Signed-off-by: Danny Zaken <dannyzaken@gmail.com> (cherry picked from commit 219699d)
1 parent a417731 commit 8f665cf

File tree

0 file changed

+0
-0
lines changed

    0 file changed

    +0
    -0
    lines changed

    0 commit comments

    Comments
     (0)