Add a way to show the contents of the ListFilesCache in `datafusion-cli`

[alamb]

Is your feature request related to a problem or challenge?

• Related to Adds memory-bound DefaultListFilesCache #18855
• part of [EPIC] ListingTable object store usage improvements #17214

As we roll out the ListingFileCache from @BlakeOrth in #18855 it would be very helpful to be able to see its contents to debug any potential issues we see

@nuno-faria made a really nice feature to view the contents of the cache: metadata_cache()

For example:

> select * from metadata_cache();
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
| path                                              | file_modified       | file_size_bytes | e_tag                                | version | metadata_size_bytes | hits | extra            |
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
| hits_compatible/athena_partitioned/hits_1.parquet | 2022-07-03T15:33:57 | 174965044       | "1f5da68e097309811a675c849491ac48-9" | NULL    | 165128              | 0    | page_index=false |
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
1 row(s) fetched.
Elapsed 0.005 seconds.

Describe the solution you'd like

I would like a table function similar to metadata_cache() for the listing files cache. Since each entry is a Vec<ObjectMeta> one option would be to flatten the entries so there is one entry per ObjectMeta stored:

Someting like

select * from list_files_cache();

path	file_modified	file_size_bytes	e_tag	version	metadata_size_bytes	expires
/foo/bar	2022-07-03T15:33:57	1234	...	...	132	NULL
/foo/baz	2022-07-03T15:33:57	5678	...	...	3112	2026-07-03T15:33:57
...	...	...	...	...	...	...

Where metadata_size_bytes shows the size of the statistics, in bytes and expires shows when the entry expires

This would mean that a single ListFilesEntry object is displayed as multiple rows.

It would also mean we would have to find some way to represent a ListFilesEntry that had no entries (e.g. metas is an empty Vec). Perhaps it could have a row entirely of nulls:

path	file_modified	file_size_bytes	e_tag	version	metadata_size_bytes	expires
NULL	NULL	NULL	NULL	NULL	NULL	2026-07-03T15:33:57

Describe alternatives you've considered

No response

Additional context

No response

[BlakeOrth]

@alamb Can you help me understand the benefit of flattening the Vec<ObjectMeta> into individual rows?

I'm concerned this could make the output table difficult to read (the s3://clickhouse-public-datasets/hits_compatible/athena_partitioned/ dataset would result in 100 rows, for instance) and I'm not sure what benefit it would provide since the TTL and metadata size for each element of the Vec will be, by and large, uniform. Additionally the cache (at least currently) evicts the entire Vec as opposed to individual entries, so we wouldn't expect any fragmentation issues with entries within a collection of metadatas.

[alamb]

@alamb Can you help me understand the benefit of flattening the Vec<ObjectMeta> into individual rows?

The core benefit in my opinion is that it would be possible to easily see and search for individual files in the cache

However, now that DataFusion's structured type support (aka ListArray) is much better, returning a ListArray<> is probably better