Tsaucer/pycatalog register on add

[timsaucer]

Related

Closes #11852

What

When new catalog providers are created, they are not automatically registered with the datafusion session context. This PR adds a check for new catalog providers when a table is created or registered.

[github-actions]

Web viewer built successfully.

Result	Commit	Link	Manifest
✅	6f3d985	https://rerun.io/viewer/pr/11878	+nightly +main

View image diff on kitdiff.

^{Note: This comment is updated whenever you push a commit.}

[abey79]

Do I understand correctly that this means everywhere in the SDK implementation we must know if we might alter the catalog, and if we do, call update_catalog_providers? This seems extremely error prone. Matter of fact, I believe at least Entry.delete() should update the catalog provider as well. (I did the exact same mistake with our very fragile __entries table stuff on OSS server.)

[timsaucer]

Do I understand correctly that this means everywhere in the SDK implementation we must know if we might alter the catalog, and if we do, call update_catalog_providers? This seems extremely error prone. Matter of fact, I believe at least Entry.delete() should update the catalog provider as well. (I did the exact same mistake with our very fragile __entries table stuff on OSS server.)

Yes, I'm trying to figure out a better way. I have started a PR for upstream to expose CatalogProviderList via FFI which will resolve this problem. That has missed the boat for DF 51, but maybe could make it into a patch release.

[timsaucer]

For reference: apache/datafusion#18657

[abey79]

As discussed last week, let's go for this as is until we have the better solution. Could you do a cleanup pass on the tests though? The _patched version should disappear (test that spam a Server instead of using a fixture should be avoided). Also, IIRC there are two such _patched tests (probably can be found by searching the todo with the issue i had opened).