Make sure anyllm class definitions are inside the `try:` block when relevant

[njbrake]

Need to make sure we properly define anyllm wrappers that depend on optional dependencies

Originally posted by @daavoo in #841 (comment)