v1 model arguments default values

[RobinPicard]

The current approach in the v1 is that the users are responsible for providing whatever arguments are necessary for the model library they have chosen (both for initialization and for inference).

An issue is that, for instance, the transformers library sets the default value of its argument max_new_tokens to 20. If a user forgets to override this default value, constrained generation will often fail because of an incomplete generation caused by the an overly low number of new tokens that can be generated. That's a problem as users may not understand the origin of the issue and blame outlines for it.

Question: should we create exceptions to our general approach and modify the default value of some arguments for some models when we anticipate that not doing so would create errors for out users?

[cpfiffer]

I think we should avoid exceptions, as this would essentially show an error on a user's first experience with Outlines. This would be kind of painful/annoying. I think my recommendation is to default max_new_tokens to something very large. Users who know what they're doing can adjust this directly, since the old Outlines behavior is to generate mostly forever (and I think this is a good default)

[rlouf]

I would agree with setting a default value "large enough". I just got bitten by this trying to run examples.

[RobinPicard]

I'm going to open a PR for max_new_tokens for Transformers and then I think we should treat other cases one by one, trying to be stringent to avoid an inflation of outlines-created default values.

[rlouf]

I am not super comfortable changing the libraries' defaults actually. I assume there must be a good reason these defaults are what they are.

[RobinPicard]

Ok. Let's not change anything then. I'm writing the migration guide at the moment, there's a part in which I warn users about the removal of outlines-created default values.

[cpfiffer]

Our use case is different though -- users will almost always have structured output that fails by default, and we have a good reason to adjust the defaults to something that match the use case that we provide.