Meta: possible improvements to robustness and developer experience when integrating a new interview with e-filing

[nonprofittechy]

Opening this issue to think through possible refactorings of this package to improve both robustness and user experience. My suspicion is that the best way to do the refactors would be in a fork of this package (or maybe just a v2 YAML file) so we can avoid too much work rewriting existing interviews.

Some top level thoughts:

1. It's pretty hard to get a sense of what block is doing what when you integrate a new interview.

• We should think about one big function call that the author is required to make, so they can easily see what variables they are sending. This is more like a traditional API call and easier for a new developer to reason over.
• Another option is suggested in Figure out if we can and should create a more declarative, configuration-file like way to setup the efiling integration #244 - what limits are there to a single config block with all of the options? how would it handle conditional logic to set, e.g., the filing code for a document? Not everything can be declarative.
• Variable names should be cleaned up to be easier to understand.
• We need to clarify which variables are mandatory, and which are just provided for convenience by this package. Probably, we should have a more clear interface between those. It's OK to be repetitive in the interview YAML, if it's easier to see what's happening. I don't mind repeating myself a bit if it's easy to copy and paste one block and modify it - having to reason through various defaults is trickier sometimes.
• The DA execution order triggering the definition of various options is hard to trace. Can we make it at most one hop? Often there are many hops that you have to work backwards to see what is actually going into the API call.
• There is a mix of variables (with relatively generic names) and attributes that are defined at multiple different levels, ALDocument or ALDocumentBundle. It's not easy to see what attributes are required or where they're going to get a definition from. Can we make more explicit, even if it means more repetitive code?

2. The existing "just file everything" block seems to be opinionated about what variables need to be defined, but not flexible enough for all real interviews. Is every field truly required? If so, how can we make it easier to
3. Some ideas seem to be obsolete, like using the numeric code to decide what to file. As far as I can tell, codes are only rarely consistent from staging to production, and in some jurisdictions they vary per-court as well. So we should really emphasize the display name, and possibly even remove the numeric codes. They don't seem to be something we can actually use.

• The current method of a "filter" is not very easy to understand when you look at it in the interview YAML.
• If the filter is wrong, we might want to let the user override it.
• We should consider some fuzzy logic for things like weird punctuation or capitalization (but that might be risky)

3. Sending the whole ALDocumentBundle blob is convenient, but not transparent. Are some things we're defining as attributes of the blob better made into parameters of the REST API call?

• Of course, one ALDocumentBundle can have multiple associated filings that need to be treated differently although they are in a single envelope.

5. When things go wrong, there are a lot of potential errors that we don't properly handle. Any time a field is missing or incorrect, the end-user should have an opportunity to select it as well as they can. E.g., recent ILAO error Issue when checking Fees IllinoisLegalAidOnline/docassemble-AppearanceEfile#15 is one that wouldn't really have confused an end-user to be presented and forced to select on their own.

Other thoughts: we have a few places that we contact the server to see what's what.

1. To see if e-filing is even available. We ask for the court first and check on that specific court. Is this necessary for all jurisdictions? Probably, but it's not always the best time to ask about what court they are in.
2. To calculate fees. This seems to be more fragile than the final filing, and isn't even needed in a responsive filing. We get a lot of our production errors here at this stage.
3. To submit the actual filing.

I think we want to carefully try a filing, loop over each value for its correctness, and then have a way for the user to correct any "incorrect" answers at least stages 2 and 3. Maybe the review/fix will be in a single question/table. Then try again and see if the corrected values solve the problem. This is basically what ILAO wants out of a free-standing e-filing system.

One nuance: can/should we let the user somehow override our default values even if they seem "correct"? This might be needed in some small cases where court practice changes but it's not enforced by the e-filing system itself yet.

Also take a look at #254 ,#252, #251, #250, #259 for some additional specific ideas. Need to go back and cross-reference more existing issues soon.

[nonprofittechy]

Main opinionated take: we save the developer a lot of time with the current version of this package, but instead I think we should optimize for clarity and ability for the developer to know what values they are sending and which values they need to provide in an API call.