[hook] `hook/generate.dart` 🪝

[dcharkes]

We'd like a standardized way to generate code in Dart packages that does not require a user step.

• Macro's can only take Dart code as input and output.
• package:build_runner requires users to do a manual extra steps

In some hook/generate.dart this would be a package:build_runner equivalent, but it would be aware of the Dart/Flutter SDK and have no race conditions between saving files and running dart and flutter commands (see below).

In #54334, we've discussed multiple aspects w.r.t. code generators. Today we discussed some more requirements offline:

• Automatically running code generators before pub publish to ensure no files are outdated.
• Automatically running code generators after editing dependencies. (Or having the IDE prompt to rerun them.)
• Automatically running code generators after pub get, before dart analyze. (Only needed if the generated code is not checked in.)

Assumptions:

• The generated files are checked in. (Maybe we should aim for not checking generated files in? E.g. it would work more like macros and you'd be able to resolve to the generated files in the IDE, but they are not checked in.)
• Code generators can be heavy to run (definitely longer-running than macros).

Some open questions:

• Should it be a different hook than hook/build.dart? Or a mode for the build hook?
  • If it's the build hook, when do we run this hook in which modes.
  • If it's not the build hook, how do we ensure this generate hook is finished before running the build hook? (E.g. generate is run on-save of its dependencies, like build_runner. But then if you hot-restart in Flutter, you want that on-save action to be fully done first.)
  • This should be a different hook from hook/build.dart, it's run in a different phase in the developer workflow. Build hooks are run on flutter build/dart build when you want to make an application bundle for a specific target (target OS, target architecture, target OS API levels, specific flavor, etc.) The generate hook should be run after dependencies change (and before running dart analyze, to prevent dart analysis errors showing up when both the source and target is Dart.

Use cases:

• FFIgen, JNIgen, Swiftgen interop
• translation messages (JSON -> Dart API)
• All build_runner setups

Background knowledge:

• build.rs enables generating of Rust files. https://doc.rust-lang.org/cargo/reference/build-script-examples.html#code-generation
  • (They don't mention how they solve IDE issues where code gen has not been run yet.)

Thanks @mosuem @HosseinYousefi @mkustermann @liamappelbe for the discussion! Please elaborate on things I left out.

[dart-github-bot]

Summary: This issue proposes a standardized way to generate code in Dart packages without requiring user intervention. It aims to address limitations of existing solutions like macros and build_runner, which either lack flexibility or require manual steps. The goal is to automatically run code generators before publishing, after dependency changes, and before analysis, ensuring generated code is up-to-date and consistent.

[mosuem]

Another option would be to include build_runner in the SDK somehow, solving the issue with package:build_runner requires users to do a manual extra steps. This might run against the macros efforts though.

[lrhn]

TL;DR: Either run automatically or check in the code, not both.

Macro's can only take Dart code as input and output.

That's an assumption for the input. We don't currently let macros access the file system, but we could.
If we did, then you could pass the URI of a file as an argument in the macro application, have the macro load the file (in a way that can be cached in the macro server between phases) and then it can generate code from that data too.

The alternative is to take the entire file, convert it to a Dart constant value, and have that in the macro annotation, which is just a two-step code generation where you generate the macro application code. (I'd totally do that.)

Assumptions:

• The generated files are checked in.

That's a very strong warning signal for me. Never check in auto-generated code.

It's OK to have a generator that you run manually to generate some code, and then you check in the generated code. It's a code updater then, not a code generator.
When people check out the code, the code they see is the code they get, nobody will overwrite it with something else. It's stable.

It's also OK to have an automatically run generator, which generates files on demand, before they are needed.
Those files should not be checked in. If the code that is checked in can be changed without you asking for it, then you can't trust the repository code anyway. It's better to have no code then.

If an automatically run generator is for Dart code, macros should be perfect for that. That's exactly the job macros are designed for, as long as they can get the data needed to generate the code. It's automatically generated code, guaranteed up-to-date, that you can still inspect, because every tool that processes Dart programs are aware of macros. It avoids any need ever storing the code, and checking it in.

Now, for non-Dart output, macros are obviously not the answer. There will need to be some code to generate such files. And a standard for where to place them. (Should the be in lib/ if another package depends on them?)

If files are checked in, I'd be more comfortable with a non-automatically run step, where you have to run dart generate to create/update the generated files. There is no need for someone just depending on a Pub packge to run dart generate for that package, if all generated files are checked in anyway.

Code generators can be heavy to run (definitely longer-running than macros).

That doesn't compute for me. Macros are code generators. You may have some heavy use-cases in mind, but whatever framework we create here should be able to handle both small and big tasks. And so should macros.

[dcharkes]

It's OK to have a generator that you run manually to generate some code, and then you check in the generated code. It's a code updater then, not a code generator.

Hm, I don't think we ever call these things code updaters, maybe we should. Then the hook should be called hook/update_code.dart? (My 2 cents, "generating" is a more understandable verb than "updating" in this context. Even if files are checked in such as with FFIgen/JNIgen/build_runner.)

If we'd like to hook build_runner into this, then let's look at build runners terminology:

The build_runner package provides a concrete way of generating files using Dart code.

https://pub.dev/packages/build_runner

cc @davidmorgan

[davidmorgan]

Thanks Daco! Looking forward to getting into the discussion here.

Re: checking in generated code, I don't think that's quite the right distinction to make. What's important is whether, after generation, the generator retains ownership of the output: has an opinion about whether it's correct.

I've been calling generators that do not retain ownership of their output "offline generators". A web page can be an offline generator, you copy its output into your source tree and you're done.

An online generator is the type of generator you want to run automatically.

Whether you check in the output of online generators or not is mostly a secondary issue related to convenience of other tools, for example you might do it so the output and diffs are visible on GitHub. And for pub, you publish the generated output today because otherwise your package doesn't work :)

Re: code generators being heavy to run--I'm working on this one, they should usually be about the same cost as lints.

[dcharkes]

An online generator is the type of generator you want to run automatically.

FFIgen and JNIgen are online generators in this distinction.

Re: code generators being heavy to run--I'm working on this one, they should usually be about the same cost as lints.

I'm not sure that FFIgen and JNIgen will be able to be that fast. So I think we'll always have faster and slower generators in our ecosystem.

That being said, I'd love for FFIgen and JNIgen to be so fast that we can run them on-save on the source files. That way users can edit native code and live see the Dart bindings being updated. 😄 cc @HosseinYousefi @liamappelbe