Reading (parts of) Mill configuration from a YAML file

[sideeffffect]

The draft of Release Notes for 1.0.0-RC1 reads

//| mill-version: 1.0.0
//| mill-jvm-version: 17
//| repositories: [$PWD_URI/custom-repo]
//| mvnDeps:
//| - com.grack:nanojson:1.8-custom-test
//| - com.lihaoyi::scalatags:0.12.0

package build
...

• The YAML front-matter allows you to configure arbitrary tasks in the meta-build.
  Above we show how to configure the def repositories and def mvnDeps tasks for the
  meta-build, but other meta-build tasks can be configured as well: scalaVersion,
  scalacOptions, etc. using the JSON version of the in-memory data structures.

• https://github.com/com-lihaoyi/mill/pull/4981/files#diff-124eabfdc4bba2a4656e255cda07a5fa4055ef1d86e686c9599ad75b71fa9501R67

That begs the question: can I configure some parts of my ordinary, non-meta, build also using YAML?
It could be nice to be able to specify some trivial things that don't require the full power of Scala as simple fields in a YAML file, like mill.yaml:

foo:
  scalaVersion: 2.13.11
  mvnDeps:
  - com.lihaoyi::scalatags:0.13.1
  - com.lihaoyi::mainargs:0.6.2
  test:
    mvnDeps: [ com.lihaoyi::utest:0.8.5 ]
    testFramework: utest.runner.Framework

with build.mill being mere

object foo extends ScalaModule {
  object test extends ScalaTests {
  }
}

[lihaoyi]

I hit this need in #5092 as well

[lihaoyi]

Probably won't do this for 1.0.0, but we can experiment with it and add it in a backwards compatible way later on

[lefou]

To experiment with it, a special meta-build could be used that reads the yaml file and does the (ideally automatic) mapping.

[lefou]

If we come up with a good spec, we could make the .yaml and the .mill setup interchangeable, so we could fall back to the code-version for modules, where custom tasks are needed, but use the YAML as a more declarative way when the setup is more standard. Don't know how easy and intuitive it is to mix YAML and Scala code in the same module. Worth a POC, to get some feeling. Having some markup-based way is definitely needed by some folks, as XML-based Maven or YAML-based Bleep shows, but without some nice way to write custom tasks like Mill, it's a dead end.

[lolgab]

How do you express super.mvnDeps() ++ using the new yaml config?

[lefou]

I think it's currently not possible.

[sideeffffect]

You don't. At that point, use the power of Scala.

[lolgab]

Which means that it's impossible to have the previous semantics of import $ivy. which appended dependencies instead of replacing. Probably it doesn't matter on the meta-build since it starts with an empty sequence.

[sideeffffect]

I should also add that this could have some significant downsides.
Currently, the pitch of Mill is that it's configured in pure and simple Scala and only Scala. No DSLs, perfect IDE integration, with all the goodies that come with it.

If Mill adopted this, it would suddenly shutter the configuration between two very different places. The normal Scala file and the new YAML file. Users would have to wonder where the configuration is coming from. Is it coming from YAML or Scala? Where should I look? Also, the same module could have some fields coming from YAML, some from Scala.

So this approach does have also some UX downsides...

[sideeffffect]

Some sort of compromise solution would require more opt-in from the side of the user, like having to extend a trait PopulateFromYaml. Those not extending would be guaranteed to be Scala-only.

object foo extends ScalaModule with PopulateFromYaml {
  object test extends ScalaTests {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.5")
    def testFramework = "utest.runner.Framework"
  }
}

foo:
  scalaVersion: 2.13.11
  mvnDeps:
  - com.lihaoyi::scalatags:0.13.1
  - com.lihaoyi::mainargs:0.6.2

[lefou]

This should implementable in user space as a POC, so it can be played with to get some feeling for. We don't need a commitment to merge it into core at this stage. But it would be nice to have the discussion here rather than somewhere else.