Logger improvements

[edolstra]

Motivation

Some logger improvements to allow collecting useful info (especially in CI) such as hash mismatches, derivation/substitution results, etc.

• New option json-log-path that writes all log messages to the specified path (which can be a Unix domain socket) in addition to the regular output. This is implemented using a new TeeLogger.
• Add a structured log message for FOD hash mismatches. It looks like this:

  {
    "action": "result",
    "id": 5930065640554503,
    "payload": {
      "got": {
        "algo": "sha256",
        "base16": "8bef61bfc134bb9b968176270017cd8c4b48ab6dd3a755244f7a359d502079d7"
      },
      "storePath": "/nix/store/g9qc0lk1ra00hb1apdaw84b2l4j7187b-nix-0.10.1.tar.bz2.drv",
      "wanted": {
        "algo": "sha256",
        "base16": "8bed21bfc134bb9b968176270017cd8c4b48ab6dd3a755244f7a359d502079d7"
      }
    },
    "type": 109
  }
  

• Add a structured log message for the BuildResult of derivation/substitution goals. Example:

  {
    "action": "result",
    "id": 5901577122480358,
    "payload": {
      "path": {
        "drvPath": "/nix/store/xv0nd4m0sf329n2vdxch2kd6v3vihai4-patchelf-0.14.3.drv",
        "outputs": {
          "out": "/nix/store/6fcrjgfjip2ww3sx51rrmmghfsf60jvi-patchelf-0.14.3"
        }
      },
      "startTime": 1741891047,
      "status": "Built",
      "stopTime": 1741891057,
      "timesBuilt": 1
    },
    "type": 110
  }
  

Context

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[github-actions]

🚀 Deployed on https://67e5952f1bfe4b008e282cc2--determinate-nix.netlify.app

[cole-h]

LGTM and worked in local testing!

[elikoga]

It looks like you are incrementally expanding the internal-json format with new Message IDs and such. This is a signal for me that the internal-json interface is actually consumed by someone (except for nix-output-monitor and us) somewhere and that we can make some soft assumptions about stability (it's not gonna get ripped out tomorrow), as well as the fact that we won't be changing the internal-json message format anytime soon.

Am I correct in my rough assessment on future path of internal-json?

I had a few discussions with with other contributors due to ergonomic pains in using internal-json, however the PR alleviates some of the pains including one of the most important: locking of the output stream so the log messages don't arrive messed up.

[grahamc]

(without speaking on behalf of Eelco and his plans:) determinate-nixd consumes it, which let us drop the use of the post-build-hook. My intention is for determinate-nixd to be the only consumer of the Nix daemon's log output, since we accept and rebroadcast the events over the determinate-nixd events API.

Since we ship determinate-nixd and nix-daemon in a matched set (they're both at version 3.1.1) we're able to have greater interface flexibility than we would otherwise. To me this means we could change this format more, and in breaking ways. But not without good reason.

I guess what I mean to say is: I don't intend for this format to be a stable API, since that is what the determinate-nixd events API is intended to provide. However, you can probably build against it without too much annoyance or worry.

[edolstra]

I can't remember why we called it internal-json, probably because we were afraid to commit to stability and it's not documented anywhere. However, it's used between the client and the daemon so it's de facto required to be a stable format.

One thing I don't like about the current format is that message types are integers. They should probably be strings (e.g. "hash-mismatch" instead of 109).

[elikoga]

message types are integers. They should probably be strings

That was also discussed as a change one could make, since we are not using the self-documenting features of json currently. I don't see any problem in doing that change since the internal-json name came across correctly, as long as one makes sure to keep in touch with nix-output-monitor and friends.

I also believe that the internal-json messages passed around between the client and daemon are not part of the nix language behaviour as the name implies and one should be allowed to break older versions with newer daemons and vice versa along those lines, if communicated correctly.

[edolstra]

Upstream PRs:

• Add json-log-path setting NixOS/nix#13003