Autodetect rockstar format

[cphyc]

PR Summary

Rockstar catalogs unfortunately are not self-descriptive. This PR is an attempt to automatically detect the format into which a catalog is written by simply trying all the formats until one is found that looks alright.

[cphyc]

Note: this should only be merged after #4661 is merged

[brittonsmith]

I'm really sorry for never seeing this. It looks good to me, although I'm having trouble finding any data I can test it on. I remember that if you guess the format revision wrong the particle masses are nonsensical, but I can't remember if they are negative. I'm willing to take this PR's word for it since it provides a nice way to make additional tweaks.

The merge-base changed after approval.

[brittonsmith]

@cphyc, is it too late to merge this? Maybe you just need to merge the PR branch with upstream changes.

[cphyc]

Duh, I have no idea how your approval got dismissed but it is not too late to merge :)

[mtryan83]

What's the expected/intended procedure for adding revisions that aren't strict upgrades?

For example, I primarily use a fork of Rockstar from Andrew Wetzel (here), which follows revision 2, but has two additional fields (m_hires and m_lowres). So the path would be straightforward: add a revision 3, version-gate the two additional fields, and go from there. E.g:

KNOWN_REVISIONS: list[int] = [0, 1, 2, 3] # Added version 3
...
    ("av_density", np.float32, (2, 100)),
    ("m_hires", np.float32, (3, 100)), # New
    ("m_lowres", np.float32, (3, 100)), # New
]

But Andrew also has another fork (alt fork) which is based off of revision 1, with an additional field, halfmass_radius, between m_pe_d and num_p. Obviously, I could add halfmass_radii with the version gate (1, 1) or similar, but then there'd be problems if I use, say, Behroozi's original version for something. I wouldn't want to define it as revision 3 or 4 (assuming the above version is 3). Would the expectation be it comes as revision 1.5 or similar? (But then what if there's another fork based on revision 1 with a different added field...)

This is directly related to the version issues I mentioned in #5031. (And for an example of that, the Wetzel revision 2-based version defines HALO_FORMAT_REVISION as 1, while the revision-1 based version defines it as 2 🤣 )

[cphyc]

Hey @mtryan83,

So long as your alternative format has a different number of bytes in the header or in the members of the halos, what you wrote above should work (assuming m_hires and m_lowres are indeed located at the end of the halo definition).

If this works, would you mind opening a PR (on top of this one)?

As a side note, it would make the code much more readable if the syntax was:

halo_dt: list[HaloDataType] = [
    ("particle_identifier", np.int64),
    ("particle_position_x", np.float32),
    ("particle_position_y", np.float32),
    ("particle_position_z", np.float32),
    ("particle_mposition_x", np.float32, {0}),
    ("particle_mposition_y", np.float32, {0}),
    ("particle_mposition_z", np.float32, {0}),
    [...]
    ("m_pe_d", np.float32, {1, 2}),
    [...]
]

At the moment, the third (optional) argument gives a lower and upper boundary for the format revisions that should include the corresponding entry; using a set would be more explicit IMO.

The merge-base changed after approval.

The merge-base changed after approval.

[mtryan83]

My bad, my comment should probably have been a new issue (#5278), it's only tangentially related to this PR.

The merge-base changed after approval.

The merge-base changed after approval.