Spec test sequence trace, a general test vector format for the Ethereum consensus layer

[leolara]

Spec test sequence trace, a general test vector format for the Ethereum consensus layer

Goals:

• Get more spec tests and generated vectors from them, by reducing the barriers and simplifying the process to create them
• Test developers should be able to use the spec normally, without having to know how to use yield statements to record stuff.
• So many formats make it more difficult for people writing test to know how to write them and choosing formats
• So many formats added all the time requires for consumers to add them in their code
• Formats are not flexible, leading us to add the same type of test many times and preventing us from creating tests with more different and complex scenarios
• Currently, if a big object is used many times in many tests, it is stored many times using disk space. This is easily fixable.

Test writer interface

The test writer just adds a decorator @spec_trace to the test. All the calls to the spec methods will be recorded automatically, because instead of a spec, they will receive a proxy to it.

Tests consumer interface

default_fork: "deneb"
trace:
	- op: "load_state"
	  state_root: 7a......
	- op: "spec_call"
	  method: "get_validator_activation_churn_limit"
	  assert_output: 100
	- op: "spec_call"
	  method: "process_slots"
	  input:
	    - slot: 15
	- op: "spec_call"
	  method: "process_slot"
	- op: "spec_call"
	  method: "process_epoch"
	- op: "assert_state"
	  state_root: 98a.....

The test developer doesn't have to add load_state or assert_state manually. The spec proxy will automatically add them.

Implementation details and space optimisations

Spec proxy

The test receives, instead of the spec object an proxy to the spec.

There are many python libraries that help implementing proxy objects, some of them are:

• wrapt
• lazy_object_proxy
• MappingProxyType
• proxy.py
• proxyscrape

Spec calls recording

The proxy records all calls to the spec internally, with its inputs and outputs in sequence.

Objects storage

The input and outputs that are ssz serializable will be treated differently. The proxy will detected them automatically by its type, and store them all in the same directory with name {hash_root}.ssz_snappy. In the YAML of the input/ouput we will store the file name.

Because the name is the hash_root if an object is used many times, it is only stored once.

State optimisation

Many spec methods, take state as an input and modify it. It would be ineficient and unnecessary to store the whole state for each of these calls.

The proxy will:

• The first time a state is used as input, will add a load_state entry to the trace.
• Will record the state as result of each call.
• When running a method, if the input state matches the previous output state, nothing is recorded.
• If the input state of a method is different from the modified state of the last method that modified the state, the proxy will ad a assert_state entry with the last output state and a load_state with the new state. This records the test developer modifies manually the state.
• At the end of the test, the proxy will add an assert_state entry to finish the test vector with the final state.

Trace capture and serialisation

The test trace is stored in a Pydantic object to help with validation and serialisation. Something like:

class BaseOperation(BaseModel):
    model_config = ConfigDict(extra="forbid")
    op: str


class LoadStateOp(BaseOperation):
    op: Literal["load_state"]
    state_root: str


class SpecCallOp(BaseOperation):
    op: Literal["spec_call"]
    method: str
    input: Optional[List[dict[str, Any]]] = None
    assert_output: Optional[Any] = None


class AssertStateOp(BaseOperation):
    op: Literal["assert_state"]
    state_root: str


class TraceConfig(BaseModel):
    default_fork: str
    trace: List[Union[LoadStateOp, SpecCallOp, AssertStateOp]] = Field(
        ..., discriminator="op"
    )

As currently the tests vectors are in YAML it will be outputted in YAML, but aditionally in json with json-schema. The json-schema will help in creating consumes more automatically.

Example

For example porpouses

@spec_trace
def test_an_example_test(spec, state):
	churn_limit = spec.get_validator_activation_churn_limit(state)
	spec.process_slots(state, 15)
	state.validators[0].balance = 0
	spec.process_slot(state)
	spec.process_epoch(state)

Will output something like:

default_fork: "deneb"
trace:
	- op: "load_state"
	  state_root: 7a......
	- op: "spec_call"
	  method: "get_validator_activation_churn_limit"
	  assert_output: 100
	- op: "spec_call"
	  method: "process_slots"
	  input:
	    - slot: 15
	- op: "load_state"
	  state_root: 8c......
	- op: "spec_call"
	  method: "process_slot"
	- op: "spec_call"
	  method: "process_epoch"
	- op: "assert_state"
	  state_root: 98a.....

Let's note several things:

• The test writer doesn't have to do anything special, creates a normal pytest test of the spec, without yields or anything outside calling the functions.
• The first time the state is used, the load of it is automatically generated
• When the state is manually modified a load of it is automatically generated
• At the end of the test the state root is automatically asserted