op-node fails to start with reth, but works with geth #305
Description
Description
When using the ethereum_package with participants: - el_type: reth, the op-node service fails to start and never becomes healthy. However, when using geth, everything works as expected.
Logs
Waiting for at most '15m0s' for service 'op-el-2151908-1-op-geth-op-node-op-kurtosis' to reach a certain state
Wait took 1 tries (275.723792ms in total). Assertion passed with following:
Request had response code '200' and body "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"id\":\"159cabb49d41e47a29267dc30be83c797f0fcc6c04a207bb74092460ca8e94ca\",\"name\":\"Geth/v1.101503.4-stable-2b9abb39/linux-arm64/go1.24.2\",\"enode\":\"enode://5655f55aa4b230b229c9439f5d6662f6a32ca36ab0d234d10f7fe524c13f4856571cc740863c1b8e5109aa3ee42e2501f5ba63c798482d587199ae8811e6bbe1@172.16.4.21:30303\",\"enr\":\"enr:-KO4QJbbvDoCdKKfqiT_J6G9gGPAnAMzg6DtXtyKq-DkmF_LF0lJlmI8KZ6-5R8sQMCWkFyAVfqNMkNRvttWGup3jvKGAZcakATLg2V0aMfGhLjEtrWAgmlkgnY0gmlwhKwQBBWJc2VjcDI1NmsxoQNWVfVapLIwsinJQ59dZmL2oyyjarDSNNEPf-UkwT9IVoRzbmFwwIN0Y3CCdl-DdWRwgnZf\",\"ip\":\"172.16.4.21\",\"ports\":{\"discovery\":30303,\"listener\":30303},\"listenAddr\":\"[::]:30303\",\"protocols\":{\"eth\":{\"network\":2151908,\"genesis\":\"0xce56c6a30911e686ad1377b05161665db3983e23b4bae8df04b20b3bda2851b0\",\"config\":{\"chainId\":2151908,\"homesteadBlock\":0,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"muirGlacierBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"arrowGlacierBlock\":0,\"grayGlacierBlock\":0,\"mergeNetsplitBlock\":0,\"shanghaiTime\":0,\"cancunTime\":0,\"pragueTime\":0,\"bedrockBlock\":0,\"regolithTime\":0,\"canyonTime\":0,\"ecotoneTime\":0,\"fjordTime\":0,\"graniteTime\":0,\"holoceneTime\":0,\"isthmusTime\":0,\"interopTime\":0,\"terminalTotalDifficulty\":0,\"depositContractAddress\":\"0x0000000000000000000000000000000000000000\",\"optimism\":{\"eip1559Elasticity\":6,\"eip1559Denominator\":50,\"eip1559DenominatorCanyon\":250}},\"head\":\"0xce56c6a30911e686ad1377b05161665db3983e23b4bae8df04b20b3bda2851b0\"},\"snap\":{}}}}\n", with extracted fields:
'extract.enr': "enr:-KO4QJbbvDoCdKKfqiT_J6G9gGPAnAMzg6DtXtyKq-DkmF_LF0lJlmI8KZ6-5R8sQMCWkFyAVfqNMkNRvttWGup3jvKGAZcakATLg2V0aMfGhLjEtrWAgmlkgnY0gmlwhKwQBBWJc2VjcDI1NmsxoQNWVfVapLIwsinJQ59dZmL2oyyjarDSNNEPf-UkwT9IVoRzbmFwwIN0Y3CCdl-DdWRwgnZf"
'extract.enode': "enode://5655f55aa4b230b229c9439f5d6662f6a32ca36ab0d234d10f7fe524c13f4856571cc740863c1b8e5109aa3ee42e2501f5ba63c798482d587199ae8811e6bbe1@172.16.4.21:30303"
Read JSON value
Command returned with exit code '0' and the following output: 0x459f58fd8ef3d9123333514d0f2445153fc29ddc43be239356dad70f506475d9
Adding service with name 'op-cl-2151908-1-op-node-op-geth-op-kurtosis' and image 'us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:develop'
There was an error executing Starlark code
An error occurred executing instruction (number 55) at github.com/ethpandaops/optimism-package/src/cl/op-node/op_node_launcher.star[114:31]:
add_service(name="op-cl-2151908-1-op-node-op-geth-op-kurtosis", config=ServiceConfig(image="us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:develop", ports={"http": PortSpec(number=8547, transport_protocol="TCP", application_protocol="http"), "metrics": PortSpec(number=9001, transport_protocol="TCP", application_protocol="http"), "rpc-interop": PortSpec(number=9645, transport_protocol="TCP", application_protocol="ws"), "tcp-discovery": PortSpec(number=9003, transport_protocol="TCP", application_protocol="", wait=None), "udp-discovery": PortSpec(number=9003, transport_protocol="UDP", application_protocol="", wait=None)}, files={"/jwt": "op_jwt_file", "/network-configs": Directory(artifact_names=["op-deployer-configs", "superchain-depset-superchain"])}, cmd=["op-node", "--log.level=DEBUG", "--l2=http://{{kurtosis:a224d75c1a154ada92154beedea9411c:ip_address.runtime_value}}:8551", "--l2.jwt-secret=/jwt/jwtsecret", "--verifier.l1-confs=1", "--rollup.config=/network-configs/rollup-2151908.json", "--rpc.addr=0.0.0.0", "--rpc.port=8547", "--rpc.enable-admin", "--l1=http://{{kurtosis:bcc0e9735d59407492343c80078466ae:ip_address.runtime_value}}:8545", "--l1.rpckind=standard", "--l1.beacon=http://{{kurtosis:b731dfadaad642cba08120567acafd3e:ip_address.runtime_value}}:4000", "--p2p.advertise.ip=KURTOSIS_IP_ADDR_PLACEHOLDER", "--p2p.advertise.tcp=9003", "--p2p.advertise.udp=9003", "--p2p.listen.ip=0.0.0.0", "--p2p.listen.tcp=9003", "--p2p.listen.udp=9003", "--safedb.path=/data/op-node/op-node-beacon-data", "--altda.enabled=false", "--altda.da-server=", "--metrics.enabled=true", "--metrics.addr=0.0.0.0", "--metrics.port=9001", "--p2p.sequencer.key={{kurtosis:78aba66702194eb6949bc6e0e7a0f8b1:output.runtime_value}}", "--sequencer.enabled", "--sequencer.l1-confs=2"], env_vars={"OP_NODE_INTEROP_DEPENDENCY_SET": "/network-configs/superchain-depset-superchain.json", "OP_NODE_INTEROP_JWT_SECRET": "/jwt/jwtsecret", "OP_NODE_INTEROP_RPC_ADDR": "0.0.0.0", "OP_NODE_INTEROP_RPC_PORT": "9645"}, private_ip_address_placeholder="KURTOSIS_IP_ADDR_PLACEHOLDER", ready_conditions=ReadyCondition(recipe=PostHttpRequestRecipe(port_id="http", endpoint="/", body="{\"jsonrpc\":\"2.0\",\"method\":\"opp2p_self\",\"params\":[],\"id\":1}", content_type="application/json", extract={"enr": ".result.ENR", "multiaddr": ".result.addresses[0]", "peer_id": ".result.peerID"}), field="code", assertion="==", target_value=200, timeout="1m"), labels={"ethereum-package.client": "op-node", "ethereum-package.client-image": "us-docker-pkg-dev-oplabs-tools-artifacts-images-op-node_develop", "ethereum-package.client-type": "beacon", "ethereum-package.connected-client": "op-geth", "ethereum-package.sha256": ""}, tolerations=[], node_selectors={}))
Caused by: Unexpected error occurred starting service 'op-cl-2151908-1-op-node-op-geth-op-kurtosis'
Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'op-cl-2151908-1-op-node-op-geth-op-kurtosis' with private IP '172.16.4.39'; this is usually due to a misconfiguration in the service itself, so here are the logs:
== SERVICE 'op-cl-2151908-1-op-node-op-geth-op-kurtosis' LOGS ===================================
t=2025-05-29T05:42:18+0000 lvl=info msg="Not opted in to ProtocolVersions signal loading, disabling ProtocolVersions contract now."
t=2025-05-29T05:42:18+0000 lvl=info msg="Using default bootnodes, none provided."
t=2025-05-29T05:42:18+0000 lvl=info msg="No persisted sequencer state loaded"
t=2025-05-29T05:42:18+0000 lvl=info msg="Rollup Config" l2_chain_id=2151908 l2_network="unknown L2" l1_chain_id=3151908 l1_network="unknown L1" l2_start_time=1748497326 l2_block_hash=0xce56c6a30911e686ad1377b05161665db3983e23b4bae8df04b20b3bda2851b0 l2_block_number=0 l1_block_hash=0x6323241d9268ea7ab811a49d550443ef1ae62c8e22da8bc519f79a6d3103cd94 l1_block_number=17 regolith_time="@ genesis" canyon_time="@ genesis" delta_time="@ genesis" ecotone_time="@ genesis" fjord_time="@ genesis" granite_time="@ genesis" holocene_time="@ genesis" isthmus_time="@ genesis" jovian_time="(not configured)" interop_time="@ genesis"
t=2025-05-29T05:42:18+0000 lvl=info msg="Initializing rollup node" version=untagged-84d9c840-1748361887
t=2025-05-29T05:42:18+0000 lvl=info msg="Connected to L1 Beacon API, ready for EIP-4844 blobs retrieval." version=teku/v25.4.1/linux-aarch_64/-eclipseadoptium-openjdk64bitservervm-java-21
t=2025-05-29T05:42:18+0000 lvl=info msg="Setting up Interop RPC server to serve supervisor sync work"
t=2025-05-29T05:42:18+0000 lvl=debug msg="Creating RPC handler"
t=2025-05-29T05:42:18+0000 lvl=debug msg="Allowed origin(s) for WS RPC interface [*]"
t=2025-05-29T05:42:18+0000 lvl=info msg="registered API" route="" namespace=interop
t=2025-05-29T05:42:18+0000 lvl=info msg="Safe head database enabled" path=/data/op-node/op-node-beacon-data
t=2025-05-29T05:42:18+0000 lvl=error msg="failed to fetch runtime config data" err="failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x71824d6d1f9e55e5f435928e29a0e166ada750060350bbda9c008b9f973b5668: distance to target block exceeds maximum proof window"
t=2025-05-29T05:42:28+0000 lvl=error msg="failed to fetch runtime config data" err="failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x9cbe5a2b05903282762f4eefd044c4af73202a5e91212df990d34c59bc46685d: distance to target block exceeds maximum proof window"
t=2025-05-29T05:42:38+0000 lvl=error msg="failed to fetch runtime config data" err="failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x77b7fb099544b75ec2849cb2672b204d43720d27f7e896f3fc0e89f9a52b8f60: distance to target block exceeds maximum proof window"
t=2025-05-29T05:42:48+0000 lvl=error msg="failed to fetch runtime config data" err="failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x60e0cd2da92fec91528ed129bc1520ed4a2c9a333520d42a85e8766d66f197c0: distance to target block exceeds maximum proof window"
t=2025-05-29T05:42:59+0000 lvl=error msg="failed to fetch runtime config data" err="failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0xa667c45c0eb0b2679d3cb77bf584bdd39af73b74e91c26fe2ae6e32922b8f798: distance to target block exceeds maximum proof window"
t=2025-05-29T05:42:59+0000 lvl=error msg="Error initializing the rollup node" err="failed to init the runtime config: failed to load runtime configuration repeatedly, last error: operation failed permanently after 5 attempts: failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0xa667c45c0eb0b2679d3cb77bf584bdd39af73b74e91c26fe2ae6e32922b8f798: distance to target block exceeds maximum proof window"
t=2025-05-29T05:42:59+0000 lvl=info msg="stopping node: sequencer already stopped" latestHead=0x0000000000000000000000000000000000000000000000000000000000000000
t=2025-05-29T05:42:59+0000 lvl=debug msg="Stopping RPC" route=""
t=2025-05-29T05:42:59+0000 lvl=debug msg="RPC server shutting down"
t=2025-05-29T05:42:59+0000 lvl=info msg="Stopped RPC server"
t=2025-05-29T05:42:59+0000 lvl=info msg="Interop sub-system stopped"
t=2025-05-29T05:42:59+0000 lvl=crit msg="Application failed" message="failed to setup: unable to create the rollup node: failed to init the runtime config: failed to load runtime configuration repeatedly, last error: operation failed permanently after 5 attempts: failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0xa667c45c0eb0b2679d3cb77bf584bdd39af73b74e91c26fe2ae6e32922b8f798: distance to target block exceeds maximum proof window"
== FINISHED SERVICE 'op-cl-2151908-1-op-node-op-geth-op-kurtosis' LOGS ===================================
Caused by: An error occurred while waiting for all TCP and UDP ports to be open
Caused by: Unsuccessful ports check for IP '172.16.4.39' and port spec '{privatePortSpec:0x4000dfd0b0}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
Caused by: An error occurred while calling network address '172.16.4.39:9001' with port protocol 'TCP' and using time out '200ms'
Caused by: dial tcp 172.16.4.39:9001: connect: no route to host
Error encountered running Starlark code.