[General] L2 blocks stop going up on specific op-batcher configuration

[tiagorvmartins]

Discussion Topic

We have noticed that L2 blocks stop being produced for about 25 minutes to 30 minutes under specific op-batcher flag configurations, and we would like to understand the reason behind it, below you can check each component settings we are using that result in this failure scenario.

op-geth

./geth \
  --datadir=$OP_GETH_DATADIR \
  --http \
  --http.corsdomain="*" \
  --http.vhosts="*" \
  --http.addr=0.0.0.0 \
  --http.api=web3,debug,eth,txpool,net,engine,miner \
  --ws \
  --ws.addr=0.0.0.0 \
  --ws.port=8546 \
  --ws.origins="*" \
  --ws.api=debug,eth,txpool,net,engine,miner \
  --syncmode=full \
  --gcmode=archive \
  --nodiscover \
  --maxpeers=0 \
  --networkid=$L2_CHAIN_ID \
  --authrpc.vhosts="*" \
  --authrpc.addr=0.0.0.0 \
  --authrpc.port=8551 \
  --authrpc.jwtsecret=./jwt.txt \
  --cache=2048 \
  --rollup.disabletxpoolgossip=true \
  --verbosity=$OP_GETH_LOG_LEVEL

op-alt-da

./da-server -addr 127.0.0.1 -port 9999 -file.path $OP_ALT_DA_FILE_PATH --log.level=$OP_ALT_DA_LOG_LEVEL

op-node

./op-node \
  --l2=http://localhost:8551 \
  --l2.jwt-secret=${HOME}/jwt.txt \
  --sequencer.enabled \
  --sequencer.l1-confs=5 \
  --verifier.l1-confs=4 \
  --rollup.config=./rollup.json \
  --rpc.addr=0.0.0.0 \
  --p2p.disable \
  --rpc.enable-admin \
  --p2p.sequencer.key=$GS_SEQUENCER_PRIVATE_KEY \
  --l1=$L1_RPC_URL \
  --l1.beacon=${L1_BEACON_URL} \
  --l1.rpckind=${L1_RPC_KIND} \
  --altda.enabled=true \
  --altda.da-server=http://localhost:9999 \
  --safedb.path=${OP_NODE_SAFEDB_PATH} \
  --log.level=$OP_NODE_LOG_LEVEL

op-batcher

./op-batcher \
  --batch-type=1 \
  --compression-algo=brotli-10 \
  --poll-interval=5s \
  --target-num-frames=6 \
  --max-channel-duration=1500 \
  --sub-safety-margin=6 \
  --wait-node-sync=true \
  --check-recent-txs-depth=5 \
  --throttle-threshold=1000000 \
  --throttle-tx-size=300 \
  --throttle-block-size=21000 \
  --throttle-always-block-size=130000 \
  --num-confirmations=5 \
  --resubmission-timeout=30s \
  --safe-abort-nonce-too-low-count=3 \
  --private-key=$GS_BATCHER_PRIVATE_KEY \
  --l1-eth-rpc=$L1_RPC_URL \
  --l2-eth-rpc=http://localhost:8545 \
  --rollup-rpc=http://localhost:9545 \
  --rpc.addr=0.0.0.0 \
  --rpc.port=8548 \
  --rpc.enable-admin=true \
  --altda.enabled=true \
  --altda.da-server=http://localhost:9999 \
  --altda.da-service=true \
  --log.level=INFO

op-proposer

./op-proposer \
    --poll-interval=$OP_PROPOSER_POLL_INTERVAL \
    --rpc.port=8560 \
    --rollup-rpc=http://localhost:9545 \
    --private-key=$GS_PROPOSER_PRIVATE_KEY \
    --l1-eth-rpc=$L1_RPC_URL \
    --game-type=$OP_PROPOSER_GAME_TYPE \
    --game-factory-address=$GAME_FACTORY_PROXY_ADDRESS \
    --wait-node-sync=true \
    --proposal-interval=$OP_PROPOSER_PROPOSAL_INTERVAL \
    --log.level=$OP_PROPOSER_LOG_LEVEL

op-challenger

./op-challenger \
    --cannon-bin="./bin/cannon" \
    --cannon-server="./bin/op-program" \
    --cannon-prestate="./bin/prestate.bin.gz" \
    --l2-genesis=./genesis.json \
    --rollup-config=./rollup.json \
    --l1-eth-rpc=$L1_RPC_URL \
    --l1-beacon=$L1_RPC_URL \
    --rollup-rpc="http://localhost:9545" \
    --l2-eth-rpc="http://localhost:8545" \
    --private-key=$GS_CHALLENGER_PRIVATE_KEY \
    --datadir=$OP_CHALLENGER_DATA_DIR \
    --game-factory-address=$GAME_FACTORY_PROXY_ADDRESS \
    --trace-type="permissioned" \
    --game-window="100h" \
    --max-concurrency=4 \
    --metrics.enabled \
    --metrics.port=7301 \
    --pprof.enabled \
    --pprof.port=6061 \
    --log.level=$OP_CHALLENGER_LOG_LEVEL

After experimenting different flags on op-batcher, we no longer experience the same issue, we changed op-batcher to the following:

./op-batcher \
  --batch-type=0 \
  --compression-algo=zlib \
  --poll-interval=2s \
  --target-num-frames=1 \
  --max-channel-duration=1800 \
  --sub-safety-margin=6 \
  --wait-node-sync=true \
  --check-recent-txs-depth=0 \
  --num-confirmations=10 \
  --resubmission-timeout=48s \
  --safe-abort-nonce-too-low-count=3 \
  --private-key=$GS_BATCHER_PRIVATE_KEY \
  --l1-eth-rpc=$L1_RPC_URL \
  --l2-eth-rpc=http://localhost:8545 \
  --rollup-rpc=http://localhost:9545 \
  --rpc.addr=0.0.0.0 \
  --rpc.port=8548 \
  --rpc.enable-admin=true \
  --altda.enabled=true \
  --altda.da-server=http://localhost:9999 \
  --altda.da-service=true \
  --log.level=$OP_BATCHER_LOG_LEVEL

We would like to understand why that is the case?
We thought we could use span batches and increase the target num frames,
Thank you!

[imtipi]

what altda you use?any logs during l2 production outage?what l1 chain you use?

--compression-algo=brotli-10 can you check cpu usage? high compression rate also means high cpu requirement

[tiagorvmartins]

Hey @imtipi thank you for your reply,
We are using op-alt-da from optimism itself, we are using sepolia as l1 chain.
Here are some logs from op-batcher during the l2 production outage, which happened today between 11h04m and 11h29m.

 Sep 11 11:01:01 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x13e25363ccfbabfc4e0a0843509ab13af3ebe529ea655a0bf85fb982d0cc4e45:1470699 tx_count=1 time=1757584854
 Sep 11 11:01:01 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xc544828a39de14a19b4f675aee29cd0ac5c76d1e6d6874b2d32fd1a914649518:1470700 tx_count=1 time=1757584856
 Sep 11 11:01:01 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x0aa840a429f81e2eabc5e967768bcf2e752f63dba0eac957abfab1029af0468e:1470701 tx_count=1 time=1757584858
 Sep 11 11:01:05 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470702 end=1470703
 Sep 11 11:01:05 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xaa5705cff034340bc8d052f321a59959dc1760f4b5d8045665b340fb1b069eab:1470702 tx_count=1 time=1757584860
 Sep 11 11:01:05 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x55aed68ccf4217f79f496a3b6db27719bbd5762a461bbe29510bb6569fab9740:1470703 tx_count=1 time=1757584862
 Sep 11 11:01:11 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470704 end=1470706
 Sep 11 11:01:11 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xca6c36698ea12de365085c07bad8ce7634203d0e10a103b584da395f7c0c2099:1470704 tx_count=1 time=1757584864
 Sep 11 11:01:11 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x621601f655fd1ef9f9239b988ae6ee65459b80a06b9ae95d43cd17afb1280955:1470705 tx_count=1 time=1757584866
 Sep 11 11:01:11 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xfbd5bcc2a2fbf9afbe39986d94a3f2d2ee3ebbb0996eaf1471cf8fc8a2a90158:1470706 tx_count=1 time=1757584868
 Sep 11 11:01:15 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470707 end=1470708
 Sep 11 11:01:15 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x884980100c7c819cfa67fe12163c4e54f4033ad85e627fb80d18c45c11028765:1470707 tx_count=1 time=1757584870
 Sep 11 11:01:15 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xf5fcd86d29d903ebb834b898c39eaf54a9c23bd4ce01bff72c0e4414782a59e9:1470708 tx_count=1 time=1757584872
 Sep 11 11:01:21 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470709 end=1470711
 Sep 11 11:01:21 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x201bf69710e7936cf41ae508cd4d195e1e51c890bf314231389b975ef3229c3a:1470709 tx_count=1 time=1757584874
 Sep 11 11:01:21 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x9b001e1a0b299a2541c8f40bcd9129b342c0a9de320a4436036fe6a427120744:1470710 tx_count=1 time=1757584876
 Sep 11 11:01:21 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xfd45ae97c9af4117b35d34f862a06a7c639cee318c1c9b77804f46c631104509:1470711 tx_count=1 time=1757584878
 Sep 11 11:01:26 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470712 end=1470713
 Sep 11 11:01:26 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xa5c88599eba9281a9da47a7940ed2b50df8f79b6c4ce7ffaf77169c45886017a:1470712 tx_count=1 time=1757584880
 Sep 11 11:01:26 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xf3a22e1f149f1d2d238ed129566f744f2a3a11647d4eb31d19967d9206441b85:1470713 tx_count=1 time=1757584882
 Sep 11 11:01:30 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470714 end=1470716
 Sep 11 11:01:30 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xcc2784bace4a56aafde167e12d8702ade8e7697904399bd780eca9137064d15f:1470714 tx_count=1 time=1757584884
 Sep 11 11:01:30 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xf3da112c430974ccd5e66d8337f6423d7a13e395f0a54f719a27330d25c83a30:1470715 tx_count=1 time=1757584886
 Sep 11 11:01:30 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x48a148673a53bd08a86b034d06d6722f82a553012799d4668e2958ce0c66eed3:1470716 tx_count=1 time=1757584888
 Sep 11 11:01:36 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470717 end=1470718
 Sep 11 11:01:36 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x81a4f4da4a3082cc7462c39e24da677a43e54fdeec04448087076850cb3f0700:1470717 tx_count=1 time=1757584890
 Sep 11 11:01:36 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x053a4766533814cf554f8c20c53ceaa5108336816fc9104fdc85174acb55fe31:1470718 tx_count=1 time=1757584892
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470719 end=1470721
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x274790663cf4cbe73270f30fa79f6182e95d3e9d3dc0c8a00ba9fd0bb4db996f:1470719 tx_count=1 time=1757584894
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x141228b600a301974d7177fb7022556f08785bd9ca9b3f7cd486b234f51c7f7e:1470720 tx_count=1 time=1757584896
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x92f7aaa911cc586fd27ddb088a4a0e27622ba8ee3f180c94d6e14dbe8a4adf58:1470721 tx_count=1 time=1757584898
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Channel closed" id=63720e61c2ce7aa3cefaf7823cad19ac blocks_pending=0 num_frames=1 input_bytes=10832 output_bytes=364 oldest_l1_origin=0x82ef247d8ec8415aaae1532eac157b29bfde59f85fb3f295529555eb5dbdc578:9178860 l1_origin=0x887888ea87d2394f742beed40ca7b953d4690f43a268729c652e611405ddac12:9180354 oldest_l2=0x97b8cba4c81ed7c42c9641304c9f7cbbb46285cd14440973ef85bee314ac31e9:1461142 latest_l2=0x92f7aaa911cc586fd27ddb088a4a0e27622ba8ee3f180c94d6e14dbe8a4adf58:1470721 full_reason="channel full: max channel duration reached" compr_ratio=0.03360413589364845
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Set altda input" commitment=00caf22e4719c268defb6e1d85d93f15703f6d0502fe8a4a49e370af5e54d4feed tx=63720e61c2ce7aa3cefaf7823cad19ac:0
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Building Calldata transaction candidate" size=34
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Publishing transaction" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f nonce=356 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=22330
 Sep 11 11:01:40 op-stack-281185 op-batcher msg="Transaction successfully published" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f nonce=356 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=22330 tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f
 Sep 11 11:01:46 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470722 end=1470723
 Sep 11 11:01:46 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xa776eb078c335ba0be1c126c2bbf62935ea61db6e266508f49a005082edfa5de:1470722 tx_count=1 time=1757584900
 Sep 11 11:01:46 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x3639c745ed4af60fa48cd3d31ebaf3c2231256cc22113f964613729f0a1ff767:1470723 tx_count=1 time=1757584902
 Sep 11 11:01:46 op-stack-281185 op-batcher msg="Created channel" id=3485ae431769e9aa8c7b1d592be6def9 l1Head=0xfa763026fc2b99c176f9abb7d388c2edd7f4fa2ab24a60a8d5c5169c0fd791d6:9180360 blocks_pending=2 l1OriginLastSubmittedChannel=0x887888ea87d2394f742beed40ca7b953d4690f43a268729c652e611405ddac12:9180354 batch_type=1 compression_algo=brotli-10 target_num_frames=6 max_frame_size=119999 use_blobs=false
 Sep 11 11:01:50 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470724 end=1470726
 Sep 11 11:01:50 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x8244b10a98c1f101aad0eda00b394c10ea1494914105f9fb7f5c36a2279b61e7:1470724 tx_count=1 time=1757584904
 Sep 11 11:01:50 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xaa18851cf69c900fafb13ecdfe42580f6117fef997b9150239c10606c7a3b6c2:1470725 tx_count=1 time=1757584906
 Sep 11 11:01:50 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x1aedfa8329a930a82b506b32361abaeb7e08539bae75948b84850285d87c4eb9:1470726 tx_count=1 time=1757584908
 Sep 11 11:01:52 op-stack-281185 op-batcher msg="Publishing transaction" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f nonce=356 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=22330
 Sep 11 11:01:56 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470727 end=1470728
 Sep 11 11:01:56 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xb550545a37317facef07bb86099878ef9d9a10a7184ae826b0dc893840d1c37a:1470727 tx_count=1 time=1757584910
 Sep 11 11:01:56 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x6117bfddac9f7e9c76d8ba5cf66551c6c7551f05452c0e1794f84c274f46f8e4:1470728 tx_count=1 time=1757584912
 Sep 11 11:02:00 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470729 end=1470731
 Sep 11 11:02:00 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xa85544db5cb8dc496276e281f94e2e456c0ddda07ae41bce68b96105e4eefdca:1470729 tx_count=1 time=1757584914
 Sep 11 11:02:00 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x2bc97bd2ef4bf82b32ed1092c7a07ade2573acf3630a6b3ab4e4f1b61f3e26ae:1470730 tx_count=1 time=1757584916
 Sep 11 11:02:00 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xeabcaef2af1528c1b812fa76e8b7143aee57aaa787aae66d426eb52d6844b18d:1470731 tx_count=1 time=1757584918
 Sep 11 11:02:06 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470732 end=1470733
 Sep 11 11:02:06 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x2efcd6af7c20bab62e51928d72821b62e16b2b018358c93ed0209c0f8f11d6b3:1470732 tx_count=1 time=1757584920
 Sep 11 11:02:06 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x4cf0b2b22810a590c63a7841abe3ee6a638c20c5c01255be275ecf914e302566:1470733 tx_count=1 time=1757584922
 Sep 11 11:02:10 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470734 end=1470736
 Sep 11 11:02:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xfa403df7bf8d00d76d5cf721f60e2c1aa50b00918ff20b8264eb09c4be2c3269:1470734 tx_count=1 time=1757584924
 Sep 11 11:02:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x2b8a66e15e9a828a266e43230157f3a183f6c1efd726761c7d3067671d6b47f0:1470735 tx_count=1 time=1757584926
 Sep 11 11:02:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x5d1cc640621c44f522efdaabb3840775e7657874a0042b8e43b070cd3e4de779:1470736 tx_count=1 time=1757584928
 Sep 11 11:02:16 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470737 end=1470738
 Sep 11 11:02:16 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x9658e7a6846a483adb0f0e87fb9d854a30936327dec4a64604c3defc943d4697:1470737 tx_count=1 time=1757584930
 Sep 11 11:02:16 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xe616551e617e18dea2f814d731d9c6c6c3a4ca6a919181cd98faf558aac3eae6:1470738 tx_count=1 time=1757584932
 Sep 11 11:02:20 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470739 end=1470741
 Sep 11 11:02:20 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x4fd52b4053fc63137c832d29d8dd09395540090446b2ef7658f4614063206759:1470739 tx_count=1 time=1757584934
 Sep 11 11:02:20 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x585acd74fba52053d8c90bc4e7e1f8f5aea209f79432853ab82b21f4d772eeed:1470740 tx_count=1 time=1757584936
 Sep 11 11:02:20 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x502866f02bc10cfb18c5273675acd7c70756af9acf71f031675fcaa7dc7f1ce4:1470741 tx_count=1 time=1757584938
 Sep 11 11:02:30 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470744 end=1470746
 Sep 11 11:02:30 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x05e599ab9fb3144b5428e29af456dab4df2cc93f91c8d889dee3423f2d4b3aa5:1470744 tx_count=1 time=1757584944
 Sep 11 11:02:30 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x1c757fe92ebdcf4f45c505bc1361b42a89495677090b7b999b432b4254bd4075:1470745 tx_count=1 time=1757584946
 Sep 11 11:02:30 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xf20f033146273a51683b91c5451b2385a97c6a1c1f09e60af2763751721b5cf4:1470746 tx_count=1 time=1757584948
 Sep 11 11:02:35 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470747 end=1470748
 Sep 11 11:02:35 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x24320a82f7ecbb84509f49de2d4e4ec910f8384f32ffe3dde1276c469605c456:1470747 tx_count=1 time=1757584950
 Sep 11 11:02:35 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xde8e63a2acfcf3cf2a81de7ab48c4d2512b3b16fcd2745bf08194e0bc379f75e:1470748 tx_count=1 time=1757584952
 Sep 11 11:02:41 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470749 end=1470751
 Sep 11 11:02:41 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xb076e69ab0ea924bb52a3fbfb7df492c3d473450c5e31032d40f846d42263c08:1470749 tx_count=1 time=1757584954
 Sep 11 11:02:41 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xf90f03d325ff54ae3098075fb1f8c342e3efee4d7743d4024dddff77bf7d91c0:1470750 tx_count=1 time=1757584956
 Sep 11 11:02:41 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xfd2b8d2b77fb1a9991c80f02718c7dde120502ef124e53dc123a5973094027de:1470751 tx_count=1 time=1757584958
 Sep 11 11:02:45 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470752 end=1470753
 Sep 11 11:02:45 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x826787bdbe57f219650846ebaccc7fedd0947fd8eff477a5bad456c94e489477:1470752 tx_count=1 time=1757584960
 Sep 11 11:02:45 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xbe2b6c843248e436cafa7f8f4396e7f4f6bc3f1b82f5fb7147294780e5891de2:1470753 tx_count=1 time=1757584962
 Sep 11 11:02:51 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470754 end=1470756
 Sep 11 11:02:51 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xd7b08d9bb9b68b0e1ce3774765e5ef4ad6ac991b2f16dbd7b2c33feb5d75e40b:1470754 tx_count=1 time=1757584964
 Sep 11 11:02:51 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x4e91b36e16bd22d63586c4b0cc1c5ed0d9169123b8eff64983432f7a36fc46da:1470755 tx_count=1 time=1757584966
 Sep 11 11:02:51 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x69e74ecbd9e12d6cfd95c2d9236c6348bdb57dba7dbdb844968fc2346d1e948a:1470756 tx_count=1 time=1757584968
 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Transaction confirmed" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f block=0xa16ce9a9e33b562722605aea5d61b4bd0635b9ca17648275dc7cc890435a9d53:9180361 effectiveGasPrice=1000028056
 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Receipt retrieval failed" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f err="Post \"https://sepolia.companynode.dev/token\": context canceled"
 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Transaction receipt not found" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f nonce=356 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=22330 err="context canceled"
 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Handling receipt" id=63720e61c2ce7aa3cefaf7823cad19ac:0
 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Transaction confirmed" tx_id=63720e61c2ce7aa3cefaf7823cad19ac:0 tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f block=0xa16ce9a9e33b562722605aea5d61b4bd0635b9ca17648275dc7cc890435a9d53:9180361
 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Channel is fully submitted" id=63720e61c2ce7aa3cefaf7823cad19ac min_inclusion_block=9180361 max_inclusion_block=9180361
 Sep 11 11:02:55 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470757 end=1470758
 Sep 11 11:02:55 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xb2a110b28633edd655fa555f814af4ac06710b81995cc55a0719a57b001c97e9:1470757 tx_count=1 time=1757584970
 Sep 11 11:02:55 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xb1abaa3e9e82ac94e4715f9ff5c8fb199be183d542a63b02b20a00ab94466157:1470758 tx_count=1 time=1757584972
 Sep 11 11:03:01 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470759 end=1470760
 Sep 11 11:03:01 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x3e17528e7773a976b861e174fa7056100b8cb3af5e5e98090cd61db2efb4a8a5:1470759 tx_count=1 time=1757584974
 Sep 11 11:03:01 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x51e4af90f1a40c8828312312a1d0919fd5f96aa50ef514d52c12da8a77e2f262:1470760 tx_count=1 time=1757584976
 Sep 11 11:29:04 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xe0dd293f4ceea83f5c1b877ee74579633ed2340a52e82e2833e0e165b31c400a:1470761 tx_count=1 time=1757584978
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Loading range of multiple blocks into state" start=1470762 end=1470772
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x06604b991e95e4658c6720199f204dbc7ad99bb0f94aac138090f0b9ee1ab77f:1470762 tx_count=1 time=1757584980
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xe2569e04d79c20888a5658e629b1b70a19f1cf688c3e961a40b1c57f449a0984:1470763 tx_count=1 time=1757584982
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xf141d2e3ddc5f5ec243ccfdfa4c6cd5a7b2b3a8465621d8eda2d5b9e1748f3b7:1470764 tx_count=1 time=1757584984
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xd7be2c696af306768d53c757f86bb4a35dd3451588a64b24d2540ecac4ec0730:1470765 tx_count=1 time=1757584986
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x5290a2dde2bd18a7d37e79476159826c2f049023e8e7ca44b13867a93342f5cf:1470766 tx_count=1 time=1757584988
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0x7c4af0d4e8df9c07bfeeb15735cd3282dd8fc9854bb1e1b51bda516cd2a9ceef:1470767 tx_count=1 time=1757584990
 Sep 11 11:29:10 op-stack-281185 op-batcher msg="Added L2 block to local state" block=0xa7d34565c49a575275eba8f3f0c4551f3a9b383b4c0206f042aaff14e5e50931:1470768 tx_count=1 time=1757584992

Also we are prety sure its not a problem of cpu usage, regarding some metrics as you can see below at that time:

11:20:04        CPU     %user     %nice   %system   %iowait    %steal     %idle
11:30:12        all      0.29      0.00      0.04      0.33      0.00     99.35
11:30:12          0      0.30      0.00      0.04      0.29      0.00     99.36
11:30:12          1      0.27      0.00      0.03      0.31      0.00     99.38
11:30:12          2      0.24      0.00      0.03      0.36      0.00     99.37
11:30:12          3      0.28      0.00      0.03      0.35      0.00     99.34
11:30:12          4      0.31      0.00      0.03      0.41      0.00     99.25
11:30:12          5      0.28      0.00      0.03      0.29      0.00     99.41
11:30:12          6      0.28      0.00      0.03      0.36      0.00     99.33
11:30:12          7      0.28      0.00      0.08      0.25      0.00     99.39
11:30:12          8      0.31      0.00      0.05      0.34      0.00     99.30
11:30:12          9      0.32      0.00      0.03      0.33      0.00     99.32

11:30:12        CPU     %user     %nice   %system   %iowait    %steal     %idle
11:40:04        all      0.27      0.00      0.03      0.30      0.00     99.40
11:40:04          0      0.24      0.00      0.04      0.32      0.00     99.40
11:40:04          1      0.24      0.00      0.03      0.27      0.00     99.45
11:40:04          2      0.29      0.00      0.03      0.32      0.00     99.36
11:40:04          3      0.27      0.00      0.03      0.29      0.00     99.41
11:40:04          4      0.25      0.00      0.03      0.28      0.00     99.44
11:40:04          5      0.25      0.00      0.03      0.28      0.00     99.44
11:40:04          6      0.20      0.00      0.02      0.24      0.00     99.54
11:40:04          7      0.37      0.00      0.07      0.28      0.00     99.29
11:40:04          8      0.27      0.00      0.03      0.42      0.00     99.27
11:40:04          9      0.29      0.00      0.03      0.29      0.00     99.40

Memory wise we noticed we are running with 8gb ram and usually it's at 3.31G/7.30G (used/total).
Regarding the --compression-algo=brotli-10 , we also tried with --compression-algo=zlib and the same issue happened,
Any clues? thank you!

[jsvisa]

Seems you have network issues between your op-batch and your l1 rpc:

 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Receipt retrieval failed" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f err="Post \"https://sepolia.companynode.dev/token\": context canceled"
 Sep 11 11:02:53 op-stack-281185 op-batcher msg="Transaction receipt not found" service=batcher tx=0x73cd6c89bca279e78e47581bf1e60356382434eab51d486a7722654a7c3f681f nonce=356 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=22330 err="context canceled"

[opfocus]

Please monitor block production by monitoring the op-node https://docs.optimism.io/operators/node-operators/management/metrics

[tiagorvmartins]

Hey @opfocus here are logs files of op-node between 11h37m and 11h38m, blocks stop going up at this time
Let me know if you spot anything, thank you
logs_op_node.txt

[opfocus]

Please provide a section of the op-node log that you think is normal. and to monitor the memory usage if something goes wrong, I read the logs you provided.