Skip to content

Conversation

ninabarbakadze
Copy link
Member

@ninabarbakadze ninabarbakadze commented Jul 23, 2025

Overview

Fixes #5133

@ninabarbakadze ninabarbakadze self-assigned this Jul 23, 2025
Copy link
Member Author

@ninabarbakadze ninabarbakadze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocked on being able to access the failure log for rejected transactions because in core we currently only check transactions in rejected cache and return tx status without error code or logs.

@ninabarbakadze ninabarbakadze added the warn:blocked item is not currently being worked on but is still blocked label Jul 29, 2025
Copy link
Member

@evan-forbes evan-forbes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apologies, reviewed then noticed it was a draft

Comment on lines 599 to 668
if apperrors.IsNonceMismatchCode(resp.ExecutionCode) {
expired, err := client.waitForTransactionToExpire(ctx, txHash)
if err != nil {
return nil, fmt.Errorf("failed to wait for transaction to expire for resubmission: %w", err)
}
if expired {
fmt.Println("tx with hash", txHash, "expired")
// resign and resubmit
txInfo, exists := client.txTracker[txHash]
if !exists {
return nil, fmt.Errorf("transaction %s not found in tracker", txHash)
}

// check if it's a blob transaction or just a normal transaction
// decode the original transaction to extract messages
decodedTx, err := client.signer.DecodeTx(txInfo.txBytes)
if err != nil {
return nil, fmt.Errorf("failed to decode transaction: %w", err)
}

// extract messages from the decoded transaction
msgs := decodedTx.GetMsgs()

// Check if this is a PayForBlob transaction
if len(msgs) > 0 {
// Check if the first message is a PayForBlob message
if _, ok := msgs[0].(*types.MsgPayForBlobs); ok {
// This is a blob transaction, we need to extract the blobs
// and recreate the PayForBlob transaction
blobTx, isBlob, err := blobtx.UnmarshalBlobTx(txInfo.txBytes)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal blob transaction: %w", err)
}
if !isBlob {
return nil, fmt.Errorf("expected blob transaction but got normal transaction")
}

// Resubmit as a PayForBlob transaction
resp, err := client.BroadcastPayForBlob(ctx, blobTx.Blobs)
if err != nil {
return nil, fmt.Errorf("failed to resubmit blob tx after expiry: %w", err)
}
fmt.Println("resubmitted blob tx", txHash, "new hash", resp.TxHash)

// confirm the resubmitted transaction using the NEW hash
confirmResp, err := client.ConfirmTx(ctx, resp.TxHash)
if err != nil {
return nil, fmt.Errorf("failed to confirm resubmitted blob tx: %w", err)
}

return confirmResp, nil

}
// This is a normal transaction, resubmit with messages
resp, err := client.BroadcastTxWithoutMutex(ctx, msgs)
if err != nil {
return nil, fmt.Errorf("failed to resubmit tx after expiry: %w", err)
}
fmt.Println("resubmitted tx", txHash, "new hash", resp.TxHash)

// confirm the resubmitted transaction using the NEW hash
confirmResp, err := client.ConfirmTx(ctx, resp.TxHash)
if err != nil {
return nil, fmt.Errorf("failed to confirm resubmitted tx: %w", err)
}

return confirmResp, nil
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we possible clean up this logic using the typical techniques?

I could be wrong, but looks like we avoid 4 layers nested if statements by exiting early with negative if statements and deduping logic

func TestRejections(t *testing.T) {
ttlNumBlocks := int64(5)
_, txClient, ctx := setupTxClient(t, ttlNumBlocks, appconsts.DefaultMaxBytes)
// t.Run("should reset the signer nonce after tx is rejected with no subsequent txs to resubmit", func(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why comment this out? can we delete if not needed?

@ninabarbakadze
Copy link
Member Author

apologies, reviewed then noticed it was a draft

No worries! It should be close to done now. I will need to look over it but feedback is welcome at this point.

Important to note: We can only merge this after we've released the recent cache changes we made on v0.38.x in core and bumped app

@ninabarbakadze ninabarbakadze removed warn:blocked item is not currently being worked on but is still blocked external labels Aug 6, 2025
@ninabarbakadze ninabarbakadze marked this pull request as ready for review August 6, 2025 20:09
@ninabarbakadze ninabarbakadze requested a review from rootulp as a code owner August 6, 2025 20:09
Comment on lines +313 to +329
latestHeight, err := client.getLatestBlockHeight(ctx)
if err != nil {
return nil, err
}
timeOutHeight := uint64(latestHeight) + client.ttlNumBlocks

txTimeoutHeight, err := GetTimeoutHeightFromOptions(client, opts...)
if err != nil {
return nil, err
}

// Do not override user set timeout height for the transaction
if txTimeoutHeight > 0 {
timeOutHeight = txTimeoutHeight
}

opts = append([]TxOption{SetGasLimit(gasLimit), SetFee(fee), SetTimeoutHeight(timeOutHeight)}, opts...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
latestHeight, err := client.getLatestBlockHeight(ctx)
if err != nil {
return nil, err
}
timeOutHeight := uint64(latestHeight) + client.ttlNumBlocks
txTimeoutHeight, err := GetTimeoutHeightFromOptions(client, opts...)
if err != nil {
return nil, err
}
// Do not override user set timeout height for the transaction
if txTimeoutHeight > 0 {
timeOutHeight = txTimeoutHeight
}
opts = append([]TxOption{SetGasLimit(gasLimit), SetFee(fee), SetTimeoutHeight(timeOutHeight)}, opts...)
latestHeight, err := client.getLatestBlockHeight(ctx)
if err != nil {
return nil, err
}
timeOutHeight := uint64(latestHeight) + client.ttlNumBlocks
opts = append([]TxOption{SetGasLimit(gasLimit), SetFee(fee), SetTimeoutHeight(timeOutHeight)}, opts...)

Since you're prepending it, if a user has it set, it will override it

Comment on lines 621 to 623
if err := client.signer.SetSequence(signer, networkSequence); err != nil {
return nil, fmt.Errorf("setting sequence: %w", err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work, if a transaction is rejected and there are several subsequent transactions, each of those subsequent ones will all get set with the same network sequence and thus only the first thereafter will be accepted, the rest will be immediately rejected in CheckTx

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥲🥲🥲🥲🥲🥲🥲🥲🥲🥲

I see it now the reason it worked in my tests was that i was confirming them 1 by 1

I'll work on better design

@ninabarbakadze ninabarbakadze marked this pull request as draft August 7, 2025 13:28
@ninabarbakadze
Copy link
Member Author

Putting my thoughts in this doc until I come up with a solution that can anticipate nonces and resign transactions in a way that won't get them rejected

https://www.notion.so/celestiaorg/Handle-rejections-in-ReCheck-2482a70c951580e2aee2c1a128728973?source=copy_link

@ninabarbakadze
Copy link
Member Author

Putting my thoughts in this doc until I come up with a solution that can anticipate nonces and resign transactions in a way that won't get them rejected

https://www.notion.so/celestiaorg/Handle-rejections-in-ReCheck-2482a70c951580e2aee2c1a128728973?source=copy_link

Currently prototyping this design

@ninabarbakadze
Copy link
Member Author

Designing a resubmission engine that consistently passes all pressure testing scenarios has proven non trivial. The current implementation is still hitting nonce errors. It’s difficult to pinpoint exactly which edge cases are being hit in these failures so further investigation is needed to identify and account for all scenarios before we can fully resolve them.

Another potential avenue to explore would be having a sort of a local mempool in the tx client that so that when we get nonce failures we will resubmit transactions that were rejected with that nonce and currently are waiting in local pool to be resubmitted.

Copy link
Member Author

@ninabarbakadze ninabarbakadze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trying to get it to pass the test that submits a bunch of transactions with low ttls and small mempool to ensure that tx client can handle different edge cases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replay rejected transactions in tx client

3 participants