Rollback unsuccessful preconfs in the mempool

[Voxelot]

When the mempool receives a pre-confirmation, it optimistically marks the inputs as "spent" in it's local cache.

Currently there is no mechanism to rollback failed preconfs in the mempool, apart from a full restart of the node. This means when a preconfirmation doesn't finalize, those inputs remain "stuck", and users are prevented from using those inputs until the mempool is reset.

This has been observed nearly anytime the block producer is restarted, we find there are stuck utxos in the RPC mempools.

1. We need to investigate why restarting the block producer causes so many dropped transactions from a pre-conf perspective. Ideally if there is graceful shutdown, the in-flight blocks / transactions get finalized before the process exits.
2. We need a mechanism to automatically clear these cached utxos in the mempool without manual intervention (i.e. rebooting the mempool), either via TTL or some other reconciliation process that detects if a transaction did not finalize. Ie. any cached preconf data older than 1 block ago should be discarded.

[MitchTurner]

I thought that the inputs would be marked as some sort of intermediate state.

We have both respresented in a lot of our Tx status types:

pub enum TransactionStatus {
    SubmittedStatus(SubmittedStatus),
    SuccessStatus(SuccessStatus),
    PreconfirmationSuccessStatus(PreconfirmationSuccessStatus),
    SqueezedOutStatus(SqueezedOutStatus),
    FailureStatus(FailureStatus),
    PreconfirmationFailureStatus(PreconfirmationFailureStatus),
}