Improve dYdX v3 resilience and reliability

[SarunasSS]

Pull Request

NautilusTrader prioritizes correctness and reliability, please follow existing patterns for validation and testing.

• I have reviewed the CONTRIBUTING.md and followed the established practices

Summary

This PR significantly improves the resilience and reliability of the dYdX adapter's order execution by implementing robust gRPC error handling and automatic retry logic. The changes handle account sequence mismatch errors (codes 4, 32) and good-til-block errors (code 11) by automatically refreshing the wallet sequence from the blockchain and retrying failed order broadcasts.

Related Issues/PRs

None

Type of change

• Bug fix (non-breaking)
• Improvement (non-breaking)
• Breaking change (impacts existing behavior)
• Documentation update
• Maintenance / chore

Breaking change details (if applicable)

N/A - All changes are backward compatible improvements to error handling and resilience.

Documentation

• Documentation changes follow the style guide (docs/developer_guide/docs.md)

No documentation changes required - internal implementation improvements only

Release notes

• I added a concise entry to RELEASES.md that follows the existing conventions (when applicable)

Suggested RELEASES.md entry:

- Improved dYdX adapter gRPC error handling with automatic retry logic for sequence mismatch and good-til-block errors
- Enhanced dYdX order cancellation tracking for BEST_EFFORT_CANCELED status

Testing

Ensure new or changed logic is covered by tests.

• Affected code paths are already covered by the test suite
• I added/updated tests to cover new or changed logic

Testing details:

• Error handling logic follows existing retry manager patterns used throughout the codebase
• Order cancellation tracking leverages existing order state management
• Manual testing performed with live dYdX integration to verify sequence mismatch recovery and position reconciliation

---

Key Changes Summary

1. Enhanced gRPC Error Handling

Changes:

• Added GOOD_TILL_BLOCK_ERROR_CODE (11) constant for good-til-block errors
• Expanded ACCOUNT_SEQUENCE_MISMATCH_ERROR_CODES to tuple (4, 32) to handle both sequence error codes
• Created DYDXOrderBroadcastError exception class for dYdX-specific broadcast errors
• Updated DYDX_RETRY_ERRORS_GRPC to include all retryable error codes
• Implemented automatic wallet sequence refresh from blockchain on sequence mismatch errors
• Added detailed error logging with diagnostic information

2. Order Cancellation Tracking

Changes:

• Added _track_order_cancel() method to track BEST_EFFORT_CANCELED orders
• Monitors blockchain height to determine when good-til-block threshold reached
• Automatically generates OrderCanceled event when block height exceeds good-til-block
• Configurable timeout (60 seconds) and check interval (0.1 seconds)
• Handles edge cases where order status changes before block threshold

4. Order Processing Improvements

Changes:

• Fixed race condition where fill messages arrive before order acceptance
• Generates inferred OrderAccepted event when fill received for submitted order
• Reordered WebSocket message processing to handle orders before fills
• Added proper cache management for venue order IDs
• Improved order status tracking and state transitions
• Enhanced logging for debugging order lifecycle issues

5. Retry and Broadcast Logic Refactoring

Files modified:

• nautilus_trader/adapters/dydx/execution.py

Changes:

• Extracted _broadcast_order() method to handle retry logic separately
• Created _attempt_order_broadcast() for single broadcast attempt
• Uses DYDXOrderBroadcastError for consistent error handling
• Ensures block height is refreshed before placing orders
• Simplified error propagation through exception-based flow

Migration Guide

No migration required - all changes are backward compatible. The dYdX adapter will automatically:

• Retry orders on sequence mismatch errors
• Track position reconciliation in the background
• Handle order cancellations more gracefully

Users may see additional log messages for:

• Sequence mismatch recovery events
• Position reconciliation checks
• Order cancellation tracking

[cjdsellers]

Hi @SarunasSS

Thanks for the contribution!

Looks like the pre-commit failed, but this is easy to fix. Just run the following and push any changes to the same PR:

make pre-commit

or (if you're on Windows):

pre-commit run --all-files

Also consider running the following to install the git hook which will run the pre-commit automatically on commit:

pre-commit install

[cjdsellers]

Thanks @SarunasSS, great work here!

[cjdsellers]

We should convert these three prints to self._logs if we'd like to keep them?

[SarunasSS]

They prob should be there at debug level, however there is no log instance in the DYDXAccountGRPCAPI. Feels like there should be one since its a pretty important component.

What should be the proper way to get one there? Pass during construction of it?

[cjdsellers]

I don't think we need additional reconciliation in the adapter itself? we should normalize the events and reports and send these back to the live execution engine.

[cjdsellers]

Should we add a config option for this?

[SarunasSS]

yeah does make sense