feat(sv-consumer): Introduce concurrent tasks configuration #483
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces several changes across multiple files to improve concurrency management, database connection pooling, and message broker setup. The most significant changes include making the number of concurrent tasks configurable, revising database pool size and connection settings, and enhancing the message broker setup process.
Concurrency Management Updates:
services/consumer/src/cli.rs
: Added a newconcurrent_tasks
configuration option to allow specifying the maximum number of concurrent tasks via the CLI.services/consumer/src/executor/block_executor.rs
: Replaced the hardcodedMAX_CONCURRENT_TASKS
constant with a dynamicconcurrent_tasks
field, enabling concurrency settings to be passed duringBlockExecutor
initialization. Updated task processing logic to use this new field. [1] [2] [3] [4] [5] [6]Database Connection Pooling:
crates/domains/src/infra/db/db_impl.rs
: Reduced the default database pool size from2000
to110
and added a minimum connection limit of2
for the pool. [1] [2]services/consumer/src/main.rs
: Updated thesetup_db
function to dynamically set the database pool size based on theconcurrent_tasks
value.Message Broker Enhancements:
crates/message-broker/src/nats.rs
: Added a newsetup_with_opts
method to allow configuring message broker options, such as acknowledgment wait times.services/consumer/src/main.rs
: Integrated thesetup_with_opts
method to configure the message broker with a 2-minute acknowledgment wait time.Publisher Service Improvements:
services/publisher/src/history.rs
: Increased the chunk size for block processing from50
to100
and added aJoinSet
for managing concurrent publishing tasks, improving efficiency and logging. [1] [2]Test Updates:
tests/tests/services/consumer.rs
: Updated theBlockExecutor
initialization in tests to include the newconcurrent_tasks
parameter.