enh(tmq): optimize logic of tmq of query

[wangmm0220]

Description

Issue(s)

• Close/close/Fix/fix/Resolve/resolve: Issue Link

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello @wangmm0220, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant optimizations to the TMQ (TAOS Message Queue) query logic, primarily focusing on enhancing data fetching efficiency and refining stream processing. It integrates a new timeout mechanism for data retrieval and provides more granular control over stream polling parameters. A major refactoring of the TMQ reader and stream aggregation components has been performed, simplifying internal state management and streamlining data handling. These changes collectively aim to improve the robustness, performance, and maintainability of TMQ queries.

Highlights

• TMQ Timeout Mechanism: A new timeout field has been introduced in TMQ response structures (SMqMetaRsp, SMqDataRsp, SMqBatchMetaRsp) and a dedicated error code TSDB_CODE_TMQ_FETCH_TIMEOUT added. This allows the system to explicitly signal when data fetching times out, with client-side logic updated to handle these timeouts by preventing emptyBlockReceiveTs from being updated.
• Enhanced Stream Polling Parameters: The stream polling mechanism has been upgraded by replacing qStreamSetSourceExcluded with qStreamSetParams. This new function provides more granular control over stream polling by accepting minPollRows and timeout parameters, enabling TMQ queries to fetch data until a minimum row count is met or a specified timeout elapses.
• Refactored TMQ Reader (STqReader): The STqReader structure has undergone significant changes, including replacing pVnodeMeta with pVnode, removing pColIdList and pResBlock, and adding pTSchema and pTableCacheForTmq. The tqNextBlockInWal function's signature was expanded to pass more context, such as the result block, column-to-slot ID mapping, pseudo expressions, minimum poll rows, and timeout, making it a more central and powerful component for data retrieval. Obsolete functions like tqGetResultBlock and tqReaderSetColIdList have been removed.
• Simplified Stream Aggregation and State Management: The SStateStore struct has been removed from SStorageAPI, and many related functions and structures (e.g., SStreamAggSupporter, SWindowSupporter, cleanupResultInfoInStream) have been either removed or refactored. This indicates a major simplification or relocation of stream aggregation and state persistence logic, aiming to reduce complexity within the executor.
• Optimized Column Data Retrieval: New internal functions (setBlockData, processSubmitRow, processSubmitCol, checkSchema, tqRetrieveCols, cacheTag, fillTag) have been introduced to efficiently retrieve and process column data. This includes improved handling of pseudo columns and caching of table tags, streamlining the construction and filtering of data blocks.
• Monitoring Hash Map Correction: In clientMonitor.c, the sendAllCounter function was corrected to iterate over monitorCounterHash instead of monitorSlowLogHash. This change addresses a potential bug and improves the accuracy of monitoring data.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[Copilot]

Pull request overview

This PR optimizes the TMQ (TDengine Message Queue) query logic by refactoring data fetching operations and introducing timeout support for fetch operations. The changes improve performance by batching data retrieval and using hash-based column mapping instead of array-based matching.

Changes:

• Added new error code TSDB_CODE_TMQ_FETCH_TIMEOUT to signal timeout conditions during data fetching
• Refactored TMQ scan logic to support batching multiple blocks until minPollRows is reached or timeout occurs
• Replaced column matching arrays with hash-based column-to-slot-id mapping for better performance
• Removed unused data structures and functions related to stream aggregation and notify events

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
source/util/src/terror.c	Added new error code definition for TMQ fetch timeout
include/util/taoserror.h	Declared TSDB_CODE_TMQ_FETCH_TIMEOUT error code
source/libs/executor/src/streamNotify.c	Removed unused notification event building functions
source/libs/executor/src/scanoperator.c	Refactored doQueueScanNext to batch data fetching with timeout support; replaced column matching with hash-based mapping
source/libs/executor/src/executor.c	Changed qStreamSetSourceExcluded to qStreamSetParams to accept additional parameters
source/libs/executor/src/executil.c	Added assertion to ensure column list is not empty
source/libs/executor/src/aggregateoperator.c	Removed unused cleanupResultInfoInStream function
source/libs/executor/inc/*.h	Updated function signatures and removed unused structure definitions
source/dnode/vnode/src/vnd/vnodeStream.c	Refactored tag processing functions to be reusable; made cacheTag and fillTag functions public
source/dnode/vnode/src/vnd/vnodeApi.c	Removed references to deprecated tqReader API functions
source/dnode/vnode/src/tq/tqScan.c	Updated to use new qStreamSetParams API and handle timeout return codes
source/dnode/vnode/src/tq/tqRead.c	Major refactoring: changed tqNextBlockInWal to batch data and support timeout; introduced hash-based column mapping; refactored data retrieval functions
source/dnode/vnode/src/tq/tq.c	Minor log message improvements
source/dnode/vnode/src/inc/vnodeInt.h	Added declarations for cacheTag and fillTag functions
source/dnode/vnode/inc/vnode.h	Updated STqReader structure and function signatures
source/common/src/msg/tmsg.c	Added timeout field encoding/decoding to response messages
include/common/tmsg.h	Added timeout field to SMqMetaRsp, SMqDataRsp, and SMqBatchMetaRsp
source/client/src/clientTmq.c	Updated timeout handling logic to distinguish between timeout and empty responses
source/client/src/clientMonitor.c	Fixed incorrect hash table reference
include/libs/executor/storageapi.h	Removed unused SStateStore structure and updated tqReader API signatures
include/libs/executor/executor.h	Updated qStreamSetParams function signature

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Code Review

This pull request introduces significant optimizations and refactoring to the TMQ query logic. Key changes include adding timeout and minimum row count parameters to control data fetching from TMQ, which should improve efficiency and responsiveness. The data retrieval pipeline in the executor and vnode layers has been substantially reworked to support this, moving from a block-by-block processing model to fetching larger, consolidated data chunks.

Additionally, there's a major refactoring that removes the SStateStore and related stream aggregation components from the generic executor library, simplifying its architecture.

The changes are extensive but appear well-structured. I've found one critical issue related to message serialization that needs to be addressed.