[BUG] Transform Job to continuously rollup data prior data is not working

[MahendraAkkina]

What is the bug?
Transform job to continuously roll up data that is at least 1 hour old is not working as expected.

I created transform configuration with continuous: true, schedule, and data_selection_query successfully processes data that’s at least 1 hour old the first time it runs, but subsequent runs (e.g., every 5 minutes) aren’t transforming new data, even though new documents are appearing in the source index with metric_timestamp values that are now more than 1 hour old.

Created transform job as follows:

PUT _plugins/_transform/hourly_rollup_transform

{
    "transform": {
        "enabled": true,
        "continuous": true,
        "schedule": {
            "interval": {
                "period": 5,
                "unit": "Minutes"
            }
        },
        "description": "Sample transform job",
        "source_index": "source-index-name",
        "target_index": "hourly-rollup-target",
        "data_selection_query": {
            "range": {
                "metric_timestamp": {
                    "lte": "now-60m"
                }
            }
        },
        "page_size": 10,
        "groups": [
            {
                "date_histogram": {
                    "fixed_interval": "60m",
                    "source_field": "metric_timestamp",
                    "target_field": "metric_timestamp",
                    "timezone": "UTC",
                    "format": null
                }
            },
            {
                "terms": {
                    "source_field": "device.keyword",
                    "target_field": "device"
                }
            }
        ],
        "aggregations": {
            "m1": {
                "avg": {
                    "field": "m1"
                }
            },
        }
    }
}

What is your host/environment?
"version": {
"distribution": "opensearch",
"number": "2.11.0",
"build_type": "tar",
"build_hash": "4dcad6dd1fd45b6bd91f041a041829c8687278fa",
"build_date": "2023-10-13T02:55:55.511945994Z",
"build_snapshot": false,
"lucene_version": "9.7.0",
"minimum_wire_compatibility_version": "7.10.0",
"minimum_index_compatibility_version": "7.0.0"
}

[krisfreedain]

Catch All Triage - 1 2

[MahendraAkkina]

@krisfreedain Thanks for bring this to the radar.

This is a major issue for us as we have an use-case where for example, we want to transform only the data that is older than 30 days on a daily basis.

I suspect it has to do with what continuous_stats's last_timestamp is set to. Let me illustrate with an example. It will be helpful if someone with expertise in transform use-case can comment

Current Date: Let’s assume today is April 21, 2025, at 09:31 AM PDT.

Data: I have 45 days of data, from March 7, 2025 (45 days ago) to April 21, 2025 (today).

Transform Job setup:

Continuous: true (the transform runs incrementally, processing only new or updated data since the last checkpoint).

Schedule: Runs once a day.

Data Query Filter: timestamp <= "now-30d" (selects documents with a timestamp older than 30 days from the current time).

Expectation: For 45 days of data, the first run transforms data from days 31 to 45 (older than 30 days). Each subsequent daily run transforms only the data that newly falls into the "older than 30 days" window (e.g., one additional day’s worth of data).

Results:

First Run (April 21, 2025):
Filter: timestamp <= "now-30d" (March 22, 2025).

Working as Expected: Processes March 7 to March 22, 2025.

continuous_stats's last_timestamp is set to April 21, 2025 (execution time). Should it be set to March 22, 2025??

Second Run (April 22, 2025):
Filter: timestamp <= "now-30d" (March 23, 2025).

Result: March 23, 2025, is skipped, and the transform processes no data.

What seems to be happening: The transform assumes it has processed all data up to April 21, 2025, and looks for new data after this timestamp. Since new data (e.g., April 21 or April 22, 2025) is outside the timestamp <= March 23, 2025 filter, no data is processed.

Subsequent Runs: This continues, with last_timestamp updating to each run’s execution time (e.g., April 22, April 23), preventing the transform from processing older data like March 24, March 25, etc.

[krisfreedain]

@vikasvb90 @bowenlan-amzn @Hailong-am @tandonks @shiv0408 @soosinha - can you take a look at this for @MahendraAkkina? thank you