too_complex_to_determinize_exception is thrown in a few cases due to the patterns in index template

[pawankartik-elastic]

Elasticsearch Version

9.0.0 ECH

Installed Plugins

No response

Java Version

bundled

OS Version

Elastic Cloud

Problem Description

After an upgrade to Elastic v9.0.0, a user reported that they were encountering too_complex_to_determinize_exception error for the index patterns in PUT _index_template/logs-production.

At the moment, we've communicated a workaround: either simplify the patterns or split the patterns across 2 templates. However, we'll need a proper fix for this.

Looking at the commit history, this error is coming from the changes introduced in Lucene 10 upgrade: #114741, i.e. due to Operations.determinize(Operations.union(automata), Operations.DEFAULT_DETERMINIZE_WORK_LIMIT) in Regex#simpleMatchToAutomaton().

Link to the SDH: https://github.com/elastic/sdh-elasticsearch/issues/8947.

Steps to Reproduce

Here's an example:

PUT _index_template/logs-production
{
  "index_patterns": [
                    "*haproxy-production*",
                    "*postgres-production*",
                    "*psql_slow_query-production*",
                    "*springboot-production*",
                    "*tomcat-production*",
                    "*security-security*",
                    "*alerts-monitoring*"
  ],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "logs-production"
        },
        "codec": "best_compression",
        "query": {
          "default_field": [
            "message"
          ]
        }
      }
    }
  },
  "composed_of": [
    "logs-mappings",
    "data-streams-mappings"
  ]

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)