Adding _source and schema merging to index_mappings

[ahkcs]

Description

Adding _source and schema merging to index_mappings so that customers can customize customize the index mapping for the OpenSearch index that stores the Flint index data created via Spark SQL.
This PR added the _source feature so that user can customize whether they want to enable _source in index_mappings
For example:

index_mappings: '{ "_source": { "enabled": false }'

This PR also added the schema merging feature so that user can Disable doc_values for specific fields to save space.

For exmaple:

"properties": { "test_field": {"index": false} } }

CREATE MATERIALIZED VIEW mv_test
AS
SELECT ... 
WITH (
  index_mappings: '{ "_source": { "enabled": false }, "properties": { "test_field": {"index": false} } }'
)

Test Results

Before disabling _source:
Input:

GET flint_mys3_default_mv_event_count2/_search

Output:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "flint_mys3_default_mv_event_count2",
        "_id": "g1ZLJ5YB8KjDxFt7EYUZ",
        "_score": 1,
        "_source": {
          "event": "click",
          "cnt": 10000
        }
      }
    ]
  }
}

After disabling _source:
Input:

GET flint_mys3_default_mv_event_count3/_search 

Output:

{
  "took": 510,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "flint_mys3_default_mv_event_count3",
        "_id": "h1ZfJ5YB8KjDxFt7D4X5",
        "_score": 1
      }
    ]
  }
}

Before adding index_mappings: '{ "_source": { "enabled": false }, "properties": { "cnt": {"index": false} } }'

Input:

GET flint_mys3_default_mv_event_count2/_mappings

Output:

{
  "flint_mys3_default_mv_event_count2": {
    "mappings": {
      "_meta": {
        "kind": "mv",
        "indexedColumns": [
          {
            "columnType": "string",
            "columnName": "event"
          },
          {
            "columnType": "bigint",
            "columnName": "cnt"
          }
        ],
        "name": "mys3.default.mv_event_count2",
        "options": {
          "auto_refresh": "false",
          "incremental_refresh": "false"
        },
        "source": "SELECT event, COUNT(*) AS cnt FROM default.test_table GROUP BY event",
        "version": "1.0.0",
        "properties": {
          "sourceTables": [
            "mys3.default.test_table"
          ],
        }
      },
      "properties": {
        "cnt": {
          "type": "long"
        },
        "event": {
          "type": "keyword"
        }
      }
    }
  }
}

After

Input:

GET flint_mys3_default_mv_event_count4/_mappings

Output:

{
  "flint_mys3_default_mv_event_count4": {
    "mappings": {
      "_meta": {
        "kind": "mv",
        "indexedColumns": [
          {
            "columnType": "string",
            "columnName": "event"
          },
          {
            "columnType": "bigint",
            "columnName": "cnt"
          }
        ],
        "name": "mys3.default.mv_event_count4",
        "options": {
          "auto_refresh": "false",
          "incremental_refresh": "false",
          "index_mappings": "{ \"_source\": { \"enabled\": false }, \"properties\": { \"cnt\": {\"index\": false} } }"
        },
        "source": "SELECT event, COUNT(*) AS cnt FROM default.test_table GROUP BY event",
        "version": "1.0.0",
        "properties": {
          "sourceTables": [
            "mys3.default.test_table"
          ],
        }
      },
      "_source": {
        "enabled": false
      },
      "properties": {
        "cnt": {
          "type": "long",
          "index": false
        },
        "event": {
          "type": "keyword"
        }
      }
    }
  }
}

Related Issues

Resolves #772
Pending implementations for Support index mapping option in create index statement #772

1. Force certain field types (e.g., IP type) as a temporary workaround until SparkSQL supports them.

[dai-chen]

Please remove spark-warehouse. I think it's generated by Spark test and normally should be removed after test complete automatically.

[dai-chen]

Thanks for the changes!

I recall current schema merge logic only supports limited field name, right? Could you create follow-up issue for improvements or updating doc with this limitation?

[noCharger]

@ahkcs please check the e2e test failing