-
Notifications
You must be signed in to change notification settings - Fork 576
Updated Is "Migration Right for you?" page and some dependencies #10158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
df3f743
c62887a
b58adc4
3e364fa
38b6d73
27dd744
a365541
a67a2be
200024d
0a801a8
dcffac5
242887c
70e22e5
7178b11
45ceabf
6e475d6
1339b1b
b9c53f9
1b225ac
61321f1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,17 +9,56 @@ | |
|
||
# Is Migration Assistant right for you? | ||
|
||
Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your use case—or whether another tool might be a better fit. | ||
Whether Migration Assistant is right for you depends on your upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant fits your use case. | ||
|
||
Migration Assistant was built to fill important gaps in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. | ||
Migration Assistant addresses key limitations in traditional migration approaches. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant enables you to complete the process in a single step. Other methods, like rolling upgrades or snapshot restores, require upgrading through each major version, often with reindexing at every stage. | ||
|
||
Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for production environments, where minimizing service disruption is critical. | ||
Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical. | ||
|
||
If your migration is limited to a static cluster configuration (like index templates and aliases), or if you're not concerned about downtime, simpler tools may be sufficient. But for complex migrations involving real-time traffic or major version jumps, Migration Assistant offers robust, flexible capabilities. | ||
## Migration Assistant assumptions and limitations | ||
|
||
Before using Migration Assistant, review the following assumptions and limitations. They are grouped by scope to clarify which apply generally and which are specific to components. | ||
|
||
### General | ||
|
||
|
||
- If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). | ||
- The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. | ||
Check failure on line 26 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||
- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. | ||
- If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). | ||
- If deploying Migration Assistant into an existing AWS VPC, you must configure VPC interface endpoints and IAM permissions for the following AWS services: | ||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Amazon CloudWatch Logs** – for log ingestion from ECS tasks. | ||
- **Amazon CloudWatch** – for metrics published during migration | ||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Amazon Elastic Container Registry (ECR)** – for pulling container images. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is this enough detail for customers? there are two ECR endpoints https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html#ecr-setting-up-vpc-create There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a level of detail that should be included on another page. On this page I'd like customers to think about the access that they will need. |
||
- **Amazon Elastic Container Service (ECS)** – for task orchestration and migration. | ||
- **Elastic Load Balancing (ELB)** – for routing traffic to Capture Proxy. | ||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **AWS Secrets Manager** – for storing credentials. | ||
- **AWS Systems Manager Parameter Store** – for storing and accessing parameter values. | ||
- **AWS Systems Manager Session Manager** – for secure shell access to EC2 (i.e., bootstrap instance). | ||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Amazon EC2** – for launching the bootstrap instance responsible for build and deployment. | ||
- **Amazon Elastic Block Store (EBS)** – for disk storage during migration. | ||
- **Amazon Virtual Private Cloud (VPC)** – for private networking and VPC endpoints. | ||
- **AWS X-Ray** – for distributed tracing across components. | ||
- **Amazon Elastic File System (EFS)** – for persistent logging. | ||
|
||
### Reindex-from-Snapshot | ||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- The source cluster must have the Amazon S3 plugin installed. | ||
- Snapshots must include global cluster state (`include_global_state` is `true`). | ||
- Shards up to 80 GiB are supported by default. Larger shards can be configured, except in AWS GovCloud, where 80 GiB is the maximum supported size. | ||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Capture-and-Replay | ||
|
||
- The Traffic Capture Proxy must be deployed to intercept relevant client traffic. | ||
- Live capture is recommended only for environments with less than 4 TB/day of incoming traffic to the source cluster. | ||
- Automatically generated document IDs are not preserved for index requests. Clients must explicitly provide document IDs when indexing or updating documents. | ||
|
||
--- | ||
|
||
## Supported migration paths | ||
|
||
The following matrix shows which source versions can be directly migrated to which OpenSearch target versions: | ||
The matrix below shows which source versions can be directly migrated to which OpenSearch target versions: | ||
Check warning on line 60 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||
|
||
|
||
<!-- Migration matrix rendering logic retained --> | ||
{% comment %}First, collect all unique target versions{% endcomment %} | ||
|
@@ -34,7 +73,7 @@ | |
<table class="migration-matrix"> | ||
<thead> | ||
<tr> | ||
<th></th> | ||
<th>Source Version</th> | ||
Check failure on line 76 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||
{% for target in unique_targets %} | ||
<th>{{ target }}</th> | ||
{% endfor %} | ||
|
@@ -59,28 +98,17 @@ | |
**Source and target platforms** | ||
|
||
- Self-managed (on premises or hosted by a cloud provider) | ||
- Amazon OpenSearch Service | ||
|
||
|
||
**AWS Regions** | ||
- Amazon OpenSearch Service (OpenSearch Serverless Collections are not supported) | ||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Migration Assistant is supported in the following AWS Regions: | ||
**Supported AWS Regions** | ||
|
||
- US East (N. Virginia, Ohio) | ||
- US West (Oregon, N. California) | ||
- Europe (Frankfurt, Ireland, London) | ||
- Asia Pacific (Tokyo, Singapore, Sydney) | ||
- AWS GovCloud (US-East, US-West)[^1] | ||
Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the for the full list of supported regions. | ||
|
||
[^1]: In AWS GovCloud (US), `reindex-from-snapshot` (RFS) is limited to shard sizes of 80 GiB or smaller. | ||
## Supported features | ||
|
||
Before starting an upgrade or migration, consider the cluster feature to be included. The table below lists what can be migrated using Migration Assistant, whether it is currently supported, and recommendations for how to handle each component. | ||
Check warning on line 109 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||
|
||
|
||
## Supported components | ||
|
||
Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. | ||
|
||
| Component | Supported | Recommendations | | ||
| Feature | Supported | Recommendations | | ||
| :--- |:--- | :--- | | ||
| **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. | | ||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. | | ||
|
@@ -95,59 +123,19 @@ | |
|
||
[^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | ||
|
||
## Choosing your migration approach | ||
|
||
Use the following checklist to determine which Migration Assistant components best fit your use case. | ||
|
||
### Metadata migration | ||
|
||
Use [metadata migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) if: | ||
|
||
- You need to migrate while mitigating breaking changes between the source and target clusters, such as differences in mappings, settings, aliases, or component templates. | ||
- You want a relatively consistent configuration between the source and target clusters. | ||
|
||
### Backfill migration | ||
|
||
Use [backfill migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) if: | ||
|
||
- You need to move historical data without disrupting live traffic. | ||
- You want to backfill indexes from a specific point in time without impacting the source cluster. | ||
- You want to verify historical data in the target cluster before switching over. | ||
- You want to backfill using an existing or incremental snapshot. | ||
- You need the fastest backfill option that includes reindexing. | ||
- You want the ability to pause and resume migration. | ||
|
||
### RFS | ||
|
||
Use [RFS]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/getting-started-data-migration/) if: | ||
|
||
- You already use OpenSearch snapshots for backups. | ||
- You need to migrate documents at scale in parallel, such as with Amazon Elastic Container Service (Amazon ECS). | ||
- You require a data migration path as part of a zero-downtime migration. | ||
- Your AWS Region supports RFS and your shard sizes are within supported limits. | ||
|
||
### Combination of all three | ||
|
||
Use a combination of all three migration types if: | ||
|
||
- You're performing a complex, multi-version migration. | ||
- You require zero downtime and full validation of the target environment. | ||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- You want end-to-end tooling for metadata, data movement, and cluster behavior comparison. | ||
- You're cloning an existing cluster and changing the source's configuration. | ||
- You're setting up disaster recovery. | ||
|
||
## Checklist | ||
|
||
Use this checklist to decide whether Migration Assistant is right for you: | ||
|
||
- Are you migrating across one or more major versions? | ||
|
||
- Do you need to maintain service availability with zero downtime? | ||
|
||
- Do you need to validate a new OpenSearch cluster before switching over? | ||
Use this checklist to determine whether Migration Assistant is the right fit for your migration: | ||
|
||
- Are you migrating across one or more major versions—for example, from Elasticsearch 5 to OpenSearch 3—in a single step? | ||
- Are you upgrading but want the ability to safely back out, reducing the risk of data loss or service disruption? | ||
- Do you need to maintain high service availability with minimal or zero downtime? | ||
- Do you need to validate a new OpenSearch cluster before switching over, with rollback capabilities? | ||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- Is your environment self-managed or running on Amazon OpenSearch Service? | ||
|
||
- Are you looking for tooling that can automate metadata migration and performance comparison? | ||
- Are you looking for tooling to migrate index settings and other metadata? | ||
- Do you need to reconfigure your target cluster—for example, by changing the sharding strategy and reindexing? | ||
- Are you migrating across regions, from on-premises, or from another cloud provider? | ||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery? | ||
|
||
If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. |
Uh oh!
There was an error while loading. Please reload this page.