Promotion 2026-03-03 prod (#7823)

.github/PULL_REQUEST_TEMPLATE/anvilprod-hotfix.md

@@ -68,6 +68,8 @@ Linked issue: #0000
 - [ ] Deleted unreferenced indices in `hammerbox` <sub>or this PR does not remove catalogs or otherwise causes unreferenced indices in `hammerbox`</sub>
 - [ ] Started reindex in `hammerbox` <sub>or this PR is not labeled `reindex:anvilprod`</sub>
 - [ ] Checked for failures in `hammerbox` <sub>or this PR is not labeled `reindex:anvilprod`</sub>
+- [ ] Started mirroring in `hammerbox` <sub>or this PR is not labeled `mirror:anvilprod`</sub>
+- [ ] Checked for failures in `hammerbox` <sub>or this PR is not labeled `mirror:anvilprod`</sub>
 
 
 ### Operator (merge the branch)
@@ -106,9 +108,9 @@ Linked issue: #0000
 
 ### Operator (mirroring)
 
-- [ ] Started mirroring in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
-- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
-- [ ] Emptied mirror fail queue in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
+- [ ] Started mirroring in `anvilprod` <sub>or neither this PR nor a failed, prior promotion is labelled `mirror:anvilprod`</sub>
+- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `anvilprod` <sub>or neither this PR nor a failed, prior promotion is labelled `mirror:anvilprod`</sub>
+- [ ] Emptied mirror fail queue in `anvilprod` <sub>or neither this PR nor a failed, prior promotion is labelled `mirror:anvilprod`</sub>
 
 
 ### Operator

.github/PULL_REQUEST_TEMPLATE/anvilprod-promotion.md

@@ -28,6 +28,12 @@ Linked issue: #0000
 - [ ] This PR is labeled `reindex:partial` and its description documents the specific reindexing procedure for `anvilprod` <sub>or requires a full reindex or is not labeled`reindex:anvilprod`</sub>
 
 
+### Author (mirror)
+
+- [ ] This PR is labeled `mirror:anvilprod` <sub>or the changes introduced by it will not require mirroring of `anvilprod`</sub>
+- [ ] This PR is labeled `mirror:partial` and its description documents the specific mirroring procedure for `anvilprod` <sub>or requires a full mirroring or is not labeled`mirror:anvilprod`</sub>
+
+
 ### Author (upgrading deployments)
 
 - [ ] This PR is labeled `upgrade` <sub>or does not require upgrading deployments</sub>
@@ -88,6 +94,8 @@ Linked issue: #0000
 - [ ] Deleted unreferenced indices in `hammerbox` <sub>or this PR does not remove catalogs or otherwise causes unreferenced indices in `hammerbox`</sub>
 - [ ] Started reindex in `hammerbox` <sub>or this PR is not labeled `reindex:anvilprod`</sub>
 - [ ] Checked for failures in `hammerbox` <sub>or this PR is not labeled `reindex:anvilprod`</sub>
+- [ ] Started mirroring in `hammerbox` <sub>or this PR is not labeled `mirror:anvilprod`</sub>
+- [ ] Checked for failures in `hammerbox` <sub>or this PR is not labeled `mirror:anvilprod`</sub>
 
 
 ### Operator (merge the branch)
@@ -133,9 +141,9 @@ Promoted PRs in status *Done* do not need to be moved.
 
 ### Operator (mirroring)
 
-- [ ] Started mirroring in `anvilprod` <sub>or this PR does not require mirroring `anvilprod`</sub>
-- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `anvilprod` <sub>or this PR does not require mirroring `anvilprod`</sub>
-- [ ] Emptied mirror fail queue in `anvilprod` <sub>or this PR does not require mirroring `anvilprod`</sub>
+- [ ] Started mirroring in `anvilprod` <sub>or this PR is not labelled `mirror:anvilprod`</sub>
+- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `anvilprod` <sub>or this PR is not labelled `mirror:anvilprod`</sub>
+- [ ] Emptied mirror fail queue in `anvilprod` <sub>or this PR is not labelled `mirror:anvilprod`</sub>
 
 
 ### Operator

.github/PULL_REQUEST_TEMPLATE/backport.md

@@ -61,6 +61,10 @@ This is the PR template for backport PRs against `develop`.
 - [ ] Started reindex in `anvilbox` <sub>or this PR is not labeled `reindex:anvildev`</sub>
 - [ ] Checked for failures in `sandbox` <sub>or this PR is not labeled `reindex:dev`</sub>
 - [ ] Checked for failures in `anvilbox` <sub>or this PR is not labeled `reindex:anvildev`</sub>
+- [ ] Started mirroring in `sandbox` <sub>or this PR is not labeled `mirror:dev`</sub>
+- [ ] Started mirroring in `anvilbox` <sub>or this PR is not labeled `mirror:anvildev`</sub>
+- [ ] Checked for failures in `sandbox` <sub>or this PR is not labeled `mirror:dev`</sub>
+- [ ] Checked for failures in `anvilbox` <sub>or this PR is not labeled `mirror:anvildev`</sub>
 
 
 ### Operator (merge the branch)

.github/PULL_REQUEST_TEMPLATE/prod-hotfix.md

@@ -95,9 +95,9 @@ Linked issue: #0000
 
 ### Operator (mirroring)
 
-- [ ] Started mirroring in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
-- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
-- [ ] Emptied mirror fail queue in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
+- [ ] Started mirroring in `prod` <sub>or neither this PR nor a failed, prior promotion is labelled `mirror:prod`</sub>
+- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `prod` <sub>or neither this PR nor a failed, prior promotion is labelled `mirror:prod`</sub>
+- [ ] Emptied mirror fail queue in `prod` <sub>or neither this PR nor a failed, prior promotion is labelled `mirror:prod`</sub>
 
 
 ### Operator

.github/PULL_REQUEST_TEMPLATE/prod-promotion.md

@@ -28,6 +28,12 @@ Linked issue: #0000
 - [ ] This PR is labeled `reindex:partial` and its description documents the specific reindexing procedure for `prod` <sub>or requires a full reindex or is not labeled`reindex:prod`</sub>
 
 
+### Author (mirror)
+
+- [ ] This PR is labeled `mirror:prod` <sub>or the changes introduced by it will not require mirroring of `prod`</sub>
+- [ ] This PR is labeled `mirror:partial` and its description documents the specific mirroring procedure for `prod` <sub>or requires a full mirroring or is not labeled`mirror:prod`</sub>
+
+
 ### Author (upgrading deployments)
 
 - [ ] This PR is labeled `upgrade` <sub>or does not require upgrading deployments</sub>
@@ -122,9 +128,9 @@ Promoted PRs in status *Done* do not need to be moved.
 
 ### Operator (mirroring)
 
-- [ ] Started mirroring in `prod` <sub>or this PR does not require mirroring `prod`</sub>
-- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `prod` <sub>or this PR does not require mirroring `prod`</sub>
-- [ ] Emptied mirror fail queue in `prod` <sub>or this PR does not require mirroring `prod`</sub>
+- [ ] Started mirroring in `prod` <sub>or this PR is not labelled `mirror:prod`</sub>
+- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `prod` <sub>or this PR is not labelled `mirror:prod`</sub>
+- [ ] Emptied mirror fail queue in `prod` <sub>or this PR is not labelled `mirror:prod`</sub>
 
 
 ### Operator

.github/pull_request_template.md

@@ -47,6 +47,15 @@ title is `Fix: ` followed by the issue title
 - [ ] This PR is labeled `reindex:partial` and its description documents the specific reindexing procedure for `dev`, `anvildev`, `anvilprod` and `prod` <sub>or requires a full reindex or carries none of the labels `reindex:dev`, `reindex:anvildev`, `reindex:anvilprod` and `reindex:prod`</sub>
 
 
+### Author (mirror)
+
+- [ ] This PR is labeled `mirror:dev` <sub>or the changes introduced by it will not require mirroring of `dev`</sub>
+- [ ] This PR is labeled `mirror:anvildev` <sub>or the changes introduced by it will not require mirroring of `anvildev`</sub>
+- [ ] This PR is labeled `mirror:anvilprod` <sub>or the changes introduced by it will not require mirroring of `anvilprod`</sub>
+- [ ] This PR is labeled `mirror:prod` <sub>or the changes introduced by it will not require mirroring of `prod`</sub>
+- [ ] This PR is labeled `mirror:partial` and its description documents the specific mirroring procedure for `dev`, `anvildev`, `anvilprod` and `prod` <sub>or requires a full mirroring or carries none of the labels `mirror:dev`, `mirror:anvildev`, `mirror:anvilprod` and `mirror:prod`</sub>
+
+
 ### Author (API changes)
 
 - [ ] This PR and its linked issues are labeled `API` <sub>or this PR does not modify a REST API</sub>
@@ -110,6 +119,7 @@ Note that after requesting changes, the PR must be assigned to only the author.
 ### Operator
 
 - [ ] Checked `reindex:…` labels and `r` commit title tag
+- [ ] Checked `mirror:…` labels
 - [ ] Checked that demo expectations are clear <sub>or all linked issues are labeled `no demo`</sub>
 - [ ] Squashed PR branch and rebased onto `develop`
 - [ ] Sanity-checked history
@@ -154,6 +164,10 @@ Note that after requesting changes, the PR must be assigned to only the author.
 - [ ] Started reindex in `anvilbox` <sub>or this PR is not labeled `reindex:anvildev`</sub>
 - [ ] Checked for failures in `sandbox` <sub>or this PR is not labeled `reindex:dev`</sub>
 - [ ] Checked for failures in `anvilbox` <sub>or this PR is not labeled `reindex:anvildev`</sub>
+- [ ] Started mirroring in `sandbox` <sub>or this PR is not labeled `mirror:dev`</sub>
+- [ ] Started mirroring in `anvilbox` <sub>or this PR is not labeled `mirror:anvildev`</sub>
+- [ ] Checked for failures in `sandbox` <sub>or this PR is not labeled `mirror:dev`</sub>
+- [ ] Checked for failures in `anvilbox` <sub>or this PR is not labeled `mirror:anvildev`</sub>
 
 
 ### Operator (merge the branch)
@@ -207,18 +221,18 @@ Note that after requesting changes, the PR must be assigned to only the author.
 
 ### Operator (mirroring)
 
-- [ ] Started mirroring in `dev` <sub>or this PR does not require mirroring `dev`</sub>
-- [ ] Started mirroring in `anvildev` <sub>or this PR does not require mirroring `anvildev`</sub>
-- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `dev` <sub>or this PR does not require mirroring `dev`</sub>
-- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `anvildev` <sub>or this PR does not require mirroring `anvildev`</sub>
-- [ ] Emptied mirror fail queue in `dev` <sub>or this PR does not require mirroring `dev`</sub>
-- [ ] Emptied mirror fail queue in `anvildev` <sub>or this PR does not require mirroring `anvildev`</sub>
+- [ ] Started mirroring in `dev` <sub>or this PR is not labelled `mirror:dev`</sub>
+- [ ] Started mirroring in `anvildev` <sub>or this PR is not labelled `mirror:anvildev`</sub>
+- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `dev` <sub>or this PR is not labelled `mirror:dev`</sub>
+- [ ] Checked for, triaged and possibly requeued messages in mirror fail queue in `anvildev` <sub>or this PR is not labelled `mirror:anvildev`</sub>
+- [ ] Emptied mirror fail queue in `dev` <sub>or this PR is not labelled `mirror:dev`</sub>
+- [ ] Emptied mirror fail queue in `anvildev` <sub>or this PR is not labelled `mirror:anvildev`</sub>
 
 
 ### Operator
 
-- [ ] Propagated the `deploy:shared`, `deploy:gitlab`, `deploy:runner`, `API`, `reindex:partial`, `reindex:anvilprod` and `reindex:prod` labels to the next promotion PRs <sub>or this PR carries none of these labels</sub>
-- [ ] Propagated any specific instructions related to the `deploy:shared`, `deploy:gitlab`, `deploy:runner`, `API`, `reindex:partial`, `reindex:anvilprod` and `reindex:prod` labels, from the description of this PR to that of the next promotion PRs <sub>or this PR carries none of these labels</sub>
+- [ ] Propagated the `deploy:shared`, `deploy:gitlab`, `deploy:runner`, `API`, `reindex:partial`, `reindex:anvilprod`, `reindex:prod`, `mirror:partial`, `mirror:anvilprod` and `mirror:prod` labels to the next promotion PRs <sub>or this PR carries none of these labels</sub>
+- [ ] Propagated any specific instructions related to the `deploy:shared`, `deploy:gitlab`, `deploy:runner`, `API`, `reindex:partial`, `reindex:anvilprod`, `reindex:prod`, `mirror:partial`, `mirror:anvilprod` and `mirror:prod` labels, from the description of this PR to that of the next promotion PRs <sub>or this PR carries none of these labels</sub>
 - [ ] PR is assigned to no one
 
 

.github/pull_request_template.md.template.py

@@ -210,7 +210,9 @@ def labels_to_promote(self, target_branch: str) -> AbstractSet[str]:
             *iif(self is T.upgrade, ['backup:gitlab'], [
                 'API',
                 'reindex:partial',
-                *('reindex:' + d for d in self.downstream_deployments(target_branch))
+                *('reindex:' + d for d in self.downstream_deployments(target_branch)),
+                'mirror:partial',
+                *('mirror:' + d for d in self.downstream_deployments(target_branch)),
             ])
         ])
 
@@ -454,6 +456,38 @@ def emit(t: T, target_branch: str):
                         ])
                     )
                 },
+                {
+                    'type': 'h2',
+                    'content': 'Author (mirror)'
+                },
+                *[
+                    {
+                        'type': 'cli',
+                        'content': f'This PR is labeled `mirror:{d}`',
+                        'alt': f'or the changes introduced by it will not require mirroring of `{d}`'
+                    }
+                    for d in t.affected_deployments(target_branch)
+                ],
+                {
+                    'type': 'cli',
+                    'content': (
+                        'This PR is labeled `mirror:partial` and ' +
+                        'its description documents the specific mirroring procedure for ' +
+                        join_grammatically([
+                            f'`{d}`' for d in t.affected_deployments(target_branch)
+                        ])
+                    ),
+                    'alt': (
+                        'or requires a full mirroring ' +
+                        iif(len(t.affected_deployments(target_branch)) == 1,
+                            'or is not labeled',
+                            'or carries none of the labels ') +
+                        join_grammatically([
+                            f'`mirror:{d}`'
+                            for d in t.affected_deployments(target_branch)
+                        ])
+                    )
+                },
                 *iif(t is T.default, [
                     {
                         'type': 'h2',
@@ -714,6 +748,10 @@ def emit(t: T, target_branch: str):
                     'type': 'cli',
                     'content': 'Checked `reindex:…` labels and `r` commit title tag'
                 },
+                {
+                    'type': 'cli',
+                    'content': 'Checked `mirror:…` labels'
+                },
                 {
                     'type': 'cli',
                     'content': 'Checked that demo expectations are clear',
@@ -862,7 +900,17 @@ def emit(t: T, target_branch: str):
                             'type': 'cli',
                             'content': f'Checked for failures in `{s}`',
                             'alt': f'or this PR is not labeled `reindex:{d}`'
-                        }
+                        },
+                        {
+                            'type': 'cli',
+                            'content': f'Started mirroring in `{s}`',
+                            'alt': f'or this PR is not labeled `mirror:{d}`'
+                        },
+                        {
+                            'type': 'cli',
+                            'content': f'Checked for failures in `{s}`',
+                            'alt': f'or this PR is not labeled `mirror:{d}`'
+                        },
                     ])
                 ]
                 for i, (d, s) in enumerate(t.target_deployments(target_branch).items())
@@ -1089,9 +1137,9 @@ def emit(t: T, target_branch: str):
                                 'type': 'cli',
                                 'content': f'{action} in `{d}`',
                                 'alt': (
-                                    'or neither this PR nor a failed, prior promotion requires it'
+                                    f'or neither this PR nor a failed, prior promotion is labelled `mirror:{d}`'
                                     if t is T.hotfix else
-                                    f'or this PR does not require mirroring `{d}`'
+                                    f'or this PR is not labelled `mirror:{d}`'
                                 )
                             }
                             for action in [

.mypy.ini

@@ -46,20 +46,11 @@ modules =
     azul.threads,
     azul.time,
     azul.uuids,
-    azul.indexer,
     azul.plugins,
-    azul.indexer.document,
-    azul.indexer.field,
-    azul.indexer.action_controller,
-    azul.indexer.mirror_controller,
     azul.service.app_controller,
     azul.digests,
     lambdas.indexer.app,
     azul.schemas,
-    azul.indexer.index_controller,
-    azul.indexer.index_queue_service,
-    azul.indexer.index_repository_service,
-    azul.indexer.aggregate,
     test.azul_test_case,
     test.test_tagging,
     scripts.reindex,
@@ -85,13 +76,12 @@ modules =
     azul.plugins.metadata.hca,
     azul.plugins.metadata.anvil.service.filter,
     azul.plugins.metadata.hca.service.filter,
-    azul.indexer.transform,
-    azul.indexer.mirror_service,
     azul.functions,
     azul.service.storage_service,
     azul.service.source_service,
 
 packages =
+    azul.indexer,
     azul.openapi
 
 [mypy-furl.*]

deployments/anvilprod/environment.py

@@ -1252,6 +1252,7 @@ def union(previous_catalog: dict[DatasetName, SourceItem | None],
 anvil13_sources = union(anvil12_sources, 382, delta([
     source('1156aa70', 'ADOPT_PGx_Acute_Pain_GRU_R1_20260122_ANV5_202601222214'),
     source('48fde710', 'ADOPT_PGx_Acute_Pain_HMB_R1_20260122_ANV5_202601222221'),
+    source('6db4e098', 'CCDG_Baylor_CVD_ARIC_20231008_ANV5_202503171456', pop),
     source('f0697a02', 'CCDG_NYGC_NP_Autism_GASD_GRU_WGS_20260109_ANV5_202602021433'),
     source('0098f4fd', 'CCDG_NYGC_NP_Autism_SEARCHLIGHT_DS_WGS_20260109_ANV5_202601261720'),
     source('042b9f74', 'CCDG_NYGC_NP_Autism_SPARK_GRU_WGS_20260109_ANV5_202601261729'),
@@ -1263,6 +1264,7 @@ def union(previous_catalog: dict[DatasetName, SourceItem | None],
     source('6c5111aa', 'GTEx_v11_hg38_20260126_ANV5_202601261749'),
     source('0a4ee218', 'HPRC_20260121_ANV5_202601222125'),
     source('042719c0', 'IGVF_GRU_PUB_NPU_R1_20260122_ANV5_202601222237'),
+    source('c2f121d2', 'IGVF_GRU_R1_20260219_ANV5_202602191350'),
     source('4e167035', 'OurHealth_GRU_R2_20260122_ANV5_202601222246'),
 ]))
 
@@ -1307,7 +1309,6 @@ def env() -> Mapping[str, str | None]:
                                                     repository=dict(name='tdr_anvil')),
                                        sources=condense(sources))
             for atlas, catalog, sources, mirror_limit, it_mirror_limit in [
-                ('anvil', 'anvil12', anvil12_sources, None, int(1.5 * 1024 ** 3)),
                 ('anvil', 'anvil13', anvil13_sources, None, int(1.5 * 1024 ** 3)),
             ]
             for suffix, is_it in [