Skip to content

Commit 22bb52e

Browse files
AlexCK-STFCAlexCK-STFC
authored
Merge pull request #23 from stfc/aggregate
ENH: Add aggregate query type
2 parents 4767a06 + 36cfaf5 commit 22bb52e

15 files changed

Lines changed: 570 additions & 12 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ Feature References
1111
- [Flavors](docs/user_docs/query_docs/FLAVORS.md)
1212
- [Projects](docs/user_docs/query_docs/PROJECTS.md)
1313
- [Hypervisors](docs/user_docs/query_docs/HYPERVISORS.md)
14+
- [Aggregates](docs/user_docs/query_docs/AGGREGATES.md)
15+
- [Images](docs/user_docs/IMAGES.md)
1416

1517
Developer-related Docs:
1618
- [Developer README](docs/developer_docs/OVERVIEW.md)

‎docs/user_docs/API.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,9 @@ The query library currently supports queries on the following Openstack resource
1919
| [Users](https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=list-users-detail#users) | Run a Query on Openstack Users | [USERS.md](query_docs/USERS.md) | `from openstackquery import UserQuery` |
2020
| [Project](https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=list-users-detail#projects) | Run a Query on Openstack Projects | [PROJECTS.md](query_docs/PROJECTS.md) | `from openstackquery import ProjectQuery` |
2121
| [Flavor](https://docs.openstack.org/api-ref/compute/#flavors) | Run a Query on Openstack Flavors | [FLAVORS.md](query_docs/FLAVORS.md) | `from openstackquery import FlavorQuery` |
22-
| [Hypervisor](https://docs.openstack.org/api-ref/compute/#hypervisors-os-hypervisors) | Run a Query on Openstack Hypervisors | [HYPERVISORS](query_docs/HYPERVISORS.md) | `from openstackquery import HypervisorQuery` |
22+
| [Hypervisor](https://docs.openstack.org/api-ref/compute/#hypervisors-os-hypervisors) | Run a Query on Openstack Hypervisors | [HYPERVISORS.md](query_docs/HYPERVISORS.md) | `from openstackquery import HypervisorQuery` |
23+
| [Aggregate](https://docs.openstack.org/api-ref/compute/#host-aggregates-os-aggregates) | Run a Query on OpenStack Aggregates | [AGGREGATES.md](query_docs/AGGREGATES.md) | `from openstackquery import AggregateQuery` |
24+
| [Image](https://docs.openstack.org/api-ref/image/v2/index.html) | Run a Query on OpenStack Images | [IMAGES.md](query_docs/IMAGES.md) | `from openstackquery import ImageQuery` |
2325

2426
#
2527
### select

‎docs/user_docs/query_docs/AGGREGATES.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Aggregates
2+
3+
Aggregates refer to **Host Aggregates** in OpenStack. These are logical groupings of compute hosts with shared metadata. They're typically used for scheduling decisions (e.g., grouping GPU hosts or separating workloads).
4+
5+
See [OpenStack Docs](https://docs.openstack.org/api-ref/compute/#host-aggregates-os-aggregates) for more info.
6+
7+
**NOTE: `AggregateQuery` will only work with admin credentials - set by `clouds.yaml`**
8+
9+
---
10+
11+
## Querying
12+
13+
To query for Aggregates using the Query Library, you can import `AggregateQuery()` like so:
14+
15+
```
16+
from openstackquery import AggregateQuery
17+
```
18+
19+
`AggregateQuery()` can then be used to setup and run queries - see [API.md](../API.md) for details on API calls.
20+
21+
---
22+
23+
## Properties
24+
25+
Each `Aggregate` has the following properties:
26+
27+
| Return Type | Property Name(s) (case-insensitive)                     | Description                                                                       |
28+
|-------------|----------------------------------------------------------|-----------------------------------------------------------------------------------|
29+
| `string`    | "created_at"                                             | Timestamp the aggregate was created. Format: ISO8601                              |
30+
| `bool`      | "deleted"                                                | Boolean indicating whether the aggregate has been deleted                         |
31+
| `string`    | "deleted_at"                                             | Timestamp the aggregate was deleted, if applicable. Format: ISO8601               |
32+
| `int`       | "metadata_gpunum", "gpunum"                              | Number of GPUs associated with this aggregate (from metadata key `gpunum`)        |
33+
| `string`    | "hosts", "host_ips"                                      | List of hosts in the aggregate (as a JSON-formatted string)                       |
34+
| `string`    | "metadata_hosttype", "hosttype"                          | Host type value (from metadata key `hosttype`)                                    |
35+
| `string` | "metadata_local_storage_type", "local_storage_type" | Local storage type (from metadata key `local-storage-type`) |
36+
| `string`    | "metadata"                                               | Full metadata dictionary (as JSON-formatted string)                               |
37+
| `string`    | "updated_at"                                             | Timestamp the aggregate was last updated. Format: ISO8601                         |
38+
| `string`    | "id", "uuid"                                             | Unique ID of the aggregate (UUID)                                                 |
39+
40+
Any of these properties can be used for any of the API methods that take a property - like `select`, `where`, `sort_by`, etc.
41+
42+
---
43+
44+
## Chaining
45+
46+
This section details valid mappings you can use to chain onto other queries or from other queries to chain into an `AggregateQuery` object.
47+
48+
This applies to API calls `then` and `append_from` - see [API.md](../API.md) for details.
49+
50+
---
51+
52+
## Query Alias
53+
54+
The aliases that can be used for the query when chaining are listed below:
55+
56+
| Aliases (case-insensitive)                                     |
57+
|----------------------------------------------------------------|
58+
| "aggregate", "aggregates", "aggregate_query", "aggregatequery" |
59+
60+
---
61+
62+
## Chaining from
63+
64+
(Currently, no query chaining *from* `AggregateQuery` is defined.)
65+
66+
---
67+
68+
## Chaining to
69+
70+
(Currently, no query chaining *to* `AggregateQuery` is defined.)
71+
72+
---
73+
74+
## run() meta-parameters
75+
76+
`AggregateQuery()` accepts no extra meta-parameters when calling `run()`
77+
78+
---

‎openstackquery/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
ProjectQuery,
88
ImageQuery,
99
HypervisorQuery,
10+
AggregateQuery,
1011
)
1112

1213
# Create logger

‎openstackquery/api/query_objects.py‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
from typing import TYPE_CHECKING
2-
from typing import Type
1+
from typing import TYPE_CHECKING, Type
32

3+
from openstackquery.mappings.aggregate_mapping import AggregateMapping
44
from openstackquery.mappings.flavor_mapping import FlavorMapping
55
from openstackquery.mappings.hypervisor_mapping import HypervisorMapping
66
from openstackquery.mappings.image_mapping import ImageMapping
@@ -70,3 +70,10 @@ def HypervisorQuery() -> "QueryAPI":
7070
Simple helper function to setup a query using a factory
7171
"""
7272
return get_common(HypervisorMapping)
73+
74+
75+
def AggregateQuery() -> "QueryAPI":
76+
"""
77+
Simple helper function to setup a query using a factory
78+
"""
79+
return get_common(AggregateMapping)

‎openstackquery/enums/props/aggregate_properties.py‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
import json
2+
from enum import auto
3+
from typing import Dict, Optional
4+
5+
from openstackquery.enums.props.prop_enum import PropEnum, PropFunc
6+
from openstackquery.exceptions.query_property_mapping_error import (
7+
QueryPropertyMappingError,
8+
)
9+
10+
11+
class AggregateProperties(PropEnum):
12+
"""
13+
An enum class for Host Aggregate Properties
14+
"""
15+
16+
AGGREGATE_CREATED_AT = auto()
17+
AGGREGATE_DELETED = auto()
18+
AGGREGATE_DELETED_AT = auto()
19+
AGGREGATE_GPUNUM = auto()
20+
AGGREGATE_HOST_IPS = auto()
21+
AGGREGATE_HOSTTYPE = auto()
22+
AGGREGATE_LOCAL_STORAGE_TYPE = auto()
23+
AGGREGATE_METADATA = auto()
24+
AGGREGATE_UPDATED_AT = auto()
25+
AGGREGATE_ID = auto()
26+
27+
@staticmethod
28+
def _get_aliases() -> Dict:
29+
"""
30+
a method that returns all valid string alias mappings
31+
"""
32+
return {
33+
AggregateProperties.AGGREGATE_CREATED_AT: ["created_at"],
34+
AggregateProperties.AGGREGATE_DELETED: ["deleted"],
35+
AggregateProperties.AGGREGATE_DELETED_AT: ["deleted_at"],
36+
AggregateProperties.AGGREGATE_GPUNUM: ["metadata_gpunum", "gpunum"],
37+
AggregateProperties.AGGREGATE_HOST_IPS: ["hosts", "host_ips"],
38+
AggregateProperties.AGGREGATE_HOSTTYPE: ["metadata_hosttype", "hosttype"],
39+
AggregateProperties.AGGREGATE_LOCAL_STORAGE_TYPE: [
40+
"metadata_local_storage_type",
41+
"local_storage_type",
42+
],
43+
AggregateProperties.AGGREGATE_METADATA: ["metadata"],
44+
AggregateProperties.AGGREGATE_UPDATED_AT: ["updated_at"],
45+
AggregateProperties.AGGREGATE_ID: ["id", "uuid"],
46+
}
47+
48+
@staticmethod
49+
def get_prop_mapping(prop) -> Optional[PropFunc]:
50+
"""
51+
Method that returns the property function if function mapping exists for a given AggregateProperty Enum
52+
how to get specified property from an openstacksdk Aggregate object is documented here:
53+
https://docs.openstack.org/openstacksdk/latest/user/resources/compute/v2/aggregate.html
54+
:param prop: A Aggregate Enum for which a function may exist for
55+
"""
56+
mapping = {
57+
AggregateProperties.AGGREGATE_CREATED_AT: lambda a: a["created_at"],
58+
AggregateProperties.AGGREGATE_DELETED: lambda a: a["deleted"],
59+
AggregateProperties.AGGREGATE_DELETED_AT: lambda a: a["deleted_at"],
60+
AggregateProperties.AGGREGATE_GPUNUM: lambda a: int(
61+
a["metadata"].get("gpunum", 0)
62+
),
63+
AggregateProperties.AGGREGATE_HOST_IPS: lambda a: json.dumps(a["hosts"]),
64+
AggregateProperties.AGGREGATE_HOSTTYPE: lambda a: a["metadata"].get(
65+
"hosttype", None
66+
),
67+
AggregateProperties.AGGREGATE_LOCAL_STORAGE_TYPE: lambda a: a[
68+
"metadata"
69+
].get("local-storage-type", None),
70+
AggregateProperties.AGGREGATE_METADATA: lambda a: json.dumps(a["metadata"]),
71+
AggregateProperties.AGGREGATE_UPDATED_AT: lambda a: a["updated_at"],
72+
AggregateProperties.AGGREGATE_ID: lambda a: a["uuid"],
73+
}
74+
try:
75+
return mapping[prop]
76+
except KeyError as exp:
77+
raise QueryPropertyMappingError(
78+
f"Error: failed to get property mapping, property {prop.name} is not supported in AggregateProperties"
79+
) from exp
80+
81+
@staticmethod
82+
def get_marker_prop_func():
83+
"""
84+
A getter method to return marker property function for pagination
85+
"""
86+
return AggregateProperties.get_prop_mapping(AggregateProperties.AGGREGATE_ID)

‎openstackquery/enums/query_types.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
from typing import Dict
22

33
from openstackquery.enums.enum_with_aliases import EnumWithAliases
4-
4+
from openstackquery.mappings.aggregate_mapping import AggregateMapping
55
from openstackquery.mappings.flavor_mapping import FlavorMapping
6+
from openstackquery.mappings.hypervisor_mapping import HypervisorMapping
67
from openstackquery.mappings.image_mapping import ImageMapping
78
from openstackquery.mappings.project_mapping import ProjectMapping
89
from openstackquery.mappings.server_mapping import ServerMapping
910
from openstackquery.mappings.user_mapping import UserMapping
10-
from openstackquery.mappings.hypervisor_mapping import HypervisorMapping
1111

1212
# pylint: disable=too-few-public-methods
1313

@@ -24,6 +24,7 @@ class QueryTypes(EnumWithAliases):
2424
USER_QUERY = UserMapping
2525
IMAGE_QUERY = ImageMapping
2626
HYPERVISOR_QUERY = HypervisorMapping
27+
AGGREGATE_QUERY = AggregateMapping
2728

2829
@staticmethod
2930
def _get_aliases() -> Dict:
@@ -61,4 +62,5 @@ def _get_aliases() -> Dict:
6162
"hypervisors",
6263
"hypervisorquery",
6364
],
65+
QueryTypes.AGGREGATE_QUERY: ["aggregate", "aggregates", "aggregatequery"],
6466
}

‎openstackquery/mappings/aggregate_mapping.py‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
from typing import Type
2+
3+
from openstackquery.aliases import QueryChainMappings
4+
from openstackquery.enums.props.aggregate_properties import AggregateProperties
5+
from openstackquery.enums.query_presets import QueryPresets
6+
from openstackquery.handlers.client_side_handler import ClientSideHandler
7+
from openstackquery.handlers.server_side_handler import ServerSideHandler
8+
from openstackquery.mappings.mapping_interface import MappingInterface
9+
from openstackquery.runners.aggregate_runner import AggregateRunner
10+
11+
12+
class AggregateMapping(MappingInterface):
13+
"""
14+
Mapping class for querying Openstack Aggregate objects
15+
Define property mappings, kwarg mappings and filter function mappings,
16+
and runner mapping related to aggregates here
17+
"""
18+
19+
@staticmethod
20+
def get_chain_mappings() -> QueryChainMappings:
21+
"""
22+
Should return a dictionary containing property pairs mapped to query mappings.
23+
This is used to define how to chain results from this query to other possible queries
24+
"""
25+
# TODO: find a way to map list of hostnames:
26+
# AggregateProperties.HOST_IPS to HypervisorProperties.HYPERVISOR_NAME
27+
return None
28+
29+
@staticmethod
30+
def get_runner_mapping() -> Type[AggregateRunner]:
31+
"""
32+
Returns a mapping to associated Runner class for the Query (AggregateRunner)
33+
"""
34+
return AggregateRunner
35+
36+
@staticmethod
37+
def get_prop_mapping() -> Type[AggregateProperties]:
38+
"""
39+
Returns a mapping of valid presets for server side attributes (HypervisorProperties)
40+
"""
41+
return AggregateProperties
42+
43+
@staticmethod
44+
def get_server_side_handler() -> ServerSideHandler:
45+
"""
46+
method to configure a server handler which can be used to get 'filter' keyword arguments that
47+
can be passed to openstack function conn.compute.aggregates() to filter results for a valid preset-property
48+
pair
49+
50+
valid filters documented here:
51+
https://docs.openstack.org/openstacksdk/latest/user/proxies/compute.html
52+
https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-detail#list-aggregates
53+
"""
54+
# No server-side filters for AggregateQuery
55+
return ServerSideHandler({})
56+
57+
@staticmethod
58+
def get_client_side_handlers() -> ClientSideHandler:
59+
"""
60+
method to configure a set of client-side handlers which can be used to get local filter functions
61+
corresponding to valid preset-property pairs. These filter functions can be used to filter results after
62+
listing all aggregates.
63+
"""
64+
date_prop_list = [
65+
AggregateProperties.AGGREGATE_DELETED_AT,
66+
AggregateProperties.AGGREGATE_UPDATED_AT,
67+
AggregateProperties.AGGREGATE_CREATED_AT,
68+
]
69+
70+
string_prop_list = [
71+
AggregateProperties.AGGREGATE_HOSTTYPE,
72+
AggregateProperties.AGGREGATE_HOST_IPS,
73+
AggregateProperties.AGGREGATE_METADATA,
74+
AggregateProperties.AGGREGATE_LOCAL_STORAGE_TYPE,
75+
]
76+
77+
return ClientSideHandler(
78+
{
79+
QueryPresets.EQUAL_TO: ["*"],
80+
QueryPresets.NOT_EQUAL_TO: ["*"],
81+
QueryPresets.ANY_IN: ["*"],
82+
QueryPresets.NOT_ANY_IN: ["*"],
83+
QueryPresets.MATCHES_REGEX: string_prop_list,
84+
QueryPresets.NOT_MATCHES_REGEX: string_prop_list,
85+
QueryPresets.YOUNGER_THAN: date_prop_list,
86+
QueryPresets.YOUNGER_THAN_OR_EQUAL_TO: date_prop_list,
87+
QueryPresets.OLDER_THAN: date_prop_list,
88+
QueryPresets.OLDER_THAN_OR_EQUAL_TO: date_prop_list,
89+
QueryPresets.LESS_THAN: [AggregateProperties.AGGREGATE_GPUNUM],
90+
QueryPresets.LESS_THAN_OR_EQUAL_TO: [
91+
AggregateProperties.AGGREGATE_GPUNUM
92+
],
93+
QueryPresets.GREATER_THAN: [AggregateProperties.AGGREGATE_GPUNUM],
94+
QueryPresets.GREATER_THAN_OR_EQUAL_TO: [
95+
AggregateProperties.AGGREGATE_GPUNUM
96+
],
97+
}
98+
)

‎openstackquery/runners/aggregate_runner.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
import logging
2+
from typing import List, Optional
3+
4+
from openstack.compute.v2.aggregate import Aggregate
5+
6+
from openstackquery.aliases import OpenstackResourceObj, ServerSideFilter
7+
from openstackquery.openstack_connection import OpenstackConnection
8+
from openstackquery.runners.runner_wrapper import RunnerWrapper
9+
10+
logger = logging.getLogger(__name__)
11+
12+
13+
class AggregateRunner(RunnerWrapper):
14+
"""
15+
Runner class for openstack Aggregate resource
16+
AggregateRunner encapsulates running any openstacksdk Aggregate commands
17+
"""
18+
19+
RESOURCE_TYPE = Aggregate
20+
21+
def parse_meta_params(self, conn: OpenstackConnection, **kwargs):
22+
"""
23+
This method is a helper function that will parse a set of meta params specific to the resource and
24+
return a set of parsed meta-params to pass to _run_query
25+
"""
26+
logger.debug("AggregateQuery has no meta-params available")
27+
return super().parse_meta_params(conn, **kwargs)
28+
29+
def run_query(
30+
self,
31+
conn: OpenstackConnection,
32+
filter_kwargs: Optional[ServerSideFilter] = None,
33+
**kwargs,
34+
) -> List[OpenstackResourceObj]:
35+
"""
36+
This method runs the query by running openstacksdk commands
37+
38+
For AggregateQuery, this command finds all aggregates that match a given set of filter_kwargs
39+
:param conn: An OpenstackConnection object - used to connect to openstacksdk
40+
:param filter_kwargs: An Optional list of filter kwargs to pass to conn.identity.aggregates()
41+
to limit the hypervisors being returned.
42+
- see https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-detail#list-aggregates
43+
"""
44+
if not filter_kwargs:
45+
# return server info
46+
filter_kwargs = {}
47+
logger.debug(
48+
"running openstacksdk command conn.compute.aggregates(%s)",
49+
",".join(f"{key}={value}" for key, value in filter_kwargs.items()),
50+
)
51+
52+
# Note: Pagination isn't supported by the API,
53+
# as the `marker` and `limit` properties don't exist for list aggregates
54+
return conn.compute.aggregates(**filter_kwargs)

0 commit comments

Comments
 (0)