Skip to content

Commit 37ae369

Browse files
authored
Improve public API type annotations & fix unit test type errors (#248)
* chore: improve typing of functions returning AnyCloudEvent kafka.conversion.from_binary() and from_structured() return AnyCloudEvent type var according to their event_type argument, but when event_type is None, type checkers cannot infer the return type. We now use an overload to declare that the return type is http.CloudEvent when event_type is None. Previously users had to explicitly annotate this type when calling without event_type. This happens quite a lot in this repo's test_kafka_conversions.py — this fixes quite a few type errors like: > error: Need type annotation for "result" [var-annotated] Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: type v1.Event chainable Set*() methods The v1.Event self-returning Set*() methods like SetData() were returning BaseEvent, which doesn't declare the same Set* methods. As a result, chaining more than one Set* method would make the return type unknown. This was causing type errors in test_event_pipeline.py. The Set*() methods now return the Self type. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: fix type errors in tests mypy was failing with lots of type errors in test modules. I've not annotated all fixtures, mostly fixed existing type errors. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: allow non-dict headers types in from_http() from_http() conversion function was requiring its headers argument to be a typing.Dict, which makes it incompatible with headers types of http libraries, which support features like multiple values per key. typing.Mapping and even _typeshed.SupportsItems do not cover these types. For example, samples/http-image-cloudevents/image_sample_server.py was failing to type check where it calls `from_http(request.headers, ...)`. To support these kind of headers types in from_http(), we now define our own SupportsDuplicateItems protocol, which is broader than _typeshed.SupportsItems. I've only applied this to from_http(), as typing.Mapping is OK for most other methods that accept dict-like objects, and using this more lenient interface everywhere would impose restrictions on our implementation, even though it might be more flexible for users. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * build: run mypy via tox Tox now runs mypy on cloudevents itself, and the samples. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * build(ci): run mypy in CI alongside linting Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: fix minor mypy type complaint in samples Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * feat: use Mapping, not Dict for input arguments Mapping imposes less restrictions on callers, because it's read-only and allows non-dict types to be passed without copying them as dict(), or passing dict-like values and ignoring the resulting type error. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: fix tests on py3.8 Tests were failing because the sanic dependency dropped support for py3.8 in its current release. sanic is now pinned to the last compatible version for py3.8 only. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * feat: support new model_validate_json() kwargs Pydantic added by_alias and by_name keyword arguments to BaseModel.model_validate_json in 2.11.1: pydantic/pydantic@acb0f10 This caused mypy to report that that the Pydantic v2 CloudEvent did not override model_validate_json() correctly. Our override now accepts these newly-added arguments. They have no effect, as the implementation does not use Pydantic to validate the JSON, but we also don't use field aliases, so the only effect they could have in the superclass would be to raise an error if they're both False. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: accept Mapping as well as SupportsDuplicateItems Although our types.SupportsDuplicateItems type is wider than Dict and Mapping, it's not a familar type to users, so explicitly accepting Mapping in the from_http() functions should make it more clear to users that a dict-like object is required for the headers argument. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: constrain deps to maintain py 3.8 support Python 3.8 is unsupported and dependencies (such as pydantic) are now shipping releases that fail to type check with mypy running in 3.8 compatibility mode. We run mypy in py 3.8 compatibility mode, so the mypy tox environments must only use deps that support 3.8. And unit tests run by py 3.8 must only use deps that support 3.8. To constrain the deps for 3.8 support, we use two constraint files, one for general environments that only constrains the dependencies that python 3.8 interpreters use, and another for mypy that constraints the dependencies that all interpreters use. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> --------- Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk>
1 parent c5645d8 commit 37ae369

25 files changed

+288
-85
lines changed

.github/workflows/main.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ jobs:
1717
- name: Install dev dependencies
1818
run: python -m pip install -r requirements/dev.txt
1919
- name: Run linting
20-
run: python -m tox -e lint
20+
run: python -m tox -e lint,mypy,mypy-samples-image,mypy-samples-json
2121

2222
test:
2323
strategy:

cloudevents/abstract/event.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ class CloudEvent:
3232
@classmethod
3333
def create(
3434
cls: typing.Type[AnyCloudEvent],
35-
attributes: typing.Dict[str, typing.Any],
35+
attributes: typing.Mapping[str, typing.Any],
3636
data: typing.Optional[typing.Any],
3737
) -> AnyCloudEvent:
3838
"""

cloudevents/conversion.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,9 @@ def from_json(
9191

9292
def from_http(
9393
event_type: typing.Type[AnyCloudEvent],
94-
headers: typing.Mapping[str, str],
94+
headers: typing.Union[
95+
typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
96+
],
9597
data: typing.Optional[typing.Union[str, bytes]],
9698
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
9799
) -> AnyCloudEvent:
@@ -260,7 +262,7 @@ def best_effort_encode_attribute_value(value: typing.Any) -> typing.Any:
260262

261263
def from_dict(
262264
event_type: typing.Type[AnyCloudEvent],
263-
event: typing.Dict[str, typing.Any],
265+
event: typing.Mapping[str, typing.Any],
264266
) -> AnyCloudEvent:
265267
"""
266268
Constructs an Event object of a given `event_type` from

cloudevents/http/conversion.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,9 @@ def from_json(
3737

3838

3939
def from_http(
40-
headers: typing.Dict[str, str],
40+
headers: typing.Union[
41+
typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
42+
],
4143
data: typing.Optional[typing.Union[str, bytes]],
4244
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
4345
) -> CloudEvent:
@@ -58,7 +60,7 @@ def from_http(
5860

5961

6062
def from_dict(
61-
event: typing.Dict[str, typing.Any],
63+
event: typing.Mapping[str, typing.Any],
6264
) -> CloudEvent:
6365
"""
6466
Constructs a CloudEvent from a dict `event` representation.

cloudevents/http/event.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,13 @@ class CloudEvent(abstract.CloudEvent):
3434

3535
@classmethod
3636
def create(
37-
cls, attributes: typing.Dict[str, typing.Any], data: typing.Optional[typing.Any]
37+
cls,
38+
attributes: typing.Mapping[str, typing.Any],
39+
data: typing.Optional[typing.Any],
3840
) -> "CloudEvent":
3941
return cls(attributes, data)
4042

41-
def __init__(self, attributes: typing.Dict[str, str], data: typing.Any = None):
43+
def __init__(self, attributes: typing.Mapping[str, str], data: typing.Any = None):
4244
"""
4345
Event Constructor
4446
:param attributes: a dict with cloudevent attributes. Minimally

cloudevents/kafka/conversion.py

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -111,11 +111,29 @@ def to_binary(
111111
return KafkaMessage(headers, message_key, data)
112112

113113

114+
@typing.overload
114115
def from_binary(
115116
message: KafkaMessage,
116-
event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
117+
event_type: None = None,
118+
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
119+
) -> http.CloudEvent:
120+
pass
121+
122+
123+
@typing.overload
124+
def from_binary(
125+
message: KafkaMessage,
126+
event_type: typing.Type[AnyCloudEvent],
117127
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
118128
) -> AnyCloudEvent:
129+
pass
130+
131+
132+
def from_binary(
133+
message: KafkaMessage,
134+
event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
135+
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
136+
) -> typing.Union[http.CloudEvent, AnyCloudEvent]:
119137
"""
120138
Returns a CloudEvent from a KafkaMessage in binary format.
121139
@@ -144,10 +162,11 @@ def from_binary(
144162
raise cloud_exceptions.DataUnmarshallerError(
145163
f"Failed to unmarshall data with error: {type(e).__name__}('{e}')"
146164
)
165+
result: typing.Union[http.CloudEvent, AnyCloudEvent]
147166
if event_type:
148167
result = event_type.create(attributes, data)
149168
else:
150-
result = http.CloudEvent.create(attributes, data) # type: ignore
169+
result = http.CloudEvent.create(attributes, data)
151170
return result
152171

153172

@@ -210,12 +229,32 @@ def to_structured(
210229
return KafkaMessage(headers, message_key, value)
211230

212231

232+
@typing.overload
213233
def from_structured(
214234
message: KafkaMessage,
215-
event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
235+
event_type: None = None,
236+
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
237+
envelope_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
238+
) -> http.CloudEvent:
239+
pass
240+
241+
242+
@typing.overload
243+
def from_structured(
244+
message: KafkaMessage,
245+
event_type: typing.Type[AnyCloudEvent],
216246
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
217247
envelope_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
218248
) -> AnyCloudEvent:
249+
pass
250+
251+
252+
def from_structured(
253+
message: KafkaMessage,
254+
event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
255+
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
256+
envelope_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
257+
) -> typing.Union[http.CloudEvent, AnyCloudEvent]:
219258
"""
220259
Returns a CloudEvent from a KafkaMessage in structured format.
221260
@@ -264,8 +303,9 @@ def from_structured(
264303
attributes["datacontenttype"] = val.decode()
265304
else:
266305
attributes[header.lower()] = val.decode()
306+
result: typing.Union[AnyCloudEvent, http.CloudEvent]
267307
if event_type:
268308
result = event_type.create(attributes, data)
269309
else:
270-
result = http.CloudEvent.create(attributes, data) # type: ignore
310+
result = http.CloudEvent.create(attributes, data)
271311
return result

cloudevents/pydantic/v1/conversion.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,9 @@
2121

2222

2323
def from_http(
24-
headers: typing.Dict[str, str],
24+
headers: typing.Union[
25+
typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
26+
],
2527
data: typing.Optional[typing.AnyStr],
2628
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
2729
) -> CloudEvent:
@@ -63,7 +65,7 @@ def from_json(
6365

6466

6567
def from_dict(
66-
event: typing.Dict[str, typing.Any],
68+
event: typing.Mapping[str, typing.Any],
6769
) -> CloudEvent:
6870
"""
6971
Construct an CloudEvent from a dict `event` representation.

cloudevents/pydantic/v1/event.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,9 @@ class CloudEvent(abstract.CloudEvent, BaseModel): # type: ignore
100100

101101
@classmethod
102102
def create(
103-
cls, attributes: typing.Dict[str, typing.Any], data: typing.Optional[typing.Any]
103+
cls,
104+
attributes: typing.Mapping[str, typing.Any],
105+
data: typing.Optional[typing.Any],
104106
) -> "CloudEvent":
105107
return cls(attributes, data)
106108

@@ -155,7 +157,7 @@ def create(
155157

156158
def __init__( # type: ignore[no-untyped-def]
157159
self,
158-
attributes: typing.Optional[typing.Dict[str, typing.Any]] = None,
160+
attributes: typing.Optional[typing.Mapping[str, typing.Any]] = None,
159161
data: typing.Optional[typing.Any] = None,
160162
**kwargs,
161163
):

cloudevents/pydantic/v2/conversion.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,9 @@
2222

2323

2424
def from_http(
25-
headers: typing.Dict[str, str],
25+
headers: typing.Union[
26+
typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
27+
],
2628
data: typing.Optional[typing.AnyStr],
2729
data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
2830
) -> CloudEvent:
@@ -64,7 +66,7 @@ def from_json(
6466

6567

6668
def from_dict(
67-
event: typing.Dict[str, typing.Any],
69+
event: typing.Mapping[str, typing.Any],
6870
) -> CloudEvent:
6971
"""
7072
Construct an CloudEvent from a dict `event` representation.

cloudevents/pydantic/v2/event.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ class CloudEvent(abstract.CloudEvent, BaseModel): # type: ignore
4444

4545
@classmethod
4646
def create(
47-
cls, attributes: typing.Dict[str, typing.Any], data: typing.Optional[typing.Any]
47+
cls,
48+
attributes: typing.Mapping[str, typing.Any],
49+
data: typing.Optional[typing.Any],
4850
) -> "CloudEvent":
4951
return cls(attributes, data)
5052

@@ -103,7 +105,7 @@ def create(
103105

104106
def __init__( # type: ignore[no-untyped-def]
105107
self,
106-
attributes: typing.Optional[typing.Dict[str, typing.Any]] = None,
108+
attributes: typing.Optional[typing.Mapping[str, typing.Any]] = None,
107109
data: typing.Optional[typing.Any] = None,
108110
**kwargs,
109111
):
@@ -173,6 +175,8 @@ def model_validate_json(
173175
*,
174176
strict: typing.Optional[bool] = None,
175177
context: typing.Optional[typing.Dict[str, Any]] = None,
178+
by_alias: typing.Optional[bool] = None,
179+
by_name: typing.Optional[bool] = None,
176180
) -> "CloudEvent":
177181
return conversion.from_json(cls, json_data)
178182

cloudevents/sdk/event/v1.py

Lines changed: 14 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,15 @@
1111
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
1212
# License for the specific language governing permissions and limitations
1313
# under the License.
14+
from __future__ import annotations
15+
1416
import typing
1517

1618
from cloudevents.sdk.event import base, opt
1719

20+
if typing.TYPE_CHECKING:
21+
from typing_extensions import Self
22+
1823

1924
class Event(base.BaseEvent):
2025
_ce_required_fields = {"id", "source", "type", "specversion"}
@@ -79,39 +84,39 @@ def Extensions(self) -> dict:
7984
return {}
8085
return dict(result)
8186

82-
def SetEventType(self, eventType: str) -> base.BaseEvent:
87+
def SetEventType(self, eventType: str) -> Self:
8388
self.Set("type", eventType)
8489
return self
8590

86-
def SetSource(self, source: str) -> base.BaseEvent:
91+
def SetSource(self, source: str) -> Self:
8792
self.Set("source", source)
8893
return self
8994

90-
def SetEventID(self, eventID: str) -> base.BaseEvent:
95+
def SetEventID(self, eventID: str) -> Self:
9196
self.Set("id", eventID)
9297
return self
9398

94-
def SetEventTime(self, eventTime: typing.Optional[str]) -> base.BaseEvent:
99+
def SetEventTime(self, eventTime: typing.Optional[str]) -> Self:
95100
self.Set("time", eventTime)
96101
return self
97102

98-
def SetSubject(self, subject: typing.Optional[str]) -> base.BaseEvent:
103+
def SetSubject(self, subject: typing.Optional[str]) -> Self:
99104
self.Set("subject", subject)
100105
return self
101106

102-
def SetSchema(self, schema: typing.Optional[str]) -> base.BaseEvent:
107+
def SetSchema(self, schema: typing.Optional[str]) -> Self:
103108
self.Set("dataschema", schema)
104109
return self
105110

106-
def SetContentType(self, contentType: typing.Optional[str]) -> base.BaseEvent:
111+
def SetContentType(self, contentType: typing.Optional[str]) -> Self:
107112
self.Set("datacontenttype", contentType)
108113
return self
109114

110-
def SetData(self, data: typing.Optional[object]) -> base.BaseEvent:
115+
def SetData(self, data: typing.Optional[object]) -> Self:
111116
self.Set("data", data)
112117
return self
113118

114-
def SetExtensions(self, extensions: typing.Optional[dict]) -> base.BaseEvent:
119+
def SetExtensions(self, extensions: typing.Optional[dict]) -> Self:
115120
self.Set("extensions", extensions)
116121
return self
117122

cloudevents/sdk/types.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,25 @@
1414

1515
import typing
1616

17+
_K_co = typing.TypeVar("_K_co", covariant=True)
18+
_V_co = typing.TypeVar("_V_co", covariant=True)
19+
1720
# Use consistent types for marshal and unmarshal functions across
1821
# both JSON and Binary format.
1922

2023
MarshallerType = typing.Callable[[typing.Any], typing.AnyStr]
2124

2225
UnmarshallerType = typing.Callable[[typing.AnyStr], typing.Any]
26+
27+
28+
class SupportsDuplicateItems(typing.Protocol[_K_co, _V_co]):
29+
"""
30+
Dict-like objects with an items() method that may produce duplicate keys.
31+
"""
32+
33+
# This is wider than _typeshed.SupportsItems, which expects items() to
34+
# return type an AbstractSet. werkzeug's Headers class satisfies this type,
35+
# but not _typeshed.SupportsItems.
36+
37+
def items(self) -> typing.Iterable[typing.Tuple[_K_co, _V_co]]:
38+
pass

cloudevents/tests/test_converters.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
def test_binary_converter_raise_unsupported():
2222
with pytest.raises(exceptions.UnsupportedEvent):
2323
cnvtr = binary.BinaryHTTPCloudEventConverter()
24-
cnvtr.read(None, {}, None, None)
24+
cnvtr.read(None, {}, None, None) # type: ignore[arg-type] # intentionally wrong type # noqa: E501
2525

2626

2727
def test_base_converters_raise_exceptions():
@@ -35,8 +35,8 @@ def test_base_converters_raise_exceptions():
3535

3636
with pytest.raises(Exception):
3737
cnvtr = base.Converter()
38-
cnvtr.write(None, None)
38+
cnvtr.write(None, None) # type: ignore[arg-type] # intentionally wrong type
3939

4040
with pytest.raises(Exception):
4141
cnvtr = base.Converter()
42-
cnvtr.read(None, None, None, None)
42+
cnvtr.read(None, None, None, None) # type: ignore[arg-type] # intentionally wrong type # noqa: E501

cloudevents/tests/test_event_from_request_converter.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
@pytest.mark.parametrize("event_class", [v03.Event, v1.Event])
2626
def test_binary_converter_upstream(event_class):
2727
m = marshaller.NewHTTPMarshaller([binary.NewBinaryHTTPCloudEventConverter()])
28-
event = m.FromRequest(event_class(), data.headers[event_class], None, lambda x: x)
28+
event = m.FromRequest(event_class(), data.headers[event_class], b"", lambda x: x)
2929
assert event is not None
3030
assert event.EventType() == data.ce_type
3131
assert event.EventID() == data.ce_id

cloudevents/tests/test_event_pipeline.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ def test_object_event_v1():
7777
_, structured_body = m.ToRequest(event)
7878
assert isinstance(structured_body, bytes)
7979
structured_obj = json.loads(structured_body)
80-
error_msg = f"Body was {structured_body}, obj is {structured_obj}"
80+
error_msg = f"Body was {structured_body!r}, obj is {structured_obj}"
8181
assert isinstance(structured_obj, dict), error_msg
8282
assert isinstance(structured_obj["data"], dict), error_msg
8383
assert len(structured_obj["data"]) == 1, error_msg

0 commit comments

Comments
 (0)