Skip to content

feat(backends): Swap to std python grpc lib#274

Merged
devsjc merged 19 commits intomainfrom
devsjc/python-grpc
Apr 28, 2026
Merged

feat(backends): Swap to std python grpc lib#274
devsjc merged 19 commits intomainfrom
devsjc/python-grpc

Conversation

@devsjc
Copy link
Copy Markdown
Contributor

@devsjc devsjc commented Apr 2, 2026

Modifies the GRPC client to use the default python GRPC library.

This is built on C, and doesn't do the same Serde setup of betterproto so should be much lighter on memory. As such, we shouldn't need the sync client any more.

I've also modified the tests such that most of the functionality is tested in unit tests, rather than integration tests. This is faster and cleaner.

Testing in development showed a much flatter memory profile and no significant change to CPU. In the below graphs, anything before the 14:30 in the middle was the old library, and afterwards, the new one:

image

@devsjc devsjc changed the title feat(backends): Swap to std sync grpc lib feat(backends): Swap to std python grpc lib Apr 2, 2026
@devsjc devsjc force-pushed the devsjc/python-grpc branch 17 times, most recently from cade8a5 to ce346a9 Compare April 9, 2026 09:23
@peterdudfield peterdudfield mentioned this pull request Apr 9, 2026
7 tasks
@devsjc devsjc force-pushed the devsjc/python-grpc branch from dc543cf to ba47979 Compare April 10, 2026 08:52
Comment thread pyproject.toml Outdated
@devsjc devsjc force-pushed the devsjc/python-grpc branch 2 times, most recently from 00f49ca to 818176e Compare April 16, 2026 10:28
@devsjc devsjc force-pushed the devsjc/python-grpc branch from 818176e to ed90d3b Compare April 16, 2026 10:39
@devsjc devsjc marked this pull request as ready for review April 16, 2026 10:39
@devsjc devsjc force-pushed the devsjc/python-grpc branch from ed90d3b to 928764a Compare April 16, 2026 11:00
Comment thread src/quartz_api/cmd/main.py
Comment thread src/quartz_api/cmd/main.py
Comment thread src/quartz_api/internal/backends/dataplatform/client.py
# assert len(data) == 11


# 6.2 check status routes
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where did these tests go?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Into a new file in the uk_national folder!

Comment thread src/quartz_api/tests/integration/uk_national/test_app.py
Copy link
Copy Markdown
Contributor

@peterdudfield peterdudfield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive left a few comments.

And it looks like we want to run it on dev for a few days, to make sure

  1. there are no errors
  2. the memory and cpu are ok

@peterdudfield
Copy link
Copy Markdown
Contributor

Looks like there are some errors at the moment

Screenshot 2026-04-17 at 12 19 24

@devsjc devsjc force-pushed the devsjc/python-grpc branch from e61dafc to 9fd488b Compare April 22, 2026 08:33
@devsjc
Copy link
Copy Markdown
Contributor Author

devsjc commented Apr 23, 2026

More stats from APItally showing a decreased memory useage:

image

@devsjc devsjc merged commit 8895569 into main Apr 28, 2026
8 checks passed
@devsjc devsjc deleted the devsjc/python-grpc branch April 28, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants