refactor: Introduce new Apify storage client #470

vdusek · 2025-05-10T09:22:35Z

Description

Integration of the Crawlee v1 changes, mostly new storages & storage clients (introduced in refactor!: Introduce new storage client system crawlee-python#1194).

Issues

Closes: Introduce new Apify storage client #469

Testing

The current test set covers the changes.

src/apify/apify_storage_client/_dataset_client.py

src/apify/apify_storage_client/__init__.py

janbuchar · 2025-06-26T14:19:49Z

docs/03_concepts/code/03_dataset_exports.py

@@ -11,14 +11,14 @@ async def main() -> None:
        await dataset.export_to(
            content_type='csv',
            key='data.csv',
-            to_key_value_store_name='my-cool-key-value-store',
+            to_kvs_name='my-cool-key-value-store',


Is this BC break worth it?

let's evaluate all the potential BCs at the end

Sure. I thought we are nearing that now 😁

Since we're just re-exporting the storages from crawlee here, there will be many more cases than this one. I'm not saying we have to rename this particular argument (and I will undo it if you insist—just I don't like those long identifiers, especially when we can use the KVS abbreviation).

src/apify/storage_clients/_apify/_dataset_client.py

src/apify/_actor.py

tests/integration/conftest.py

## Description - I consolidated all commits from #1107 into this new PR. - The previous storage-clients implementation was completely replaced with a redesigned clients, including: - new storage-client interface, - in-memory storage client, - file-system storage client, - Apify storage client (implemented in the SDK; see apify/apify-sdk-python#470), - and various adjustments elsewhere in the codebase. - The old "memory plus persist" client has been split into separate memory and file-system implementations. - The `Configuration.persist_storage` and `Configuration.persist_metadata` options were removed. - All old collection clients have been removed, they're no longer needed. - Each storage client now prints warnings if you pass method arguments it does not support. - The creation management modules in the storage clients and storages were removed. - Storage client parameters (e.g. `purge_on_start`, or `token` and `base_api_url` for the Apify client) are configured via the `Configuration`. - Every storage, and its corresponding client, now provides both a `purge` method (which clears all items but preserves the storage and metadata) and a `drop` method (which removes the entire storage, metadata included). - All unused types, models, and helper utilities have been removed. - The detailed, per-storage/client changes are listed below... ### Dataset - Properties: - `id` - `name` - `metadata` - Methods: - `open` - `purge` (new method) - `drop` - `push_data` - `get_data` - `iterate_items` - `list_items` (new method) - `export_to` - Breaking changes: - `from_storage_object` method has been removed - Use the `open` method with `name` or `id` instead. - `get_info` -> `metadata` property - `storage_object` -> `metadata` property - `set_metadata` method has been removed (it wasn't propage to clients) - Do we want to support it (e.g. for renaming)? - `write_to_json` -> method has been removed, use `export_to` instead - `write_to_csv` -> method has been removed, use `export_to` instead ```python import asyncio from crawlee.storage_clients import FileSystemStorageClient from crawlee.storages import Dataset async def main() -> None: dataset = await Dataset.open(storage_client=FileSystemStorageClient()) print(f'default dataset - ID: {dataset.id}, name: {dataset.name}') await dataset.push_data({'name': 'John'}) await dataset.push_data({'name': 'John', 'age': 20}) await dataset.push_data({}) dataset_with_name = await Dataset.open( name='my_dataset', storage_client=FileSystemStorageClient(), ) print(f'named dataset - ID: {dataset_with_name.id}, name: {dataset_with_name.name}') await dataset_with_name.push_data([{'age': 30}, {'age': 25}]) print('Default dataset items:') async for item in dataset.iterate_items(skip_empty=True): print(item) print('Named dataset items:') async for item in dataset_with_name.iterate_items(): print(item) items = await dataset.get_data() print(items) dataset_by_id = await Dataset.open(id=dataset_with_name.id) print(f'dataset by ID - ID: {dataset_by_id.id}, name: {dataset_by_id.name}') if __name__ == '__main__': asyncio.run(main()) ``` ### Key-value store - Properties: - `id` - `name` - `metadata` - Methods: - `open` - `purge` (new method) - `drop` - `get_value` - `set_value` - `delete_value` (new method, Apify platform's set_value support setting an empty value to a key, so having a separate method for deleting is useful) - `iterate_keys` - `list_keys` (new method) - `get_public_url` - `get_auto_saved_value` - `persist_autosaved_values` - Breaking changes: - `from_storage_object` method has been removed - Use the `open` method with `name` or `id` instead. - `get_info` -> `metadata` property - `storage_object` -> `metadata` property - `set_metadata` method has been removed (it wasn't propage to clients) - Do we want to support it (e.g. for renaming)? ```python import asyncio import requests from crawlee.storage_clients import FileSystemStorageClient from crawlee.storages import KeyValueStore async def main() -> None: print('Opening key-value store "my_kvs"...') storage_client = FileSystemStorageClient() kvs = await KeyValueStore.open(name='my_kvs', storage_client=storage_client) print('Setting value to "file.json"...') await kvs.set_value('file.json', {'key': 'value'}) print('Setting value to "file.jpg"...') response = requests.get('https://avatars.githubusercontent.com/u/25082181') await kvs.set_value('file.jpg', response.content) print('Iterating over keys:') async for key in kvs.iterate_keys(): print(f'Key: {key}') print('Listing keys:') keys = [key.key for key in await kvs.list_keys()] print(f'Keys: {keys}') for key in keys: print(f'Getting value of {key}...') value = await kvs.get_value(key) print(f'Value: {str(value)[:100]}') print('Deleting value of "file.json"...') await kvs.delete_value('file.json') kvs_default = await KeyValueStore.open(storage_client=storage_client) special_key = 'key with spaces/and/slashes!@#$%^&*()' test_value = 'Special key value' await kvs_default.set_value(key=special_key, value=test_value) record = await kvs_default.get_value(key=special_key) assert record is not None assert record == test_value result = await kvs_default.list_keys() print(f'kvs_default list keys = {result}') kvs_2 = await KeyValueStore.open() result = await kvs_2.list_keys() print(f'kvs_2 list keys = {result}') if __name__ == '__main__': asyncio.run(main()) ``` ### Request queue - Properties: - `id` - `name` - `metadata` - Methods: - `open` - `purge` (new method) - `drop` - `add_request` - `add_requests_batched` -> `add_requests` - `fetch_next_request` - `get_request` - `mark_request_as_handled` - `reclaim_request` - `is_empty` - `is_finished` - Breaking changes: - `from_storage_object` method has been removed - Use the `open` method with `name` or `id` instead. - `get_info` -> `metadata` property - `storage_object` -> `metadata` property - `set_metadata` method has been removed (it wasn't propage to clients) - Do we want to support it (e.g. for renaming)? - `get_handled_count` method had been removed - Use `metadata.handled_request_count` instead. - `get_total_count` method has been removed - Use `metadata.total_request_count` instead. - `resource_directory` from the `RequestQueueMetadata` was removed, use `path_to...` property instead. - `RequestQueueHead` model has been removed - Use `RequestQueueHeadWithLocks` instead. - Notes: - New RQ `add_requests` contain `forefront` arg (Apify API supports it) ```python import asyncio from crawlee import Request from crawlee.configuration import Configuration from crawlee.storage_clients import FileSystemStorageClient from crawlee.storages import RequestQueue async def main() -> None: rq = await RequestQueue.open( name='my-queue', storage_client=FileSystemStorageClient(), configuration=Configuration(purge_on_start=True), ) print(f'RequestQueue: {rq}') print(f'RequestQueue client: {rq._client}') await rq.add_requests( requests=[ Request.from_url('https://example.com', use_extended_unique_key=True), Request.from_url('https://crawlee.dev', use_extended_unique_key=True), Request.from_url('https://apify.com', use_extended_unique_key=True), ], ) print('Requests were added to the queue') is_empty = await rq.is_empty() is_finished = await rq.is_finished() print(f'Is empty: {is_empty}') print(f'Is finished: {is_finished}') request = await rq.fetch_next_request() print(f'Fetched request: {request}') await rq.add_request('https://facebook.com', forefront=True) request = await rq.fetch_next_request() print(f'Fetched request: {request}') rq_default = await RequestQueue.open( storage_client=FileSystemStorageClient(), configuration=Configuration(purge_on_start=True), ) await rq_default.add_request('https://example.com/1') await rq_default.add_requests( [ 'https://example.com/priority-1', 'https://example.com/priority-2', 'https://example.com/priority-3', ] ) await rq_default.add_request('https://example.com/2') if __name__ == '__main__': asyncio.run(main()) ``` ### BaseDatasetClient - Properties: - `metadata` - Methods: - `open` - `purge` - `drop` - `push_data` - `get_data` - `iterate_items` ### BaseKeyValueStoreClient - Properties: - `metadata` - Methods: - `open` - `purge` - `drop` - `get_value` - `set_value` - `delete_value` - `iterate_keys` - `get_public_url` ### BaseRequestQueueClient - Properties: - `metadata` - Methods: - `open` - `purge` - `drop` - `add_requests_batch` -> `add_batch_of_requests` (one backend method for 2 frontend methods) - `get_request` - `fetch_next_request` - `mark_request_as_handled` - `reclaim_request` - `is_empty` - Models - `RequestQueueHeadWithLocks` -> `RequestQueueHead` - `BatchRequestsOperationResponse` -> `AddRequestsResponse` - Notes: - Old file system (memory) version didn't persist the in-progress requests - Old file system (memory) version didn't persist the forefront values (now there is a FS-specific `_sequence` field in the FS Request) - The methods manipulating locks and listing heads are now only internal methods of Apify RQ client. ## Issues - Closes: #92 - Closes: #147 - Closes: #783 - Closes: #1247 - Relates: #1175 - Relates: #1191 ## Testing - The original tests were mostly removed and replaced with a new ones. - Each storage-client implementation now has its own dedicated tests at the client level (more targeted/edge-case coverage). - On top of that, there are storage-level tests that use a parametrized fixture for each storage client (`file-system` and `memory`), ensuring every storage test runs against every client implementation. ## Checklist - [x] CI passed

src/apify/storage_clients/_apify/_dataset_client.py

vdusek · 2025-07-02T13:21:26Z

Probably let's wait for @janbuchar, and also I'll add an upgrading guide for v2.

janbuchar

I haven't made it through the request queue client, but here's a batch of comments.

janbuchar · 2025-07-15T20:15:07Z

src/apify/storage_clients/_file_system/_key_value_store_client.py

+    """
+
+    @override
+    async def purge(self) -> None:


Does this override mean that Crawlee does not treat INPUT.json differently from other files anymore and this logic is only added in the SDK? I'm a fan, but I wonder if the rest of the team is on board with this.

Exactly - because Crawlee does not use INPUT.json for anything. We are in the same boat with @Pijukatel here.

@B4nan, would you be OK with this change? TLDR; so you don't have to study the code: As Honza wrote, Crawlee now treats INPUT.json just like any other file. Here in SDK, I've added a minimal override of FileSystemStorageClient that only customizes the purge method, preventing it from deleting INPUT.json. That's it.

are we talking about the input.json in the project root? since that is implemented on crawlee level (in JS) and i think it should stay this way. i am not talking about the purging logic, in JS, if there is an INPUT.json file in the root, KVS.getInput() will prefer it over anything in the storage folder (the local/memory implementation, with apify client it is ignored completely). not sure if we even do this in the python version (and I am open to discuss if we want to remove this from v4, but it feels like a good thing to me)

if this is just about purging, i am not entirely sure i understand what it means, we dont want to autopurge inputs on crawlee level, crawlee shouldnt behave differently because you init the SDK (not locally).

this is the part where we handle input file in the project root in JS

https://github.com/apify/crawlee/blob/5c4726df96e358a9bbf44a0cd2760e4e269f0fae/packages/core/src/storages/key_value_store.ts#L697

From the "Crawlee is a web scraping framework" point of view, handling input feels a bit redundant here - if you intend to only run the crawler outside of Apify, you'll probably want to accept input some other way - CLI arguments, environment variables, some domain-speciffic config file... Providing a helper for loading a JSON file from two possible locations doesn't add much value IMO.

Asking Apifiers about this doesn't make much sense in this situation, because basically everything we write is intended to run on Apify 🙂

src/apify/storage_clients/_apify/_key_value_store_client.py

Pijukatel

While testing the change on platform I used modified benchmark tests. They are specific, because they scrape locally hosted site and so the speed of scraping is unusually fast - no network related delays. This exposes RequestQueue race condition that can happen when checking if the RQ is empty or not.
When running the test using this branch, I saw that the crawler premature finished even though RQ was still quite full. The hypothesis about premature exit was confirmed when running same test on branch modified with extra log and sleep.

Please see following tiny commit based on this branch and attached log:
5b25728

From this run: https://console.apify.com/actors/UdJ0NQp2j8140G9db/runs/4yfZqLvLo1xhrk2Cb#log

Pijukatel · 2025-07-22T13:44:22Z

src/apify/storage_clients/_apify/_request_queue_client.py

+        # Prepare requests for API by converting to dictionaries
+        requests_dict = [request.model_dump(by_alias=True) for request in requests]
+
+        # Remove 'id' fields from requests as the API doesn't accept them


I think this could be achieved during model dump by request.model_dump(by_alias=True, exclude={"id"})

Pijukatel · 2025-07-22T14:08:48Z

src/apify/storage_clients/_apify/_request_queue_client.py

+        self._api_client = api_client
+        """The Apify request queue client for API operations."""
+
+        self._api_public_base_url = api_public_base_url


Is this used anywhere if the client is already created?

vdusek self-assigned this May 10, 2025

github-actions bot added this to the 114th sprint - Tooling team milestone May 10, 2025

github-actions bot added the t-tooling Issues with this label are in the ownership of the tooling team. label May 10, 2025

vdusek changed the title ~~New apify storage clients~~ refactor: Introduce new Apify storage client May 10, 2025

vdusek mentioned this pull request May 10, 2025

refactor!: Introduce new storage client system apify/crawlee-python#1194

Merged

1 task

janbuchar reviewed May 16, 2025

View reviewed changes

src/apify/apify_storage_client/_dataset_client.py Outdated Show resolved Hide resolved

vdusek force-pushed the new-apify-storage-clients branch from d27c080 to 82efd3e Compare June 12, 2025 12:44

github-actions bot added the tested Temporary label used only programatically for some analytics. label Jun 18, 2025

vdusek force-pushed the new-apify-storage-clients branch 2 times, most recently from 067b793 to 104a168 Compare June 23, 2025 09:12

vdusek added 6 commits June 26, 2025 08:22

Rm old Apify storage clients

5c437c9

Add init version of new Apify storage clients

bf55338

Move specific models from Crawlee to SDK

6b2f82b

Adapt to Crawlee v1

38bef68

Adapt to Crawlee v1 (p2)

1f85430

Fix default storage IDs

a3d68a2

vdusek force-pushed the new-apify-storage-clients branch from dc7f0a7 to a3d68a2 Compare June 26, 2025 06:25

vdusek modified the milestones: 114th sprint - Tooling team, 117th sprint - Tooling team Jun 26, 2025

vdusek added 3 commits June 26, 2025 10:42

Fix integration test and Not implemented exception in purge

c77e8d5

Fix unit tests

8731aff

fix lint

8dfaffb

vdusek marked this pull request as ready for review June 26, 2025 13:04

vdusek requested a review from Pijukatel June 26, 2025 13:05

janbuchar self-requested a review June 26, 2025 13:27

vdusek added 2 commits June 26, 2025 16:40

add KVS record_exists not implemented

53fad07

update to apify client 1.12 and implement record exists

5869f8e

janbuchar reviewed Jun 26, 2025

View reviewed changes

vdusek added 2 commits June 27, 2025 08:43

Move default storage IDs to Configuration

82e65fc

opening storages get default id from config

8de950b

Pijukatel reviewed Jul 2, 2025

View reviewed changes

src/apify/storage_clients/_apify/_dataset_client.py Outdated Show resolved Hide resolved

renaming

3cd7dfe

vdusek requested a review from Pijukatel July 2, 2025 10:54

Pijukatel approved these changes Jul 2, 2025

View reviewed changes

vdusek and others added 7 commits July 3, 2025 16:08

Merge branch 'master' into new-apify-storage-clients

6fe9eb3

fixes after merge commit

1547cbd

Merge branch 'master' into new-apify-storage-clients

bb47efc

Change from orphan commit to master in crawlee version

4e4fa93

Merge branch 'master' into new-apify-storage-clients

683cb31

fix encrypted secrets test

e5b2bc4

Add Apify's version of FS client that keeps the INPUT json

638756f

janbuchar reviewed Jul 15, 2025

View reviewed changes

vdusek added 4 commits July 16, 2025 09:31

update metadata fixes

931b0ca

Merge branch 'master' into new-apify-storage-clients

ad7c0d8

KVS metadata extended model

1f3c481

fix url signing secret key

44d8e09

Pijukatel self-requested a review July 17, 2025 13:23

Pijukatel requested changes Jul 17, 2025

View reviewed changes

Apify storage client fixes and new docs groups

ca72313

vdusek force-pushed the new-apify-storage-clients branch from 2f39b35 to ca72313 Compare July 20, 2025 18:49

Pijukatel and others added 4 commits July 21, 2025 12:37

Add test for RequestQueue.is_finished

bc61fee

Check _queue_has_locked_requests in is_empty

16b76dd

Merge branch 'master' into new-apify-storage-clients

b6e8a5f

Package structure update

a3f8c6e

Pijukatel reviewed Jul 22, 2025

View reviewed changes

Pijukatel self-requested a review July 22, 2025 13:45

Pijukatel reviewed Jul 22, 2025

View reviewed changes

Fix request list (HttpResponse.read is now async)

594a8e5

refactor: Introduce new Apify storage client #470

Are you sure you want to change the base?

refactor: Introduce new Apify storage client #470

Uh oh!

Conversation

vdusek commented May 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Testing

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek commented Jul 2, 2025

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Pijukatel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vdusek commented May 10, 2025 •

edited

Loading

Pijukatel left a comment •

edited

Loading