Skip to content

Mikepaw.comment changes #460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 16, 2025
Merged

Mikepaw.comment changes #460

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a29c6b7
Some comment changes around the client makeup.
mpawliszyn May 13, 2025
56f64ae
Adding comment around 100 file S3 limit.
mpawliszyn May 16, 2025
b05c806
Refactoring the S3 Client Readme.
mpawliszyn May 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ...t-s3/src/main/kotlin/app/cash/backfila/client/s3/internal/S3DatasourceBackfillOperator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ class S3DatasourceBackfillOperator<R : Any, P : Any>(
require(fileKeys.isNotEmpty()) {
"No files found for bucket:${backfill.getBucket(config)} prefix:$pathPrefix. At least one file must exist."
}
// We limit to 100 files since each file is run in parallel.
require(fileKeys.size <= 100) {
"Listing files matching the prefix contains ${fileKeys.size} which is more than 100 files. " +
"Check your prefix. First 3 file keys ${fileKeys.slice(0..2)}"
Expand Down
6 changes: 6 additions & 0 deletions client/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,9 @@

Contains the general, public client definitions for Backfila so clients and the service can
communicate. Never use this directly, instead use the specialized clients.

Java/Kotlin
Client - public API for communicating with Backfila, Customers depend on this for general backfila features (Parameters, dry run, logging)
Client-<service framework> - provides the base interaction with backfila for your service framework(Logging setup, client service, registration on startup). You install one of these in your real implementation. backfila-embedded is your test implementation.
Client-base - base functionality that all downstream datasource clients need (implementions of common features, parameters, Operator caching). Provides and SPI for those clients to satisfy in order to get this base functionality. This is private. Customers cannot depend on this. Only datasource clients depend on this.
client-<specific datasouce> - The specific datasouce Backfila implementation. This is where the ergonomics of working with your particular datasource exist