Releases: dragonflyoss/dragonfly
v2.3.2
Changelog
Dependency updates
- c5f39b7 chore(deps): bump github/codeql-action from 3.29.8 to 3.30.1 (#4311)
- a3a685d chore(deps): bump google.golang.org/api from 0.244.0 to 0.248.0 (#4303)
Chores
Contributors
Thanks to all the contributors! ❤️
v2.2.5
v2.3.1
Changelog
Features
- 7374114 feat(job): add timeout parameter to job scheduling (#4293)
- 584a612 feat(middlewares): support token from query parameter (#4294)
- 212c64d feat: add proxy port support across all host models (#4283)
- 447b510 feat: update the cache key for distinguishing from the old searcher way (#4301)
Bug Fixes
Dependency updates
- c9ee6d6 chore(deps): bump codecov/codecov-action from 5.4.3 to 5.5.0 (#4290)
- 95db8b3 chore(deps): bump github.com/bits-and-blooms/bitset from 1.22.0 to 1.24.0 (#4287)
- f3911bd chore(deps): bump github.com/stretchr/testify from 1.10.0 to 1.11.0 (#4284)
- fb93f29 chore(deps): bump go.uber.org/mock from 0.5.2 to 0.6.0 (#4288)
- 95f24f3 chore(deps): bump google.golang.org/grpc from 1.74.2 to 1.75.0 (#4285)
- 052db13 chore(deps): bump goreleaser/goreleaser-action from 6.3.0 to 6.4.0 (#4291)
- 44db190 chore(deps): bump kentaro-m/auto-assign-action from 9f6dbe84a80c6e7639d1b9698048b201052a2a94 to 1a583dfb5dd75d0df34ecc781081e5a6a73d2dce (#4286)
Chores
- 5986310 chore(manager): update subproject commit reference (#4296)
- 99e0155 chore: update submodule commits for client and console (#4313)
Contributors
Thanks to all the contributors! ❤️
v2.3.1-rc.4
Changelog
Features
- e169394 feat: Add WithGroupUUID logger and optimize GetGroupJobState in internal job (#4270)
- b3e8695 feat: Update JSON tag for CachedLayers field in Peer struct to "cached_layers" (#4258)
Bug Fixes
- 2fc87da refactor(job): use goroutine to unmarshal task result (#4245)
- bd2ae63 refactor: refacts the seed peer selection logic from grpc resolver to… (#4235)
- 1a87bfc refactor: remove unused seed peer related logic (#4259)
Dependency updates
- 364464c chore(deps): bump actions/cache from 4.2.2 to 4.2.4 (#4266)
- 3c15fbf chore(deps): bump actions/checkout from 4.2.2 to 5.0.0 (#4261)
- 883d6d8 chore(deps): bump actions/download-artifact from 4.3.0 to 5.0.0 (#4263)
- 88dc8e1 chore(deps): bump client-rs to v1.0.9 (#4256)
- 2b4df83 chore(deps): bump github.com/docker/go-connections from 0.5.0 to 0.6.0 (#4272)
- 686f339 chore(deps): bump github.com/prometheus/client_golang from 1.20.5 to 1.23.0 (#4268)
- 06b8563 chore(deps): bump github.com/redis/go-redis/v9 from 9.11.0 to 9.12.1 (#4271)
- 632c7b2 chore(deps): bump github.com/swaggo/swag from 1.16.5 to 1.16.6 (#4257)
- f30a79e chore(deps): bump github/codeql-action from 3.29.7 to 3.29.8 (#4260)
- d7cfc85 chore(deps): bump golang.org/x/crypto from 0.40.0 to 0.41.0 (#4265)
- b2b21c2 chore(deps): bump golang.org/x/sys from 0.34.0 to 0.35.0 (#4264)
Chores
Contributors
Thanks to all the contributors! ❤️
v2.3.1-rc.3
Changelog
Features
- 9641c3e feat: Moved preheatImage struct and parseManifestURL from manager/job/types.go to internal/job/image.go for better encapsulation (#4243)
- b3e8695 feat: Update JSON tag for CachedLayers field in Peer struct to "cached_layers" (#4258)
- baa061a feat: implement PreheatImage and StatImage for scheduler (#4246)
Dependency updates
- 88dc8e1 chore(deps): bump client-rs to v1.0.9 (#4256)
- 249bdfc chore(deps): bump docker/login-action from 3.4.0 to 3.5.0 (#4249)
- e875480 chore(deps): bump github.com/aws/aws-sdk-go from 1.55.7 to 1.55.8 (#4255)
- d9b06c5 chore(deps): bump github.com/onsi/gomega from 1.36.3 to 1.38.0 (#4241)
- 632c7b2 chore(deps): bump github.com/swaggo/swag from 1.16.5 to 1.16.6 (#4257)
- 7f071ea chore(deps): bump github/codeql-action from 3.29.3 to 3.29.4 (#4237)
- 28fa02f chore(deps): bump github/codeql-action from 3.29.4 to 3.29.5 (#4248)
- c0c8763 chore(deps): bump google.golang.org/api from 0.229.0 to 0.244.0 (#4251)
- 1de067f chore(deps): bump kentaro-m/auto-assign-action from 586b61c136c65d09c1775da39cc4a80e026834f4 to 9f6dbe84a80c6e7639d1b9698048b201052a2a94 (#4236)
Chores
- f535615 Merge branch 'main' of github.com:dragonflyoss/dragonfly into release-2.3
Contributors
Thanks to all the contributors! ❤️
v2.3.1-rc.2
Changelog
Features
- 9641c3e feat: Moved preheatImage struct and parseManifestURL from manager/job/types.go to internal/job/image.go for better encapsulation (#4243)
- 3f0b7b0 feat: enhance preheat job concurrency control (#4233)
- baa061a feat: implement PreheatImage and StatImage for scheduler (#4246)
Dependency updates
- 249bdfc chore(deps): bump docker/login-action from 3.4.0 to 3.5.0 (#4249)
- e875480 chore(deps): bump github.com/aws/aws-sdk-go from 1.55.7 to 1.55.8 (#4255)
- d9b06c5 chore(deps): bump github.com/onsi/gomega from 1.36.3 to 1.38.0 (#4241)
- 7f071ea chore(deps): bump github/codeql-action from 3.29.3 to 3.29.4 (#4237)
- 28fa02f chore(deps): bump github/codeql-action from 3.29.4 to 3.29.5 (#4248)
- c0c8763 chore(deps): bump google.golang.org/api from 0.229.0 to 0.244.0 (#4251)
- 1de067f chore(deps): bump kentaro-m/auto-assign-action from 586b61c136c65d09c1775da39cc4a80e026834f4 to 9f6dbe84a80c6e7639d1b9698048b201052a2a94 (#4236)
Chores
- 6c0ebc4 chore: bump console to v0.3.1 (#4232)
- a7efec4 test(e2e): update config to fix the slow resolution of seed peer address (#4234)
Contributors
Thanks to all the contributors! ❤️
v2.3.1-rc.1
v2.3.1-rc.0
Changelog
Features
- d67f30a feat(manager): add scheduler cluster ID based listing (#4222)
- 41da9b7 feat: add RemoteIp for calling client (#4231)
Dependency updates
- a64385f chore(deps): bump github.com/swaggo/swag from 1.16.4 to 1.16.5 (#4228)
- 9c4f628 chore(deps): bump github/codeql-action from 3.29.2 to 3.29.3 (#4224)
- 01ff406 chore(deps): bump kentaro-m/auto-assign-action from db77befe126df3163e8ee988af621569f7f2b82f to 586b61c136c65d09c1775da39cc4a80e026834f4 (#4225)
- 688f64c chore(deps): bump sigstore/cosign-installer from 3.9.1 to 3.9.2 (#4223)
Contributors
Thanks to all the contributors! ❤️
v2.3.1-beta.0
Changelog
Features
- be2e223 feat: Add support for selecting specific peer IPs for preheating in job scheduler (#4194)
- d32cf54 feat: Change Result column type to longtext in Job model to fix data too long error (#4181)
- 9fb26ae feat: add gc task to clean up inactive seed peers (#4176)
- b3b7d15 feat: add the gc task to clean up the inactive schedulers (#4175)
- cfc3180 feat: support preheat for multiple urls (#4185)
Bug Fixes
- 1a132f5 fix: populate the User from UserID when get application and job (#4178)
- cae253c fix: revert the remove of ip and port in mamager rpc server (#4180)
Chores
- b0a6a55 Add fmt, vet, and precheck targets to Makefile for better CI and code quality (#4182)
- b66c49f Tune dragonfly to Remove Old Entries & Use Longtext to fit larger warming jobs (#4145)
- d3e1c89 chore: bump client-rs to v1.0.1 (#4174)
- d13d290 chore: bump console version to v0.3.0 (#4196)
Contributors
Thanks to all the contributors! ❤️
v2.3.0
Dragonfly v2.3.0 is released!
Dragonfly v2.3.0 is released! 🎉🎉🎉 Thanks the contributors who made this release happend and welcome you to visit d7y.io website.
Features
Persistent Cache Task
It designs to provide persistent caching for tasks. This tool can import file and export file in P2P network. The solution is specifically engineered for high-speed read and write operations. This makes it particularly advantageous for scenarios involving large files, such as machine learning model checkpoints, where rapid, reliable access and distribution across the network are critical for training and inference workflows. By leveraging P2P distribution and persistent caching, dfcache significantly reduces I/O bottlenecks and accelerates the lifecycle of large data assets.
For documentation on how to use the dfcache command-line tool, please refer to the following link: https://d7y.io/docs/next/reference/commands/client/dfcache/
$ dfcache import /tmp/file.txt
⣷ Done: 2229733261$ dfcache export 2229733261 -O /tmp/file.txt
Resource Search (Tasks and Persistent Cache Tasks)
The Resource Search feature enables seamless querying of tasks, including files, images, and persistent cache tasks. It optimizes resource access, improving task management and retrieval efficiency.
Vortex: A P2P File Transfer Protocol Based on TLV
Vortex protocol is a high-performance peer-to-peer (P2P) file transfer protocol implementation in Rust, designed as part of the Dragonfly project. It utilizes the TLV (Tag-Length-Value) format for efficient and flexible data transmission, making it ideal for large-scale file distribution scenarios.
Packet Format:
- Packet Identifier (8 bits): Uniquely identifies each packet
- Tag (8 bits): Specifies data type in value field
- Length (32 bits): Indicates Value field length, up to 4 GiB
- Value (variable): Actual data content, maximum 1 GiB
Protocol Format:
-------------------------------------------------------------------------------------------------
| | | | |
| Packet Identifier (8 bits) | Tag (8 bits) | Length (32 bits) | Value (up to 4 GiB) |
| | | | |
-------------------------------------------------------------------------------------------------
For more information, please refer to the Vortex Protocol.
Enhanced Large File Distribution
This release significantly enhances Dragonfly's large file distribution capabilities, delivering improved efficiency and performance. We've revamped our scheduling algorithms for large file scenarios to ensure smarter resource and task allocation. Additionally, new mechanisms now more effectively balance the load across peers during large file transfers. Optimizations to the peer-to-peer (P2P) protocol and network transport layers further boost transmission efficiency.
These improvements include performance optimizations for both Client and Scheduler. You can find more details in the project's pull request.
Support scopes for Personal Access Tokens (PATs)
By enabling users to define specific access rights (scopes) for each PAT, we significantly enhance the security of Open API interactions. Instead of granting broad permissions, PATs can now be limited to only the necessary privileges required for a particular integration or task.
Enhanced Preheating
Implement Distributed Rate Limiting for Preheating Tasks
By limiting the rate at which preheating requests are initiated across the distributed system, it prevents excessive preheating activities from stressing the origin. This enhancement ensures a more stable preheating.
Support to set piece length for preheating
By allowing adjustment of the piece size, users can optimize data transfer efficiency, particularly in scenarios involving large files.
Flexible Preheating: Set Peer scope by Percentage or Count
This feature enhances preheating capabilities by allowing users to specify the preheating scope more precisely.
Implement Audit Logging for User Operations
This feature introduces comprehensive audit logging capabilities to track user operations within the system. Audit logs will record critical actions performed by users, such as initiating preheating tasks, deleting task caches, and other significant system interactions.
Garbage Collection
Dragonfly supports Garbage Collection (GC) Audit Logs and GC Job Records to track and manage garbage collection activities. The Manager enables automated GC retention, allowing records to be preserved for a configurable time period. Additionally, it provides the capability to manually trigger forced GC operations as needed.
This feature ensures efficient monitoring and management of GC processes, offering flexibility through automated retention policies and manual intervention for immediate GC execution.
Optimized File Download with Hard Link
File download needs to be done in a way that is efficient and secure. If users are downloading a large file, it is not efficient to download the file and copy to the output path. Instead, we can create a hard link to the file and send the link to the user. This way, we can avoid copying the file and save time and resources. If hard link fails (e.g. due to different file systems), dfdaemon will fallback to copying the file.
For more information, please refer to the file download workflow.
Hardware Acceleration for Piece Hash Computation
This feature enables hardware-accelerated Piece hash computation, significantly boosting performance and efficiency. By utilizing specialized hardware, the hash computation process is accelerated, allowing faster processing of large file.
Advanced Storage Management
Disk Space Validation for Operations
This feature enhances the client's storage functionality by implementing disk space validation. When insufficient disk space is detected, the client will return a failure response, preventing potential data corruption or incomplete operations.
Disk Garbage Collection Management
This feature enhances Peer's disk management by introducing configurable garbage collection (GC) thresholds based on disk usage. The distThreshold parameter allows users to define a specific disk capacity (e.g., 10TiB) as the base for calculating GC trigger points. If set, the distHighThresholdPercent (e.g., 80%) and distLowThresholdPercent (e.g., 60%) are applied relative to this capacity. If distThreshold is not provided or set to 0, these percentages are calculated based on the total actual disk space. When disk usage exceeds the high threshold, Dragonfly triggers GC(LRU) to reclaim space; GC stops when usage falls below the low threshold. This enables efficient management of a logical disk portion for caching, improving resource utilization and system performance.
Support for OpenTelemetry Tracing
Dragonfly supports for tracing based on OpenTelemetry, covering the Manager, Scheduler, and Peers. This enables end-to-end visibility into the download process, allowing users to query detailed information, such as overall download latency, using a specific task ID. The integration ensures efficient monitoring and performance analysis across the entire system.
Add tracing configuration as follows(in Manager, Scheduler and Peer):
You can access the Jaeger UI to visualize the traces.
For more information, please refer to the Tracing.
Security Enhancements
We extend our sincere gratitude to the CNCF TAG Security for their collaboration on a joint security audit. Their expertise and thorough review were invaluable in helping us identify areas for security improvement within Dragonfly.
For detailed information on the specific security issues addressed and the corresponding fixes, please refer to the following issue: #3811
Nydus
New features and enhancements
- nydusify copy: support chunked upload and retry mechanisms to handle large image blobs.
-...