Adding version control to samples, starting materials and cells #1373
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… and save the same version. Added better error handling for if an invalid id is used
Adds deepdiff ~= 8.1 to project dependencies to enable proper comparison of nested dictionaries and lists in version control functionality.
Replaces simple dict_diff function with DeepDiff library to properly handle nested dictionaries, lists, type changes, and provide detailed change information for version comparisons.
Adds comprehensive safety checks to restore_version: - Permissions check requiring write access - Protected fields list preventing restoration of critical system fields (refcode, _id, immutable_id, creator_ids, file_ObjectIds, version) - Type consistency check preventing cross-type restoration - Model validation ensuring restored data passes schema validation - Atomic version incrementing using shared counter to prevent collisions The version field now always increments forward to avoid duplicate version numbers when restoring and then making subsequent changes.
Adds action field to track why each version was created: - 'manual_save': User explicitly saved (save-version endpoint or save-item) - 'auto_save': Reserved for future block-triggered auto-saves - 'pre_restore_backup': System backup created before restoring Refactored version saving into _save_version_snapshot() helper function that can be called with different action parameters. The restore_version endpoint also tracks which version was restored to via restored_from_version field.
Changes save_item to update the item BEFORE saving the version snapshot, preventing orphaned versions if the item update fails. Previously: save version → update item (if item update failed, orphaned version) Now: update item → save version (if version save fails, item is still saved) If version save fails after successful item update, the error is logged but the request still succeeds since the user's work has been saved.
Add version field to the HasRevisionControl Pydantic model to support the version control system's snapshot tracking. Fix the save_item endpoint to correctly increment version by adding it to updated_data rather than the discarded item object.
Add 33 tests covering all version control functionality: - Save, list, get, compare, restore, and delete version endpoints - Auto-versioning on save_item - Atomic version counter with race condition prevention - Protected field validation during restore - Permissions enforcement - Error handling and edge cases
- Add action and restored_from_version fields to list_versions endpoint - Change restore to create version snapshot AFTER restoring (not before) - Version snapshot now contains the restored data for clearer audit trail - Update action type from "pre_restore_backup" to "restored"
- Add version control API service methods to server_fetch_utils.js - Create VersionHistoryModal component for viewing and managing versions - Add version history button to EditPage navbar - Support version preview and restore functionality with proper state management
- Add new TestActionFields class with 5 tests validating action values - Test manual_save action from save-version endpoint - Test manual_save action from save-item endpoint (user saves) - Test restored action with restored_from_version reference - Test that restored version snapshots contain the restored data - Test complete audit trail across multiple saves and restore - Rename test_list_versions_action_field to be more descriptive - Update test_restore_version_creates_backup to _creates_snapshot - Remove duplicate action field tests from TestRestoreVersion class - Fix unused variable in test_get_version_success
be-smith
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1373 +/- ##
==========================================
+ Coverage 80.14% 80.46% +0.31%
==========================================
Files 70 70
Lines 4799 4949 +150
==========================================
+ Hits 3846 3982 +136
- Misses 953 967 +14
🚀 New features to boost your workflow:
|
datalab
|
||||||||||||||||||||||||||||
| Project |
datalab
|
| Branch Review |
bes/revision_history_clean_history
|
| Run status |
|
| Run duration | 07m 28s |
| Commit |
|
| Committer | Ben Smith |
| View all properties for this run ↗︎ | |
| Test results | |
|---|---|
Failures
|
0
|
Flaky
|
0
|
Pending
|
0
|
Skipped
|
0
|
Passing
|
336
|
| View all changes introduced in this branch ↗︎ | |
item_versions.refcode for finding history of one sample item_versions.user_id for user contributions to versions refcode and version number for ordered version history version_counters.refcode for version numbering
…ot at the time a version is made, i.e won't reflect changes to display name. Also has an user_id as an ObjectId that can be used for fast lookups and joins with the user collection
… restoring data. Added software version test
be-smith
requested review from
BenjaminCharmes,
jdbocarsly and
ml-evs
as code owners
ml-evs
reviewed
Nov 5, 2025
Comment on lines
+1041
to
+1049
| "_id": 1, | ||
| "timestamp": 1, | ||
| "user": 1, | ||
| "software_version": 1, | ||
| "version_number": 1, | ||
| "action": 1, | ||
| "restored_from_version": 1, | ||
| "data.version": 1, | ||
| }, |
Member
There was a problem hiding this comment.
Could you add a model for things that go in this collection?
As discussed, we really need to use the pydantic models wherever possible when operating on these.
ml-evs
reviewed
Nov 5, 2025
| "restored_from_version": str( | ||
| version_object_id | ||
| ), # Track which version was restored from | ||
| "user": user_snapshot, # Snapshot for fast display |
Member
There was a problem hiding this comment.
Suggested change
| "user": user_snapshot, # Snapshot for fast display |
As mentioned, I'd just store the ID then recreate it on egress via something like the creators_lookup method in this file which does an aggregation as:
def creators_lookup() -> dict:
return {
"from": "users",
"let": {"creator_ids": "$creator_ids"},
"pipeline": [
{"$match": {"$expr": {"$in": ["$_id", {"$ifNull": ["$$creator_ids", []]}]}}},
{"$addFields": {"__order": {"$indexOfArray": ["$$creator_ids", "$_id"]}}},
{"$sort": {"__order": 1}},
{"$project": {"_id": 1, "display_name": 1, "contact_email": 1}},
],
"as": "creators",
}| flask_mongo.db.item_versions.insert_one( | ||
| { | ||
| "refcode": refcode, | ||
| "version_number": next_version_number, |
Member
There was a problem hiding this comment.
Suggested change
| "version_number": next_version_number, | |
| "version": next_version_number, |
Comment on lines
+1234
to
+1236
| "restored_from_version": str( | ||
| version_object_id | ||
| ), # Track which version was restored from |
Member
There was a problem hiding this comment.
Suggested change
| "restored_from_version": str( | |
| version_object_id | |
| ), # Track which version was restored from | |
| "restored_from_version": ObjectId(version_object_id), # Track which version was restored from |
better to use the real ID in the database
| ), # Track which version was restored from | ||
| "user": user_snapshot, # Snapshot for fast display | ||
| "user_id": user_id, # ObjectId for efficient querying | ||
| "software_version": software_version, |
Member
There was a problem hiding this comment.
Suggested change
| "software_version": software_version, | |
| "datalab_version": software_version, |
Comment on lines
+1221
to
+1225
| # Get the software version | ||
| try: | ||
| software_version = get_package_version("datalab-server") | ||
| except PackageNotFoundError: | ||
| software_version = "unknown" |
Member
There was a problem hiding this comment.
Suggested change
| # Get the software version | |
| try: | |
| software_version = get_package_version("datalab-server") | |
| except PackageNotFoundError: | |
| software_version = "unknown" | |
| from pydatalab import __version__ | |
| software_version = __version__ |
better to use this, which defaults to develop if it can't find it
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add version control system for items
Closes #1057
Summary
This PR implements a version control system for datalab items (samples, cells, equipment, starting materials), enabling users to save, compare, and restore previous versions of their item pages.
Features
Core functionality
Data Model
API endpoints
POST /items/<refcode>/save-version/Manually save a version snapshot of the current item state.
{"status": "success", "version_number": 1, ...}GET /items/<refcode>/versions/List all versions for an item (sorted newest first).
{"status": "success", "versions": [...]}GET /items/<refcode>/versions/<version_id>/Get detailed data for a specific version.
{"status": "success", "version": {...}}GET /items/<refcode>/compare-versions/?v1=<id>&v2=<id>Compare two versions using DeepDiff.
{"status": "success", "diff": {...}, "v1_version_number": 1, "v2_version_number": 2}POST /items/<refcode>/restore-version/Restore item to a previous version (creates new version with action="restored").
{"version_id": "..."}{"status": "success", "restored_version": {...}, "new_version_number": 3}DELETE /items/<refcode>/versions/<version_id>/Delete a specific version snapshot.
{"status": "success", "message": "..."}Protected Fields on Restore
The following fields are protected during version restoration and will not be overwritten:
_id(MongoDB ObjectId)refcode(immutable identifier)last_modified(updated automatically)type(cannot change item type via restore)Automatic Versioning Integration
Version snapshots are automatically created when:
/new-sample/(action="created")/save-item/(action="manual_save")/restore-version/(action="restored")Database Optimization
item_versions.refcodefor fast version history lookupitem_versions.user_idfor user contribution queries(refcode, version_number)for sorted version historyversion_counters.refcodefor atomic version numberingUI Components (Currently Hidden)
A Vue.js
VersionHistoryModalcomponent has been implemented with:Dependencies
deepdiff>=7.0.0for nested structure comparisonFuture Work