New feature: Rollback the last migration

Author: RolandBende

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.5.2.
+
+### Feature
+
+- Rollback the last migration
+
+### Documentation
+
+- README.md
+    - Modified sections
+        - Getting Started
+        - Rolling Back Migrations
+
+
 ## 0.5.1
 
 ### Documentation

README.md

@@ -1,23 +1,29 @@
 # python-bigquery-migrations
 
-Python bigquery-migrations package is for creating and manipulating BigQuery databases easily.
+The `python-bigquery-migrations` package provides a streamlined way to create and manage BigQuery databases using intuitive CLI commands, such as the following:
+
+```bash
+bigquery-migrations run
+```
+
+**What are the benefits of using migrations?**
 
 Migrations are like version control for your database, allowing you to define and share the application's datasets and table schema definitions.
 
-## Prerequisite
+## Getting Started
+
+## 0. Prerequisite
 
 - Google Cloud Project with enabled billing
 - Enabled Google Cloud BigQuery API
 - Google Cloud Service Account JSON file
 
-## Getting Started
-
-## Install
+## 1. Install
 ```
 pip install bigquery-migrations
 ```
 
-## Create the project folder structure
+## 2. Create the project folder structure
 
 Create two subdirectory:
 1. credentials
@@ -30,10 +36,45 @@ your-project-root-folder
 └── ...
 ```
 
-## Create the neccessary files in the folders
+## 3. Create the neccessary files in the folders
+
+### Google Cloud Service Account JSON file
 
 Put your Google Cloud Service Account JSON file in the credentials subdirectory. See more info in the [Authorize BigQuery Client section](#authorize-bigquery-client)
 
+```
+your-project
+├── credentials
+│   ├── gcp-sa.json
+├── migrations
+└── ...
+```
+
+You can use different folder name and file name but in that case you must specify them with command arguments, such as the following:
+
+```bash
+bigquery-migrations run --gcp-sa-json-dir  my-creds --gcp-sa-json-fname my-service-account.json
+```
+
+|argument             |description                                                |
+|---------------------|-----------------------------------------------------------|              
+|--gcp-sa-json-dir    |Name of the service account JSON file directory (optional) |
+|--gcp-sa-json-fname  |Name of the service account JSON file (optional)           |
+
+> **IMPORTANT!**  
+> Never check the Google Cloud Service Account JSON file into version control. This file contains sensitive credentials that could compromise your Google Cloud account if exposed.
+
+To prevent accidental commits, make sure to add the file to your .gitignore configuration. For example:
+
+```bash
+# .gitignore
+gcp-sa.json
+```
+
+By ignoring this file, you reduce the risk of unintentional leaks and maintain secure practices in your repository.
+
+### Migrations
+
 Create your own migrations and put them in the migrations directory. See the [Migration structure section](#migration-structure) and [Migration naming conventions section](#migration-naming-conventions) for more info.
 
 ```
@@ -45,12 +86,14 @@ your-project
 └── ...
 ```
 
-You can use different folder names but in that case you must specify them with command arguments:
+You can use different folder name but in that case you must specify it with a command argument:
+
+```bash
+bigquery-migrations run --migrations-dir bq-migrations
+```
 
 |argument             |description                                                |
 |---------------------|-----------------------------------------------------------|              
-|--gcp-sa-json-dir    |Name of the service account JSON file directory (optional) |
-|--gcp-sa-json-fname  |Name of the service account JSON file (optional)           |
 |--migrations-dir     |Name of the migrations directory (optional)                |
 
 
@@ -97,19 +140,31 @@ The migration_log.json file content should look like this:
 
 ## Rolling Back Migrations
 
+### Rollback the last migration
+
+To reverse the last migration, execute the `rollback` command and pass `last` with the `--migration-name` argument:
+
+```bash
+bigquery-migrations rollback --migration-name last
+```
+
+### Rollback a specific migration
+
 To reverse a specific migration, execute the `rollback` command and pass the migration name with the `--migration-name` argument:
 
 ```bash
 bigquery-migrations rollback --migration-name 2024_12_10_121000_create_users_table
 ```
 
+### Rollback all migrations
+
 To reverse all of your migrations, execute the `reset` command:
 
 ```bash
 bigquery-migrations reset
 ```
 
-### Authorize BigQuery Client
+## Authorize BigQuery Client
 
 Put your service account JSON file in the credentials subdirectory in the root of your project.
 
@@ -120,7 +175,7 @@ your-project
 ...
 ```
 
-#### Creating a Service Account for Google BigQuery
+### Creating a Service Account for Google BigQuery
 
 You can connect to BigQuery with a user account or a service account. A service account is a special kind of account designed to be used by applications or compute workloads, rather than a person. Service accounts don’t have passwords and use a unique email address for identification.
 

justfile

@@ -38,8 +38,12 @@ migrations-list:
 migrations-run:
     python3 -m src.bigquery_migrations.migration_cli run
 
+# Rollback migration
+migrations-rollback-last:
+    python3 -m src.bigquery_migrations.migration_cli rollback --migration-name last
+
 # Rollback migrations
-migrations-rollback:
+migrations-rollback-specified:
     python3 -m src.bigquery_migrations.migration_cli rollback --migration-name 2024_12_10_121000_create_users_table
 
 # Rollback all migrations

src/bigquery_migrations/__init__.py

@@ -10,4 +10,4 @@
 __all__ = [
     "Migration"
 ]
-__version__ = "0.5.1"
+__version__ = "0.5.2"

src/bigquery_migrations/migration_cli.py

@@ -32,6 +32,13 @@ def run_migrations(migrator: MigrationManager):
     except Exception as e:
         print(f"Error during migration: {e}")
 
+def rollback_last_migration(migrator: MigrationManager):
+    try:
+        migrator.rollback_last()
+        print("Rollback process completed successfully.")
+    except Exception as e:
+        print(f"Error during rollback: {e}")
+    return
 
 def rollback_migration(migrator: MigrationManager, migration_name: str):
     try:
@@ -53,27 +60,38 @@ def reset_migrations(migrator: MigrationManager):
 def main():
     parser = argparse.ArgumentParser(description="Perform BigQuery migrations")
     parser.add_argument(
-        'command', choices=['list', 'run', 'rollback', 'reset'],
+        'command',
+        choices=['list', 'run', 'rollback', 'reset'],
+        type=str,
         help="Choose the operation to perform: list, run, rollback, reset"
     )
     parser.add_argument(
         '--gcp-sa-json-dir',
+        type=str,
+        required=False,
         help="Name of the service account JSON file directory (optional)"
     )
     parser.add_argument(
         '--gcp-sa-json-fname',
+        type=str,
+        required=False,
         help="Name of the service account JSON file (optional)"
     )
     parser.add_argument(
         '--migrations-dir',
+        type=str,
+        required=False,
         help="Name of the migrations directory (optional)"
     )
     parser.add_argument(
         '--gcp-project-id',
+        type=str,
+        required=False,
         help="Specify the Google Cloud Project ID (optional)"
     )
     parser.add_argument(
         '--migration-name',
+        type=str,
         help="Specify the name of the migration to rollback eg. 2024_01_02_120000_create_users_example (required for rollback command)"
     )
 
@@ -93,7 +111,10 @@ def main():
     elif args.command == 'rollback':
         if not args.migration_name:
             parser.error("The --migration-name argument is required for the rollback command.")
-        rollback_migration(migrator, args.migration_name)
+        if args.migration_name == 'last':
+            rollback_last_migration(migrator)
+        else:
+            rollback_migration(migrator, args.migration_name)
     elif args.command == 'reset':
         reset_migrations(migrator)
 

src/bigquery_migrations/migration_manager.py

@@ -105,6 +105,15 @@ def rollback(self, migration_name: str) -> Tuple[str, Optional[str]]:
             migration_name = None
         finally:
             return migration_name, prev_migration
+    
+    def rollback_last(self):
+        """Rollback the last applied migration"""
+        last_migration, last_timestamp = self.get_last_migration()
+        if not last_migration:
+            print(Fore.CYAN + "No migrations have been applied yet.")
+            return
+        return self.rollback(last_migration)
+
 
     def reset(self) -> list[Optional[str]]:
         """

tests/unit/migrations/2024_01_04_120000_create_another_table_example.py

@@ -0,0 +1,36 @@
+from google.cloud import bigquery
+from src.bigquery_migrations.migration import Migration
+
+class CreateAnotherTableExample(Migration):
+    """
+    See:
+    https://github.com/googleapis/python-bigquery/tree/main/samples
+    """
+
+    def up(self):
+        # TODO: Set table_id to the ID of the table to create.
+        # table_id = "your_project.your_dataset.example_table"
+        
+        # TODO: Define table schema
+        '''
+        schema = [
+            bigquery.SchemaField("id", "INTEGER", mode="REQUIRED"),
+            bigquery.SchemaField("name", "STRING", mode="REQUIRED"),
+            bigquery.SchemaField("created_at", "TIMESTAMP", mode="NULLABLE"),
+        ]
+        '''
+    
+        # table = bigquery.Table(table_id, schema=schema)
+        # table = self.client.create_table(table)
+        class_name = self.__class__.__name__
+        print(f"Class: {class_name}, Method: up")
+
+    def down(self):
+        # TODO: Set table_id to the ID of the table to fetch.
+        # table_id = "your_project.your_dataset.example_table"
+        
+        # If the table does not exist, delete_table raises
+        # google.api_core.exceptions.NotFound unless not_found_ok is True.
+        # self.client.delete_table(table_id, not_found_ok=True)
+        class_name = self.__class__.__name__
+        print(f"Class: {class_name}, Method: down")

tests/unit/migrations/migration_log.json

@@ -1 +1 @@
-{"last_migration": null, "timestamp": "2024-12-18T12:34:49.269255+00:00"}
+{"last_migration": null, "timestamp": "2024-12-23T16:12:39.282297+00:00"}

tests/unit/test_migration_dir_handler.py

@@ -23,7 +23,8 @@ def test_filename_list(self):
         expected_value = [
             "2024_01_01_120000_create_dataset_example",
             "2024_01_02_120000_create_table_example",
-            "2024_01_03_120000_create_table_from_json_schema_example"
+            "2024_01_03_120000_create_table_from_json_schema_example",
+            "2024_01_04_120000_create_another_table_example"
         ]
         current_value = under_test.filename_list()
         self.assertEqual(expected_value, current_value)