Fix: SQL export reports success on failure #63

[monozoide]

🐞 Bug Summary

The --sql-export command incorrectly reports success even when data conversion errors occur, resulting in an invalid or empty SQL file being created. When values like 'N/A' cannot be converted to integers for NOT NULL columns, the current code logs a warning and proceeds silently, giving users a false sense that their export succeeded.

Severity: high

🔁 Reproduction

1. Run --sql-export on a dataset containing null-like values ('N/A', 'null', etc.) in NOT NULL integer columns
2. Observe that conversion errors are logged as warnings but processing continues
3. Verify that the command displays a success message despite conversion failures
4. Check that an invalid or incomplete SQL file was created

Actual result:
The command returns a success message and zero exit code, but creates an invalid SQL file with missing or improperly converted data. Conversion errors are silently suppressed.

Expected result:
The command should detect conversion errors for NOT NULL columns, fail immediately, provide appropriate error feedback, and clean up any incomplete SQL files before returning a failure status.

🌍 Runtime Context

• Affected version(s): 5.15.x
• Environment(s): PROD
• OS: Debian 13.1 - Trixie
• Python: 3.13, 3.12, 3.11

🕵️ Analysis & Root Cause

The root cause stems from overly permissive error handling in the data conversion pipeline:

• The format_sql_value function silently returns NULL when conversion fails, even for NOT NULL columns, without raising any error signal
• The generate_insert_statement function does not validate conversion success or propagate errors
• The run_sql_export function has no error aggregation mechanism, so it cannot detect when conversion failures have occurred
• Incomplete SQL files are never cleaned up, leaving invalid artifacts behind

This silent error suppression allows the process to report success while producing unusable data.

• Confirmed root cause: Insufficient validation in format_sql_value combined with lack of error tracking in run_sql_export

✅ Applied Fix

The fix implements strict error handling and validation throughout the conversion pipeline:

Changes to format_sql_value:

• Now checks if a column is NOT NULL
• Raises SQLExportError (new custom exception) when a null-like value ('N/A', 'null', etc.) is provided for a NOT NULL column
• Raises SQLExportError when type conversion (e.g., integer conversion) fails for a NOT NULL column
• Returns NULL only for nullable columns with invalid values

Changes to generate_insert_statement:

• Wraps format_sql_value calls to catch SQLExportError
• Re-raises caught exceptions with additional context (row data) for better debugging

Changes to run_sql_export:

• Added a conversion_errors counter initialized to 0
• Wrapped the main processing loop in try...except block to catch SQLExportError
• When an error is caught: logs the error with details, increments the error counter, and continues to the next row
• After the loop, checks if any errors occurred
• If conversion_errors > 0: logs a final error message, deletes the incomplete SQL file, and returns False
• Only reports success if the export completed without errors

This ensures strict validation of data conversions while providing comprehensive error reporting and cleanup.

• Affected modules: sql_exporter.py
• Risks/side effects:
  • Exports that previously appeared to succeed but contained conversion errors will now correctly fail and delete the incomplete file
  • Existing workflows or scripts relying on the previous (incorrect) behavior may need adjustment to handle failures appropriately
  • Performance impact: minimal (only additional validation checks and error tracking)
  • The stricter validation may surface previously hidden data quality issues in source datasets

🧪 Tests

Manual validation procedure:

1. Prepare a test dataset with a NOT NULL integer column containing the value 'N/A'
2. Run --sql-export on this dataset
3. Verify the command returns a non-zero exit code (failure)
4. Verify an error log message identifies the conversion failure with row details
5. Verify no SQL file (or an incomplete one) exists in the output directory
6. Prepare a valid dataset with all properly formatted values matching column types
7. Run --sql-export on the valid dataset
8. Expected result: Command returns zero exit code (success), produces a valid SQL file, no error logs

🔗 Links

Closes #63

---

✅ Project checklist

• All my commits are signed (GPG/SSH) in accordance with the contribution guide
• lint passes locally and in CI
• No sensitive data exposed (credentials, tokens)
• Manual validation completed with both valid and invalid datasets
• Security/performance impacts have been reviewed
• I have described a clear manual validation procedure