Testing Framework

[a5dur]

Test Suite Implementation for datapusher-plus

Create a complete, modern test suite for datapusher-plus (replace outdated test files) through GitHub Actions.
The implementation covers all core functionality from file processing to PostgreSQL operations, ensuring reliability and maintainability.

Core Test Modules

tests/test_jobs.py - Core job processing, validation, and datastore operations
tests/test_qsv_utils.py - QSV command wrapper and data processing operations
tests/test_plugin.py - CKAN plugin interface implementation
tests/test_helpers.py - Helper functions and utilities
tests/test_integration.py - End-to-end workflow testing

Infrastructure Files

tests/conftest.py - Pytest configuration and shared fixtures
tests/requirements-test.txt - Test dependencies
tests/README.md - Complete test documentation

GitHub Actions CI/CD

.github/workflows/testing_suite.yml - Automated testing pipeline

Automated Checks

Unit, integration, and performance tests
Code coverage reporting (Codecov)
Security scanning (Bandit, Safety)
Code quality checks (Flake8)

[minhajuddin2510]

Create a Testing Framework for datapusher plus to test uploading different types of files to datastore. Use github actions to test the extension.

[a5dur]

[14/08/2025]
Workflow for setting up a complete CKAN development environment with all necessary services (database, search, caching) and installs datapusher_plus extension done

[jqnatividad]

It'd be great if we can design the testing framework so that users can contribute test data and expected results.

We should test for both successes and failures.

The default position is that data quality is ALWAYS suspect.

Already, DP+ has a multi-step validation process where it:

1. Checks if the data is in the expected formats (csv and variants, spreadsheet, geojson, shapefile, zip)
2. If tabular data, checks if it is utf-8 encoded
3. converts the tabular data to normalized CSV
4. optionally checks if there are dupes
5. checks if the header names are valid Postgres/CKAN names

And only then, does it do the analysis proper. Otherwise, if validation fails, it exits gracefully with verbose, actionable error messages.

Still, errors may get through - especially for spatial formats, which is relatively new and now battle-tested.

[jqnatividad]

For the initial MVP release, prioritize using TX's test datasets.

[a5dur]

API parsing blocker See Run

Run echo "Creating demo organization..."
Creating demo organization...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   817  100   221  100   596  49629   130k --:--:-- --:--:-- --:--:--  199k
Organization creation response:
{
    "help": "http://localhost:5000/api/3/action/help_show?name=organization_create",
    "error": {
        "__type": "Authorization Error",
        "message": "Access denied: User None not authorized to create organizations"
    },
    "success": false
}
❌ Failed to create organization
❌ Authorization failed - API key might be invalid
API Key used: <Userid=cb24c116-c180-4003-a8a7-5051988b3b37name=ckan_adminpassword=$pbkdf2-sha512$25000$nJMyBqCUkjIGAGBsLaV0Lg$os6Sd2EomongmOvlYC8AzwMvJGY0eKVRDUHTPHgmKdxDZMEa3IpCbiWxvtEtqTSv17VHHKO8phoiMOBeUT9pmQfullname=CKANAdministratoremail=admin@example.comapikey=2912c59a-[85](https://github.com/a5dur/datapusher-plus/actions/runs/17067690191/job/48388673213#step:15:86)24-4acf-92dd-c4efc025f71ecreated=2025-08-1911
Error: Process completed with exit code 1.

[a5dur]

[21/08/2025]
Workflow sets-up CKAN environment successfully
To-do
Setup datapusher plus and run test files

[a5dur]

ckanapi is outputting warnings mixed with the JSON response.

Run set -eu
Creating sysadmin user (ckan_admin) if not exist...
/srv/app/src/ckan/ckan/plugins/core.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import iter_entry_points
/srv/app/src/ckan/ckan/cli/cli.py:7: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import iter_entry_points
2025-08-25 10:41:48,257 WARNI [ckan.common] Option lang is not declared
2025-08-25 10:41:48,532 WARNI [ckan.config.middleware.flask_app] Extensions are excluded from CSRF protection! We allow extensions to run without CSRF protection but it will be forced future releases. Read the documentation for more information on how to add CSRF protection to your extension.
2025-08-25 10:41:48,599 WARNI [ckan.common] Option ckanapi.log_local is not declared
{
  "about": "Created by GitHub Actions test",
  "activity_streams_email_notifications": false,
  "apikey": "3470001b-fde6-4788-98bf-5e2e64f93a72",
  "created": "2025-08-25T10:41:48.627092",
  "display_name": "CKAN Administrator",
  "email": "admin@example.com",
  "email_hash": "e64c7d89f26bd1972efa854d13d7dd61",
  "fullname": "CKAN Administrator",
  "id": "fa5630a9-426f-4e59-8244-f307e38bc872",
  "image_display_url": null,
  "image_url": null,
  "last_active": null,
  "name": "ckan_admin",
  "number_created_packages": 0,
  "state": "active",
  "sysadmin": false
}
/srv/app/src/ckan/ckan/cli/cli.py:7: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import iter_entry_points
2025-08-25 10:41:50,202 WARNI [ckan.common] Option lang is not declared
2025-08-25 10:41:50,477 WARNI [ckan.config.middleware.flask_app] Extensions are excluded from CSRF protection! We allow extensions to run without CSRF protection but it will be forced future releases. Read the documentation for more information on how to add CSRF protection to your extension.
Added ckan_admin as sysadmin
ckan_admin promoted to sysadmin
Creating API token for ckan_admin...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)

Leading to this error in resource creation

Exception: No API token found. API token required for DP+ service account.

[a5dur]

Solved blocker using
ckan config-tool /etc/ckan/default/ckan.ini "ckanext.datapusher_plus.api_token=$(ckan -c /etc/ckan/default/ckan.ini user token add CKAN_ADMIN dpplus | tail -n 1 | tr -d '\t')"

[a5dur]

[26/08/2025]
See http://github.com/a5dur/datapusher-plus/actions/runs/17237614999/job/48906105414 for run log

• Improved sysadmin user creation, promotion, and API key retrieval.
• Updated dataset and resource creation logic to be more reliable.
• Switched several steps (user details, CKAN status, dataset commands) to use the HTTP API instead of direct CLI calls because of mixing of warnings in the json response.
• Added support for creating and listing users in workflows.
• Implemented a manual trigger for datapusher-plus and automated API token generation for it.
• Cleaned up the workflow by removing the redundant final summary.
• Iterated on multiple fixes and adjustments to main.yml to stabilize the pipeline.

Next Steps:

• Verify datastore functionality (In Progress)
• Add testing files

[a5dur]

Currently Debugging Datastore Access
status: Ongoing
See run https://github.com/a5dur/datapusher-plus/actions/runs/17297734286/job/49100141177

Testing datastore functionality...
=== Testing datastore read access ===
Table metadata response: {"help": "http://localhost:5000/api/3/action/help_show?name=datastore_search", "success": true, "result": {"include_total": true, "limit": 100, "records_format": "objects", "resource_id": "_table_metadata", "total_estimation_threshold": null, "records": [], "fields": [{"id": "_id", "type": "int"}, {"id": "name", "type": "name"}, {"id": "oid", "type": "oid"}, {"id": "alias_of", "type": "name"}], "_links": {"start": "/api/3/action/datastore_search?resource_id=_table_metadata", "next": "/api/3/action/datastore_search?resource_id=_table_metadata&offset=100"}, "total": 0, "total_was_estimated": false}}
**✓ Datastore read access working**
=== Testing datastore write access ===
Test table creation response: {"help": "http://localhost:5000/api/3/action/help_show?name=datastore_create", "error": {"__type": "Authorization Error", "message": "Access denied: User  not authorized to edit package 13e9c76e-521a-478a-8fd6-c05d2dee8c86"}, "success": false}
**✗ Datastore write access failed**

most probably an api authorization issue

[a5dur]

Solved

Testing datastore functionality...
=== Testing datastore read access ===
Table metadata response: {"help": "http://localhost:5000/api/3/action/help_show?name=datastore_search", "success": true, "result": {"include_total": true, "limit": 100, "records_format": "objects", "resource_id": "_table_metadata", "total_estimation_threshold": null, "records": [], "fields": [{"id": "_id", "type": "int"}, {"id": "name", "type": "name"}, {"id": "oid", "type": "oid"}, {"id": "alias_of", "type": "name"}], "_links": {"start": "/api/3/action/datastore_search?resource_id=_table_metadata", "next": "/api/3/action/datastore_search?resource_id=_table_metadata&offset=100"}, "total": 0, "total_was_estimated": false}}
✓ Datastore read access working
=== Testing datastore write access ===
Test table creation response: {"help": "http://localhost:5000/api/3/action/help_show?name=datastore_create", "success": true, "result": {"delete_fields": false, "fields": [{"id": "test_col", "type": "text"}, {"id": "value", "type": "int"}], "resource": {"package_id": "my-first-dataset", "url": "_datastore_only_resource", "mimetype": null}, "resource_id": "806a6ca5-3acd-45dd-8c1f-5d8b1698b344", "method": "insert"}}
✓ Datastore write access working
=== Testing datastore query ===
Query response: {"help": "http://localhost:5000/api/3/action/help_show?name=datastore_search", "success": true, "result": {"include_total": true, "limit": 100, "records_format": "objects", "resource_id": "806a6ca5-3acd-45dd-8c1f-5d8b1698b344", "total_estimation_threshold": null, "records": [{"_id":1,"test_col":"hello","value":1},{"_id":2,"test_col":"world","value":2}], "fields": [{"id": "_id", "type": "int"}, {"id": "test_col", "type": "text"}, {"id": "value", "type": "int4"}], "_links": {"start": "/api/3/action/datastore_search?resource_id=806a6ca5-3acd-45dd-8c1f-5d8b1698b344", "next": "/api/3/action/datastore_search?resource_id=806a6ca5-3acd-45dd-8c1f-5d8b1698b344&offset=100"}, "total": 2, "total_was_estimated": false}}
=== Cleaning up test table ===
✓ Datastore functionality test completed successfully

[a5dur]

To-do:

Add Test Files