Release 1.6.0

* [TTDB-831] fixed communication with pg_dump and pg_restore
Added:
- added a postgres version definition to use suitable pg_dump versions
- run option `--config` for configuration file with paths to specific versions of pg_dump/pg_restore utils

Updated:
- moved all modules with pg_anon modes into package `modes`
- actualized pyproject.toml
- tests. Function `anon_funcs.random_inn()` in tests replaced cause this function can't guarantee values unique

* [TTDB-831] pg_anon modes refactoring

Updated:
- modes `init` and `create-dict` packed into classes
- class `MainRoutine` decomposed on small methods
- class `PgAnonResult` now using only in `MainRoutine` and includes methods for updating statuses
- actualized tests

* [TTDB-831] Improving regexp searching

Added:
- for `skip_rules` and `include_rules` searching by `schema_mask` and `table_mask`

Updated:
- search in `data_regex.rules` for searching in text including `\n`
- search in `data_const.partial_constants` now case-insensitive
- actualized tests and README.md

* [TTDB-831] Optimization for reading data from database

* [TTDB-831] Excluded `view-fields` and `view-data` modes from checking postgres utils

* [TTDB-831] Updated phone regexp in examples and tests

* [TTDB-831] Added postgres version requirement into README.md

---------

Co-authored-by: Max Ibragimov <maxim.ibragimov@tantorlabs.ru>

Author: max-ibragimow

README.md

@@ -221,6 +221,8 @@ It uses the following tools:
 - PostgreSQL [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) tool for dumping the database structure.
 - PostgreSQL [`pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html) tool for restoring the database structure.
 
+**Warning:** Requires PostgreSQL 14 or higher
+
 ## Installation Guide
 
 ### Preconditions
@@ -266,6 +268,7 @@ Installation processes slightly differ depending on your operating system.
 ## Testing
 
 To test `pg_anon`, you need to have a local database installed. This section covers the installation of postgres and running the test suite.
+Your operating system also need have a locale `en_US.UTF-8`, because in tests creating database in this locale.
 
 ### Setting Up PostgreSQL
 
@@ -335,6 +338,7 @@ set TEST_DB_HOST=127.0.0.1
 set TEST_DB_PORT=5432
 set TEST_SOURCE_DB=test_source_db
 set TEST_TARGET_DB=test_target_db
+set TEST_CONFIG=/path/to/pg_anon/tests/config.yaml
 ```
 
 ## Usage
@@ -347,12 +351,13 @@ python pg_anon.py --help
 
 Common pg_anon options:
 
-| Option                          | Description                                                          |
-|---------------------------------|----------------------------------------------------------------------|
-| `--debug`                       | Enable debug mode (default false)                                    |
-| `--verbose`                     | Configure verbose mode: [info, debug, error] (default info)          |
-| `--db-connections-per-process`  | Amount of connections for IO operations for each process (default 4) |
-| `--processes`                   | Amount of processes for multiprocessing operations (default 4)       |
+| Option                         | Description                                                                       |
+|--------------------------------|-----------------------------------------------------------------------------------|
+| `--debug`                      | Enable debug mode (default false)                                                 |
+| `--verbose`                    | Configure verbose mode: [info, debug, error] (default info)                       |
+| `--db-connections-per-process` | Amount of connections for IO operations for each process (default 4)              |
+| `--processes`                  | Amount of processes for multiprocessing operations (default 4)                    |
+| `--config`                     | Path to the config file, where can be specified `pg_dump` and `pg_restore` utils. |
 
 Database configuration options:
 
@@ -409,7 +414,7 @@ python pg_anon.py --mode=create-dict \
 | `--output-sens-dict-file`      | Output file with sensitive fields will be saved to this value                                                                                                |
 | `--output-no-sens-dict-file`   | Output file with not sensitive fields will be saved to this value (Optional)                                                                                 |
 | `--scan-mode`                  | defines whether to scan all data or only part of it ["full", "partial"] (default "partial")                                                                  |
-| `--scan-partial-rows`          | In `--scan-mode partial` defines amount of rows to scan (default 10000)                                                                                      |
+| `--scan-partial-rows`          | In `--scan-mode partial` defines amount of rows to scan (default 10000). Actual rows count can be smaller after getting unique values                        |
 
 #### Requirements for input --meta-dict-file (metadict):
 
@@ -428,24 +433,24 @@ var = {
     },
     "skip_rules": [  # List of schemas, tables, and fields to skip
         {
-            # possibly some schema or table contains a lot of data that is not worth scanning. Skipped objects will not be automatically included in the resulting dictionary. Masks are not supported in this object.
-            "schema": "schm_mask_ext_exclude_2",  # Schema specification is mandatory
-            "table": "card_numbers",  # Optional. If there is no "table", the entire schema will be skipped.
+            # possibly some schema or table contains a lot of data that is not worth scanning. Skipped objects will not be automatically included in the resulting dictionary
+            "schema": "schm_mask_ext_exclude_2",  # Can use "schema" for full name matching or "schema_mask" for regexp matching. Required one of them
+            "table": "card_numbers",  # Optional. Can use "table" for full name matching or "table_mask" for regexp matching. If there is no "table"/"table_mask", the entire schema will be skipped.
             "fields": ["val_skip"]  # Optional. If there are no "fields", the entire table will be skipped.
         }
     ],
     "include_rules": [ # List of schemas, tables, and fields which will be scanning
         {
             # possibly you need specific fields for scanning or you can debug some functions on specific field
-            "schema": "schm_other_2", # Required. Schema specification is mandatory
-            "table": "tbl_test_anon_functions", # Optional. If there is no "table", the entire schema will be included.
-            "fields": ["fld_5_email"] # Optional. If there are no "fields", the entire table will be included.             
+            "schema": "schm_other_2",  # Can use "schema" for full name matching or "schema_mask" for regexp matching. Required one of them
+            "table": "tbl_test_anon_functions",  # Optional. Can use "table" for full name matching or "table_mask" for regexp matching. If there is no "table"/"table_mask", the entire schema will be skipped.
+            "fields": ["fld_5_email"]  # Optional. If there are no "fields", the entire table will be skipped.
         }
     ],
     "data_regex": {  # List of regular expressions to search for sensitive data
         "rules": [
-            """[A-Za-z0-9]+([._-][A-Za-z0-9]+)*@[A-Za-z0-9-]+(\.[A-Za-z]{2,})+""",  # email
-            "7?[\d]{10}"  # phone 7XXXXXXXXXX 
+            r"""[A-Za-z0-9]+([._-][A-Za-z0-9]+)*@[A-Za-z0-9-]+(\.[A-Za-z]{2,})+""",  # email
+            r"^(7?\d{10})$",				# phone 7XXXXXXXXXX
         ]
     },
     "data_const": {
@@ -545,15 +550,15 @@ var = {
 
 Possible options in mode=dump:
 
-| Option                         | Description                                                                                                                                |
-|--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
-| `--prepared-sens-dict-file`    | Input file or file list with sensitive fields, which was obtained in previous use by option `--output-sens-dict-file` or prepared manually |
-| `--dbg-stage-1-validate-dict`  | Validate dictionary, show the tables and run SQL queries without data export (default false)                                               |
-| `--dbg-stage-2-validate-data`  | Validate data, show the tables and run SQL queries with data export in prepared database (default false)                                   |
-| `--dbg-stage-3-validate-full`  | Makes all logic with "limit" in SQL queries (default false)                                                                                |
-| `--clear-output-dir`           | In dump mode clears output dict from previous dump or another files. (default true)                                                        |
-| `--pg-dump`                    | Path to the `pg_dump` Postgres tool (default `/usr/bin/pg_dump`).                                                                          |
-| `--output-dir`                 | Output directory for dump files. (default "")                                                                                              |
+| Option                        | Description                                                                                                                                |
+|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
+| `--prepared-sens-dict-file`   | Input file or file list with sensitive fields, which was obtained in previous use by option `--output-sens-dict-file` or prepared manually |
+| `--dbg-stage-1-validate-dict` | Validate dictionary, show the tables and run SQL queries without data export (default false)                                               |
+| `--dbg-stage-2-validate-data` | Validate data, show the tables and run SQL queries with data export in prepared database (default false)                                   |
+| `--dbg-stage-3-validate-full` | Makes all logic with "limit" in SQL queries (default false)                                                                                |
+| `--clear-output-dir`          | In dump mode clears output dict from previous dump or another files. (default true)                                                        |
+| `--pg-dump`                   | Path to the `pg_dump` Postgres tool (default `/usr/bin/pg_dump`).                                                                          |
+| `--output-dir`                | Output directory for dump files. (default "")                                                                                              |
 
 ### Run restore mode
 
@@ -721,6 +726,40 @@ from (
 	]
 ```
 
+### Configuring of pg_anon
+
+For specifying `pg_dump` and `pg_restore` utils, the parameters `--pg-dump` and `--pg-restore` can be used. 
+Also `--config` can be used for advanced configuring. This parameter accept YAML file in this format:
+```yaml
+pg-utils-versions:
+  <postgres_major_version>:
+    pg_dump: "/path/to/<postgres_major_version>/pg_dump"
+    pg_restore: "/path/to/<postgres_major_version>/pg_restore"
+  <another_postgres_major_version>:
+    pg_dump: "/path/to/<postgres_major_version>/pg_dump"
+    pg_restore: "/path/to/<postgres_major_version>/pg_restore"
+  default:
+      pg_dump: "/path/to/default_postgres_version/pg_dump"
+      pg_restore: "/path/to/default_postgres_version/pg_restore"
+```
+
+For example can be specified config for postgres 15, postgres 17:
+
+```yaml
+pg-utils-versions:
+  15:
+    pg_dump: "/usr/lib/postgresql/14/bin/pg_dump"
+    pg_restore: "/usr/lib/postgresql/14/bin/pg_restore"
+  17:
+    pg_dump: "/usr/lib/postgresql/17/bin/pg_dump"
+    pg_restore: "/usr/lib/postgresql/17/bin/pg_restore"
+  default:
+      pg_dump: "/usr/lib/postgresql/17/bin/pg_dump"
+      pg_restore: "/usr/lib/postgresql/17/bin/pg_restore"
+```
+
+In case of mismatch current postgres version with this config, will be used version of `pg_dump` and `pg_restore` from `default` section. For example `pg_anon` can be run with this config on postgres 16. In this case will be used `pg_dump 17` and `pg_restore 17`, i.e. from `default` section.
+
 ### Debug stages in dump and restore modes
 
 #### Debug stages:

pg_anon/common/db_queries.py

@@ -106,10 +106,13 @@ def get_data_from_field_query(field_info: FieldInfo, limit: int = None, conditio
     query_limit = get_limit_query(limit)
 
     query = f"""
-    SELECT distinct(substring(\"{field_info.column_name}\"::text, 1, 8196))
-    FROM \"{field_info.nspname}\".\"{field_info.relname}\"
-    {query_condition}
-    {query_limit}
+    SELECT distinct t1._field 
+    FROM (
+        SELECT (substring(\"{field_info.column_name}\"::text, 1, 8196)) as _field
+        FROM \"{field_info.nspname}\".\"{field_info.relname}\"
+        {query_condition}
+        {query_limit}
+    ) as t1
     """
 
     return query

pg_anon/common/db_utils.py

@@ -1,3 +1,4 @@
+import re
 from typing import Dict, List
 
 import asyncpg
@@ -200,7 +201,13 @@ async def run_query_in_pool(pool: Pool, query: str):
             logger.info(f"Execute query: {query}")
     except Exception as e:
         logger.error("Exception in run_query_in_pool:\n" + exception_helper())
-        raise Exception(f"Can't execute query: {query}")
+        raise RuntimeError(f"Can't execute query: {query}")
 
     logger.info(f"<================ Finished query {query}")
 
+
+async def get_pg_version(connection_params: ConnectionParams, server_settings: Dict = SERVER_SETTINGS):
+    db_conn = await create_connection(connection_params, server_settings=server_settings)
+    pg_version = await db_conn.fetchval("select version()")
+    await db_conn.close()
+    return re.findall(r"(\d+\.\d+)", str(pg_version))[0]

pg_anon/common/dto.py

@@ -1,4 +1,5 @@
 import json
+import time
 from dataclasses import dataclass
 from typing import Optional, Callable, Dict, List
 
@@ -9,7 +10,29 @@ class PgAnonResult:
     params = None  # JSON
     result_code = ResultCode.UNKNOWN
     result_data = None
-    elapsed = None
+    start_time = None
+    end_time = None
+    _elapsed = None
+
+    def start(self):
+        self.start_time = time.time()
+
+    def fail(self):
+        self.end_time = time.time()
+        self.result_code = ResultCode.FAIL
+
+    def complete(self):
+        self.end_time = time.time()
+        self.result_code = ResultCode.DONE
+
+    @property
+    def elapsed(self):
+        if not self._elapsed:
+            if self.start_time is None or self.end_time is None:
+                return None
+
+            self._elapsed = round(self.end_time - self.start_time, 2)
+        return self._elapsed
 
 
 @dataclass

pg_anon/common/multiprocessing_utils.py

@@ -14,7 +14,7 @@ async def init_process(name: str, ctx, target_func: Callable, tasks: List, *args
 
     p = aioprocessing.AioProcess(
         target=target_func,
-        args=(name, ctx, queue, tasks, *args),
+        args=(name, queue, tasks, *args),
         kwargs=kwargs,
     )
     p.start()

pg_anon/common/utils.py

@@ -3,12 +3,14 @@
 import hashlib
 import json
 import os.path
+import pathlib
 import re
 import subprocess
 import sys
 import traceback
 from typing import List, Optional, Dict, Union
 
+import yaml
 from pkg_resources import parse_version as version
 
 from pg_anon.common.db_utils import get_fields_list
@@ -56,6 +58,7 @@ def f(*args, **kwargs):
             func(*args, **kwargs)
         except:
             print(exception_helper(show_traceback=True))
+            raise
 
     return f
 
@@ -327,3 +330,14 @@ def get_folder_size(folder_path: str) -> int:
 
 def simple_slugify(value: str):
     return re.sub(r'\W+', '-', value).strip('-').lower()
+
+
+def read_yaml(file_path: str) -> Dict:
+    path = pathlib.Path(file_path)
+    if path.suffix not in ('.yml', '.yaml'):
+        raise ValueError("File must be .yml or .yaml")
+
+    with open(os.path.abspath(file_path), "r") as file:
+        data = yaml.safe_load(file)
+
+    return data

pg_anon/context.py

@@ -5,18 +5,18 @@
 from pg_anon.common.constants import ANON_UTILS_DB_SCHEMA_NAME, SERVER_SETTINGS
 from pg_anon.common.dto import ConnectionParams
 from pg_anon.common.enums import VerboseOptions, AnonMode, ScanMode
-from pg_anon.common.utils import (
-    exception_handler,
-    parse_comma_separated_list,
-)
+from pg_anon.common.utils import exception_handler, parse_comma_separated_list, read_yaml
 
 
 class Context:
     @exception_handler
     def __init__(self, args):
         self.current_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
         self.args = args
+        self.config = read_yaml(args.config) if args.config else None
         self.pg_version = None
+        self.pg_dump = args.pg_dump
+        self.pg_restore = args.pg_restore
         self.validate_limit = "LIMIT 100"
         self.meta_dictionary_obj: Dict = {}
         self.prepared_dictionary_obj: Dict = {}
@@ -220,9 +220,11 @@ def get_arg_parser():
             default=False,
         )
         parser.add_argument(
-            "--db-host",
-            type=str,
+            "--config",
+            help="Path to configuration file of pg_anon in YAML",
+            default=None,
         )
+        parser.add_argument("--db-host", type=str)
         parser.add_argument("--db-port", type=str, default="5432")
         parser.add_argument("--db-name", type=str, default="default")
         parser.add_argument("--db-user", type=str, default="default")
@@ -408,3 +410,26 @@ def get_arg_parser():
             help="Appends suffix for connection name. Just for comfortable automation",
         )
         return parser
+
+    def set_postgres_version(self, pg_version: str):
+        self.pg_version = pg_version
+        if not self.config:
+            return
+
+        pg_major_version = int(pg_version.split('.')[0])
+
+        utils_versions = self.config.get('pg-utils-versions')
+        pg_utils = utils_versions.get(pg_major_version)
+        if not pg_utils:
+            pg_utils = utils_versions.get('default')
+            if not pg_utils:
+                return
+
+        pg_dump = pg_utils.get('pg_dump')
+        pg_restore = pg_utils.get('pg_restore')
+
+        if not pg_dump or not pg_restore:
+            return ValueError("Config incorrect. Must be specified pg_dump and pg_restore utils paths")
+
+        self.pg_dump = pg_dump
+        self.pg_restore = pg_restore