From c6504c8d9e96bba65836fb9f1a4a429ddf171e94 Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 10:30:27 +0200 Subject: [PATCH 01/10] document high-level design PGM Signed-off-by: Martijn Govers --- .../high-level-design.md | 180 ++++++++++++++++++ docs/index.md | 1 + 2 files changed, 181 insertions(+) create mode 100644 docs/advanced_documentation/high-level-design.md diff --git a/docs/advanced_documentation/high-level-design.md b/docs/advanced_documentation/high-level-design.md new file mode 100644 index 000000000..9cf47823e --- /dev/null +++ b/docs/advanced_documentation/high-level-design.md @@ -0,0 +1,180 @@ + + +# High-level design + +The power-grid-model follows a typical dynamic/shared library approach, in which the user +interface is separated from the core implementation using a strict system boundary. Depending +on the use case and programming language used by the user to call the interface, the user can +opt to interface with the C API in different ways. + +## Layers of abstraction + +For the sake of explanation, we consider the +following layers of abstraction, in increasing order of ease-of-use and abstraction. + +* The raw interface layer + * System symbols only + * Everything is handled by the user, including which symbols are supported +* Exposition-only + * Exposition-only interface of the symbols in a language supported by the user + * Memory management and error handling is the responsibility of the user +* Simple wrapper + * Wraps the interface in a language supported by the user + * Handles memory management, basic error handling and type conversions + * Contains no additional features except maybe some basic utility tools +* The feature-rich layer + * Extensive wrapper around the interface with easy functionality exposure and utility functions + * Extensive type checks + +## Existing library interfaces + +The following library interfaces are currently included in the power-grid-model. + +| Interface type | Status | Layer | Explanation | Supported by | +| -------------- | ------------ | ------------------ | --------------------------------------------------------------- | --------------------------------------------------- | +| C API | Stable | Raw interface | Shared object / DLL that contains the core implementation | All programming languages with dynamic load support | +| C headers | Stable | Exposition-only | Exposition-only library using dynamic linking | C and C++ | +| C++ headers | Experimental | Wrapper | Handles memory management and basic error handling | C++ | +| Python library | Stable | Feature-rich layer | Library with useful functions, conversions and extensive checks | Python | + +Note that the Python library in turn also follows the pattern of a feature-rich library that uses a +module-internal wrapper layer core module, that wraps the exposition-only core module, that exposes +the raw interface. + +This can be visualized graphically as follows. + +```{mermaid} + :title: Public interfaces + +flowchart TD + classDef user_node fill:#9f6,stroke:#333,stroke-width:2px + classDef public_interface fill:#69f,stroke:#333,stroke-width:2px + classDef experimental_interface fill:#99b,stroke:#333,stroke-width:2px + classDef inclusion_method fill:#ddd,stroke:#fff,stroke-width:2px + + subgraph User + any_language_user(["Any language user"]):::user_node + c_user(["C user"]):::user_node + cpp_user(["C++ user"]):::user_node + python_user(["Python user"]):::user_node + end + + dynamic_loading{ }:::inclusion_method + c_includes{ }:::inclusion_method + + subgraph Raw interface + power_grid_model_c_dll("power_grid_model_c
(shared library)"):::public_interface + end + + subgraph Exposition + power_grid_model_c("power_grid_model_c
(C library)"):::public_interface + end + + subgraph Wrapper + power_grid_model_cpp("power_grid_model_cpp
(experimental,
C++ library)"):::experimental_interface + end + + subgraph Feature-rich library + power_grid_model_python("power_grid_model
(Python library)"):::public_interface + end + + any_language_user --> dynamic_loading + c_includes --> dynamic_loading + dynamic_loading -->|dynamic loading| power_grid_model_c_dll + c_user --> c_includes + cpp_user --> c_includes + c_includes -->|links +
includes| power_grid_model_c -->|dynamic linking| power_grid_model_c_dll + cpp_user -->|experimental
links +
includes| power_grid_model_cpp -->|links +
includes| power_grid_model_c + python_user -->|import| power_grid_model_python -->|"CDLL
(dynamic loading)"| power_grid_model_c_dll +``` + +## Full high-level design + +This can be visualized graphically as follows. + +```{mermaid} + :title: Full design + +flowchart TD + classDef user_node fill:#9f6,stroke:#333,stroke-width:2px + classDef public_interface fill:#69f,stroke:#333,stroke-width:2px + classDef experimental_interface fill:#99b,stroke:#333,stroke-width:2px + classDef private_interface fill:#999,stroke:#333,stroke-width:2px + classDef inclusion_method fill:#ddd,stroke:#fff,stroke-width:2px + + subgraph User + any_language_user(["Any language user"]):::user_node + c_user(["C user"]):::user_node + cpp_user(["C++ user"]):::user_node + python_user(["Python user"]):::user_node + end + + dynamic_loading{ }:::inclusion_method + c_includes{ }:::inclusion_method + + subgraph Raw interface + power_grid_model_c_dll("power_grid_model_c
(shared library)"):::public_interface + end + + subgraph Exposition + power_grid_model_c("power_grid_model_c
(C library)"):::public_interface + power_grid_core_python("power_grid_model._core
.power_grid_core.py
(exposition-only
Python module)"):::private_interface + end + + subgraph Wrapper + power_grid_model_cpp("power_grid_model_cpp
(experimental,
C++ library)"):::experimental_interface + power_grid_model_core_python("power_grid_model._core
(Python wrapper library)"):::private_interface + end + + subgraph Feature-rich library + power_grid_model_python("power_grid_model
(Python library)"):::public_interface + end + + any_language_user --> dynamic_loading + c_includes --> dynamic_loading + dynamic_loading -->|dynamic loading| power_grid_model_c_dll + c_user --> c_includes + cpp_user --> c_includes + c_includes -->|links +
includes| power_grid_model_c -->|dynamic linking| power_grid_model_c_dll + cpp_user -->|experimental
links +
includes| power_grid_model_cpp -->|links +
includes| power_grid_model_c + python_user -->|import| power_grid_model_python -->|internal import| power_grid_model_core_python -->|internal import| power_grid_core_python -->|"CDLL
(dynamic loading)"| power_grid_model_c_dll +``` + +## Creating a custom library or interface + +We seek to provide an optimal user experience, but with sheer amount of programming languages and +features, it would be impossible to provide a full feature-rich library for every single one. We, +being a [community-driven](#../contribution/GOVERNANCE.md) project strongly in favor of modern +software engineering practices, therefore encourage people to create their own libraries and +interfaces to improve their own experience. There are several possible reasons a user may want to +create their own library or interface, e.g.: + +* Support for a new programming language +* Extending library support for a specific programming language +* A custom wrapper that provides extra features or useful commands for specific custom requirements + +In all cases, it is recommended that the user determines their own desired +[layer of abstraction](#layers-of-abstraction) and then creates internal wrappers for all +lower-level ones, following the same pattern as the power-grid-model +[uses internally](#full-high-level-design) for the custom interfaces. + +### Hosting a custom library or interface + +The Power Grid Model organization supports people creating and hosting custom libraries and +interfaces. If you are doing so and are willing to notify us, please create an item on our +[discussion board](https://github.com/orgs/PowerGridModel/discussions) on GitHub. The Power Grid +Model organization will review your item and we may decide to mention your custom library on our +project website and documentation. + +### Contributing a custom library or interface + +When a custom library or interface becomes mature enough and the circumstances allow making it +publicly available, please consider contributing it to the Power Grid Model organization. If you are +considering contributing your custom library or interface, please read and follow our +[contributing guidelines](#../contribution/CONTRIBUTING.md) and open an item on our +[discussion board](https://github.com/orgs/PowerGridModel/discussions) on GitHub. The Power Grid +Model organization will review your item and contact you accordingly. diff --git a/docs/index.md b/docs/index.md index e15fb5dca..f465e4a54 100644 --- a/docs/index.md +++ b/docs/index.md @@ -108,6 +108,7 @@ algorithms/lu-solver advanced_documentation/native-data-interface advanced_documentation/build-guide advanced_documentation/c-api +advanced_documentation/high-level-design advanced_documentation/core-design advanced_documentation/python-wrapper-design ``` From 3241bfa083b70fe97a1e0e59024d6329bc90c2e3 Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 11:43:28 +0200 Subject: [PATCH 02/10] document high-level design PGM Signed-off-by: Martijn Govers --- docs/advanced_documentation/high-level-design.md | 14 +++++++------- docs/conf.py | 4 ++++ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/advanced_documentation/high-level-design.md b/docs/advanced_documentation/high-level-design.md index 9cf47823e..f0c66f89f 100644 --- a/docs/advanced_documentation/high-level-design.md +++ b/docs/advanced_documentation/high-level-design.md @@ -148,10 +148,10 @@ flowchart TD We seek to provide an optimal user experience, but with sheer amount of programming languages and features, it would be impossible to provide a full feature-rich library for every single one. We, -being a [community-driven](#../contribution/GOVERNANCE.md) project strongly in favor of modern -software engineering practices, therefore encourage people to create their own libraries and -interfaces to improve their own experience. There are several possible reasons a user may want to -create their own library or interface, e.g.: +being a {{ "[community-driven]({}/GOVERNANCE.md)".format(pgm_project_root) }} project strongly in +favor of modern software engineering practices, therefore encourage people to create their own +libraries and interfaces to improve their own experience. There are several possible reasons a user +may want to create their own library or interface, e.g.: * Support for a new programming language * Extending library support for a specific programming language @@ -175,6 +175,6 @@ project website and documentation. When a custom library or interface becomes mature enough and the circumstances allow making it publicly available, please consider contributing it to the Power Grid Model organization. If you are considering contributing your custom library or interface, please read and follow our -[contributing guidelines](#../contribution/CONTRIBUTING.md) and open an item on our -[discussion board](https://github.com/orgs/PowerGridModel/discussions) on GitHub. The Power Grid -Model organization will review your item and contact you accordingly. +{{ "[contributing guidelines]({}/CONTRIBUTING.md)".format(pgm_project_contribution) }} and open an +item on our [discussion board](https://github.com/orgs/PowerGridModel/discussions) on GitHub. The +Power Grid Model organization will review your item and contact you accordingly. diff --git a/docs/conf.py b/docs/conf.py index aa44d9edf..23c83d1f9 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -25,9 +25,13 @@ commit_version = git.Repo(search_parent_directories=True).head.object.hexsha link_head_gh_blob = link_head_gh + "blob/" + commit_version link_head_gh_tree = link_head_gh + "tree/" + commit_version + pgm_project_root = "#/" + pgm_project_contribution = pgm_project_root + "/contributing" else: link_head_gh_blob = "" link_head_gh_tree = "" + pgm_project_root = "https://github.com/PowerGridModel/" + pgm_project_contribution = pgm_project_root # -- General configuration --------------------------------------------------- From f52e27bb72303df66930bad8320800436f834612 Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 11:49:28 +0200 Subject: [PATCH 03/10] fix Signed-off-by: Martijn Govers --- docs/conf.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 23c83d1f9..5c2becd90 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -30,7 +30,7 @@ else: link_head_gh_blob = "" link_head_gh_tree = "" - pgm_project_root = "https://github.com/PowerGridModel/" + pgm_project_root = "https://github.com/PowerGridModel/.github/blob/main" pgm_project_contribution = pgm_project_root @@ -82,6 +82,8 @@ myst_substitutions = { "gh_link_head_blob": link_head_gh_blob, "gh_link_head_tree": link_head_gh_tree, + "pgm_project_root": pgm_project_root, + "pgm_project_contribution": pgm_project_contribution, } From a1db04de69a4260e808c2f1a71dfe04c208bb64c Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 11:55:57 +0200 Subject: [PATCH 04/10] minor Signed-off-by: Martijn Govers --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 5c2becd90..330e80981 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -25,7 +25,7 @@ commit_version = git.Repo(search_parent_directories=True).head.object.hexsha link_head_gh_blob = link_head_gh + "blob/" + commit_version link_head_gh_tree = link_head_gh + "tree/" + commit_version - pgm_project_root = "#/" + pgm_project_root = "/" pgm_project_contribution = pgm_project_root + "/contributing" else: link_head_gh_blob = "" From 3dbba1528727c0129452f441c467d89dc9688c16 Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 12:04:26 +0200 Subject: [PATCH 05/10] minor fix Signed-off-by: Martijn Govers --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 330e80981..221ee4abf 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -25,7 +25,7 @@ commit_version = git.Repo(search_parent_directories=True).head.object.hexsha link_head_gh_blob = link_head_gh + "blob/" + commit_version link_head_gh_tree = link_head_gh + "tree/" + commit_version - pgm_project_root = "/" + pgm_project_root = "" pgm_project_contribution = pgm_project_root + "/contributing" else: link_head_gh_blob = "" From a2838cc6e4bebb80fec25551047a9482cee5a65d Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 12:10:46 +0200 Subject: [PATCH 06/10] minor Signed-off-by: Martijn Govers --- docs/advanced_documentation/high-level-design.md | 2 +- docs/conf.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/advanced_documentation/high-level-design.md b/docs/advanced_documentation/high-level-design.md index f0c66f89f..809523257 100644 --- a/docs/advanced_documentation/high-level-design.md +++ b/docs/advanced_documentation/high-level-design.md @@ -148,7 +148,7 @@ flowchart TD We seek to provide an optimal user experience, but with sheer amount of programming languages and features, it would be impossible to provide a full feature-rich library for every single one. We, -being a {{ "[community-driven]({}/GOVERNANCE.md)".format(pgm_project_root) }} project strongly in +being a {{ "[community-driven]({}/GOVERNANCE.md)".format(pgm_project_contribution) }} project strongly in favor of modern software engineering practices, therefore encourage people to create their own libraries and interfaces to improve their own experience. There are several possible reasons a user may want to create their own library or interface, e.g.: diff --git a/docs/conf.py b/docs/conf.py index 221ee4abf..94fecf87e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -26,7 +26,7 @@ link_head_gh_blob = link_head_gh + "blob/" + commit_version link_head_gh_tree = link_head_gh + "tree/" + commit_version pgm_project_root = "" - pgm_project_contribution = pgm_project_root + "/contributing" + pgm_project_contribution = pgm_project_root + "/contribution" else: link_head_gh_blob = "" link_head_gh_tree = "" From e838b52610b14b9f3ea76dcf85c169e5e091cbc7 Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 12:19:03 +0200 Subject: [PATCH 07/10] minor improvements Signed-off-by: Martijn Govers --- docs/advanced_documentation/high-level-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/advanced_documentation/high-level-design.md b/docs/advanced_documentation/high-level-design.md index 809523257..af6cc1f71 100644 --- a/docs/advanced_documentation/high-level-design.md +++ b/docs/advanced_documentation/high-level-design.md @@ -20,7 +20,7 @@ following layers of abstraction, in increasing order of ease-of-use and abstract * System symbols only * Everything is handled by the user, including which symbols are supported * Exposition-only - * Exposition-only interface of the symbols in a language supported by the user + * Exposes the supported symbols in a language supported by the user * Memory management and error handling is the responsibility of the user * Simple wrapper * Wraps the interface in a language supported by the user From 883427b9c348a87b3d08c674e3ad4c6b3eb2734a Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Fri, 25 Apr 2025 12:19:57 +0200 Subject: [PATCH 08/10] minor Signed-off-by: Martijn Govers --- docs/advanced_documentation/high-level-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/advanced_documentation/high-level-design.md b/docs/advanced_documentation/high-level-design.md index af6cc1f71..0265f3e0b 100644 --- a/docs/advanced_documentation/high-level-design.md +++ b/docs/advanced_documentation/high-level-design.md @@ -16,7 +16,7 @@ opt to interface with the C API in different ways. For the sake of explanation, we consider the following layers of abstraction, in increasing order of ease-of-use and abstraction. -* The raw interface layer +* Raw system interface * System symbols only * Everything is handled by the user, including which symbols are supported * Exposition-only From 9b16de3777de4dbc00f21161699cd5ac40581e88 Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Mon, 28 Apr 2025 08:27:33 +0200 Subject: [PATCH 09/10] Update docs/advanced_documentation/high-level-design.md Signed-off-by: Martijn Govers Co-authored-by: Santiago Figueroa Manrique Signed-off-by: Martijn Govers --- docs/advanced_documentation/high-level-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/advanced_documentation/high-level-design.md b/docs/advanced_documentation/high-level-design.md index 0265f3e0b..6bf8cc200 100644 --- a/docs/advanced_documentation/high-level-design.md +++ b/docs/advanced_documentation/high-level-design.md @@ -146,7 +146,7 @@ flowchart TD ## Creating a custom library or interface -We seek to provide an optimal user experience, but with sheer amount of programming languages and +We seek to provide an optimal user experience, but with the sheer amount of programming languages and features, it would be impossible to provide a full feature-rich library for every single one. We, being a {{ "[community-driven]({}/GOVERNANCE.md)".format(pgm_project_contribution) }} project strongly in favor of modern software engineering practices, therefore encourage people to create their own From 18fe92b6eb7916c062ac7b2782a8b6965dd43c9b Mon Sep 17 00:00:00 2001 From: Martijn Govers Date: Tue, 29 Apr 2025 08:53:24 +0200 Subject: [PATCH 10/10] minor Signed-off-by: Martijn Govers --- .../high-level-design.md | 51 +------------------ 1 file changed, 1 insertion(+), 50 deletions(-) diff --git a/docs/advanced_documentation/high-level-design.md b/docs/advanced_documentation/high-level-design.md index 6bf8cc200..47b54ddde 100644 --- a/docs/advanced_documentation/high-level-design.md +++ b/docs/advanced_documentation/high-level-design.md @@ -47,55 +47,6 @@ the raw interface. This can be visualized graphically as follows. -```{mermaid} - :title: Public interfaces - -flowchart TD - classDef user_node fill:#9f6,stroke:#333,stroke-width:2px - classDef public_interface fill:#69f,stroke:#333,stroke-width:2px - classDef experimental_interface fill:#99b,stroke:#333,stroke-width:2px - classDef inclusion_method fill:#ddd,stroke:#fff,stroke-width:2px - - subgraph User - any_language_user(["Any language user"]):::user_node - c_user(["C user"]):::user_node - cpp_user(["C++ user"]):::user_node - python_user(["Python user"]):::user_node - end - - dynamic_loading{ }:::inclusion_method - c_includes{ }:::inclusion_method - - subgraph Raw interface - power_grid_model_c_dll("power_grid_model_c
(shared library)"):::public_interface - end - - subgraph Exposition - power_grid_model_c("power_grid_model_c
(C library)"):::public_interface - end - - subgraph Wrapper - power_grid_model_cpp("power_grid_model_cpp
(experimental,
C++ library)"):::experimental_interface - end - - subgraph Feature-rich library - power_grid_model_python("power_grid_model
(Python library)"):::public_interface - end - - any_language_user --> dynamic_loading - c_includes --> dynamic_loading - dynamic_loading -->|dynamic loading| power_grid_model_c_dll - c_user --> c_includes - cpp_user --> c_includes - c_includes -->|links +
includes| power_grid_model_c -->|dynamic linking| power_grid_model_c_dll - cpp_user -->|experimental
links +
includes| power_grid_model_cpp -->|links +
includes| power_grid_model_c - python_user -->|import| power_grid_model_python -->|"CDLL
(dynamic loading)"| power_grid_model_c_dll -``` - -## Full high-level design - -This can be visualized graphically as follows. - ```{mermaid} :title: Full design @@ -160,7 +111,7 @@ may want to create their own library or interface, e.g.: In all cases, it is recommended that the user determines their own desired [layer of abstraction](#layers-of-abstraction) and then creates internal wrappers for all lower-level ones, following the same pattern as the power-grid-model -[uses internally](#full-high-level-design) for the custom interfaces. +[uses internally](#existing-library-interfaces) for the custom interfaces. ### Hosting a custom library or interface