Merge pull request #1 from xnuinside/v0.2.0_release

release 0.2.0 with support for Dataclasses

Author: xnuinside

.gitignore

@@ -5,4 +5,5 @@ py_models_parser/grammars/__pycache__
 dist/
 .pytest_cache/
 .vscode/
-models/
+models/
+test.py

.isort.cfg

@@ -1,2 +1,2 @@
 [settings]
-known_third_party = parsimonious
+known_third_party = gino,parsimonious,sqlalchemy

CHANGELOG.txt

@@ -1,2 +1,7 @@
+**v0.2.0**
+1. Added support for Dataclasses
+2. Added parse_from_file method
+3. Added correct work with types with comma inside, like: Union[dict, list] or Union[dict, list, tuple, anything] 
+
 **v0.1.1**
 1. Added base parser logic & tests for Pydantic, Enums, SQLAlchemy Models, GinoORM models, TortoiseORM models 

README.md

@@ -4,10 +4,10 @@
 
 
 It's as second Parser that done by me, first is a https://github.com/xnuinside/simple-ddl-parser for SQL DDL with different dialects.
-Py-Models-Parser supports now ORM Sqlalchemy, Gino, Tortoise; Pydantic, Python Enum models & in nearest feature I plan to add Dataclasses & pure pyton classes. And next will be added other ORMs models.
+Py-Models-Parser supports now ORM Sqlalchemy, Gino, Tortoise; Pydantic, Python Enum models, Dataclasses & in nearest feature I plan to add pure pyton classes. And next will be added other ORMs models.
 
+Py-Models-Parser written with PEG parser and it's python implementation - parsimonious. It's pretty new and I did not cover all possible test cases, so if you will have an issue  - please just open an issue in this case with example, I will fix it as soon as possible.
 
-Py-Models-Parser written with PEG parser and it's python implementation - parsimonious.
 Py-Models-Parser take as input different Python code with Models and provide output in standard form:
 
 ```python
@@ -55,7 +55,7 @@ You can parse models from python string:
 
 ```python
 
-from py_models_parser.core import parse
+from py_models_parser import parse
 
 models_str =  """from gino import Gino
 
@@ -74,6 +74,19 @@ class OrderItems(db.Model):
 result = parse(models_str)
 
 ```
+
+or just provide the path to file:
+
+```python
+
+    from py_models_parser import parse_from_file
+
+
+    file_path = "path/to/your/models.py"
+    # for example: tests/data/dataclass_defaults.py
+    result = parse_from_file(file_path)
+```
+
 It will produce the result:
 
 ```python
@@ -124,10 +137,15 @@ It will produce the result:
 
 1. Parse from file method
 2. Add cli
-3. Add more tests for supported models (and fix existed not covered cases): Pydantic, Enums, Dataclasses, SQLAlchemy Models, GinoORM models, TortoiseORM models 
+3. Add more tests for supported models (and fix existed not covered cases): Pydantic, Enums, Dataclasses, SQLAlchemy Models, GinoORM models, TortoiseORM models
 4. Add support for pure Python classes
 5. Add support for pure SQLAlchemy Core Tables
 
 ## Changelog
-**v0.1.0**
+**v0.2.0**
+1. Added support for Dataclasses
+2. Added parse_from_file method
+3. Added correct work with types with comma inside, like: Union[dict, list] or Union[dict, list, tuple, anything] 
+
+**v0.1.1**
 1. Added base parser logic & tests for Pydantic, Enums, SQLAlchemy Models, GinoORM models, TortoiseORM models 

docs/README.rst

@@ -2,10 +2,29 @@
 Py-Models-Parser
 ----------------
 
+
+.. image:: https://img.shields.io/pypi/v/py-models-parser
+   :target: https://img.shields.io/pypi/v/py-models-parser
+   :alt: badge1
+ 
+.. image:: https://img.shields.io/pypi/l/py-models-parser
+   :target: https://img.shields.io/pypi/l/py-models-parser
+   :alt: badge2
+ 
+.. image:: https://img.shields.io/pypi/pyversions/py-models-parser
+   :target: https://img.shields.io/pypi/pyversions/py-models-parser
+   :alt: badge3
+ 
+.. image:: https://github.com/xnuinside/py-models-parser/actions/workflows/main.yml/badge.svg
+   :target: https://github.com/xnuinside/py-models-parser/actions/workflows/main.yml/badge.svg
+   :alt: workflow
+
+
 It's as second Parser that done by me, first is a https://github.com/xnuinside/simple-ddl-parser for SQL DDL with different dialects.
-Py-Models-Parser supports now ORM Sqlalchemy, Gino, Tortoise; Pydantic, Python Enum models & in nearest feature I plan to add Dataclasses & pure pyton classes. And next will be added other ORMs models.
+Py-Models-Parser supports now ORM Sqlalchemy, Gino, Tortoise; Pydantic, Python Enum models, Dataclasses & in nearest feature I plan to add pure pyton classes. And next will be added other ORMs models.
+
+Py-Models-Parser written with PEG parser and it's python implementation - parsimonious. It's pretty new and I did not cover all possible test cases, so if you will have an issue  - please just open an issue in this case with example, I will fix it as soon as possible.
 
-Py-Models-Parser written with PEG parser and it's python implementation - parsimonious.
 Py-Models-Parser take as input different Python code with Models and provide output in standard form:
 
 .. code-block:: python
@@ -55,7 +74,7 @@ You can parse models from python string:
 .. code-block:: python
 
 
-   from py_models_parser.core import parse
+   from py_models_parser import parse
 
    models_str =  """from gino import Gino
 
@@ -73,6 +92,18 @@ You can parse models from python string:
        """
    result = parse(models_str)
 
+or just provide the path to file:
+
+.. code-block:: python
+
+
+       from py_models_parser import parse_from_file
+
+
+       file_path = "path/to/your/models.py"
+       # for example: tests/data/dataclass_defaults.py
+       result = parse_from_file(file_path)
+
 It will produce the result:
 
 .. code-block:: python
@@ -124,14 +155,21 @@ TODO: in next Release
 
 #. Parse from file method
 #. Add cli
-#. Add more tests for supported models (and fix existed not covered cases): Pydantic, Enums, Dataclasses, SQLAlchemy Models, GinoORM models, TortoiseORM models 
+#. Add more tests for supported models (and fix existed not covered cases): Pydantic, Enums, Dataclasses, SQLAlchemy Models, GinoORM models, TortoiseORM models
 #. Add support for pure Python classes
 #. Add support for pure SQLAlchemy Core Tables
 
 Changelog
 ---------
 
-**v0.1.0**
+**v0.2.0**
+
+
+#. Added support for Dataclasses
+#. Added parse_from_file method
+#. Added correct work with types with comma inside, like: Union[dict, list] or Union[dict, list, tuple, anything] 
+
+**v0.1.1**
 
 
 #. Added base parser logic & tests for Pydantic, Enums, SQLAlchemy Models, GinoORM models, TortoiseORM models 

py_models_parser/__init__.py

@@ -1 +1,3 @@
-__version__ = "0.1.0"
+from py_models_parser.core import parse, parse_from_file
+
+__all__ = ["parse", "parse_from_file"]

py_models_parser/core.py

@@ -1,3 +1,4 @@
+import os
 from typing import Dict, List
 
 from py_models_parser.grammar import grammar
@@ -25,7 +26,7 @@ def get_models_type(models_source: str) -> str:
 
 def pre_processing(models: str):
     models = models.split("\n")
-    start_statements = ["from", "import", "#", '"', "'"]
+    start_statements = ["from", "import", "#", '"', "'", "@"]
     inline_statements = ["Gino", "declarative_base"]
     to_process = []
     comment_start = True
@@ -62,7 +63,16 @@ def output(input: str):
 
 def parse(models: str) -> List[Dict]:
     models = pre_processing(models)
-    print(models)
     result = grammar.parse(models)
     result = output(result)
     return result
+
+
+def parse_from_file(file_path: str) -> List[Dict]:
+    if not os.path.isfile(file_path):
+        print(
+            f"Path {file_path} is not a file or not exists. You need to provide valid path to .py module with models"
+        )
+    with open(file_path, "r") as f:
+        models = f.read()
+        return parse(models)

py_models_parser/grammar.py

@@ -4,7 +4,7 @@
     r"""
     expr = (class / call_result / attr_def / emptyline)*
     class = class_def attr_def* ws?
-    class_def   = intend? class_name args? ":" ws?
+    class_def   = intend? class_name args? ":"* ws?
     attr_def  = intend? id ("=" right_part)* ws?
     right_part = args / call_result / id / string / text
     string = one_quote_str / double_quotes_str
@@ -13,7 +13,7 @@ class = class_def attr_def* ws?
     args     = "(" (( call_result / args / attr_def / id )* ","* )* ")"
     call_result = id args ws?
     class_name  = "class" id
-    id          = (dot_id / text) ws?
+    id          = (((dot_id / text)+ ","*) *  / dot_id / text) ws?
     dot_id      = (text".")*text
     intend      = "    " / "\t"
     text        = !class ~"['\_A-Z 0-9\{\}\[\]_\"\-\/\$:<%>\w]*"i

py_models_parser/visitor.py

@@ -6,7 +6,7 @@
 class Visitor(NodeVisitor):
     def visit_class_name(self, node, visited_children):
         """get class name"""
-        class_name = node.children[1].children[0].text.strip()
+        class_name = node.children[1].children[0].text.strip().replace(":", "")
         return {"name": class_name}
 
     def visit_class_def(self, node, visited_children):
@@ -58,7 +58,6 @@ def extract_orm_attr(self, text: str):
                 text = text.split(",")
 
                 text = self.clean_up_cases_with_inner_pars(text)
-                print(text)
                 if i == "Field":
                     # for tortoise orm
                     split_by_field = base_text.split("Field")[0].split(".")
@@ -122,38 +121,45 @@ def visit_attr_def(self, node, visited_children):
                         attr["attr"]["properties"] = children[-1][-1]["properties"]
                         if children[-1][-1]["type"] is not None:
                             attr["attr"]["type"] = children[-1][-1]["type"]
-
         return attr
 
-    def process_chld(self, child, final_child, meta):
+    def process_chld(self, child, final_child):
         if "attr" in child and child["attr"]["name"]:
             if "tablename" in child["attr"]["name"]:
                 final_child["properties"]["table_name"] = child["attr"]["default"]
-            elif "table_args" in child["attr"]["name"] or meta:
+            elif "table_args" in child["attr"]["name"]:
                 final_child["properties"][child["attr"]["name"]] = (
                     child["attr"]["type"] or child["attr"]["default"]
                 )
             else:
-                if "class Meta" == child["attr"]["name"]:
-                    return final_child, True
                 final_child["attrs"].append(child["attr"])
         else:
             if isinstance(child, dict):
                 final_child.update(child)
             elif isinstance(child, list):
                 for i in child:
-                    final_child, meta = self.process_chld(i, final_child, meta)
-        return final_child, meta
+                    final_child = self.process_chld(i, final_child)
+        return final_child
 
     def visit_expr(self, node, visited_children):
         """Makes a dict of the section (as key) and the key/value pairs."""
         children_values = []
+        n = -1
         for i in visited_children:
-            meta = False
             final_child = {"name": None, "attrs": [], "parents": [], "properties": {}}
-            final_child, _ = self.process_chld(i, final_child, meta)
-            if final_child["name"]:
+            final_child = self.process_chld(i, final_child)
+            if (
+                final_child.get("name")
+                and final_child["name"] == "Meta"
+                and children_values
+            ):
+                for attr in final_child["attrs"]:
+                    children_values[n]["properties"][attr["name"]] = (
+                        attr["type"] or attr["default"]
+                    )
+            elif final_child.get("name"):
                 children_values.append(final_child)
+                n += 1
             if "attr" in final_child:
                 del final_child["attr"]
         return children_values

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "py-models-parser"
-version = "0.1.0"
+version = "0.2.0"
 description = "Parser for Different Python Models (Pydantic, Enums, ORMs: Tortoise, SqlAlchemy, GinoORM) to extract information about columns(attrs), model, table args,etc in one format."
 authors = ["Iuliia Volkova <xnuinside@gmail.com>"]
 license = "MIT"