Merge pull request #164 from flask-dashboard/development

Development

Author: FlyingBird95

.gitignore

@@ -100,3 +100,6 @@ ENV/
 # Rope project settings
 .ropeproject
 
+# own scripts
+flask_monitoringdashboard/make_requests.py
+flask_monitoringdashboard/session_test.py

README.md

@@ -3,29 +3,37 @@
 [![Documentation Status](https://readthedocs.org/projects/flask-monitoringdashboard/badge/?version=latest)](http://flask-monitoringdashboard.readthedocs.io/en/latest/?badge=latest)
 [![codecov](https://codecov.io/gh/flask-dashboard/Flask-MonitoringDashboard/branch/master/graph/badge.svg)](https://codecov.io/gh/flask-dashboard/Flask-MonitoringDashboard)
 [![PyPI version](https://badge.fury.io/py/Flask-MonitoringDashboard.svg)](https://badge.fury.io/py/Flask-MonitoringDashboard)
-[![Gitter chat](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Flask-MonitoringDashboard/Lobby)
 [![Py-version](https://img.shields.io/pypi/pyversions/flask_monitoringdashboard.svg)](https://img.shields.io/pypi/pyversions/flask_monitoringdashboard.svg)
 [![Downloads](http://pepy.tech/badge/flask-monitoringdashboard)](http://pepy.tech/count/flask-monitoringdashboard)
 
 Dashboard for automatic monitoring of Flask web-services.
 
 The Flask Monitoring Dashboard is an extension that offers four main functionalities with little effort from the Flask developer:
-- **Monitor the Flask application:**
-Our Dashboard allows you to see which endpoints process a lot of request and how fast. 
-Additionally, it provides information about the evolving performance of an endpoint throughout different versions if you’re using git.
+
+- **Monitor the performance and utilization:**
+  The Dashboard allows you to see which endpoints process a lot of requests and how fast. 
+  Additionally, it provides information about the evolving performance of an endpoint throughout different versions if you're using git.
+
+- **Profile requests and endpoints:**
+  The execution path of every request is tracked and stored into the database. This allows you to gain
+  insight over which functions in your code take the most time to execute. Since all requests for an 
+  endpoint are also merged together, the Dashboard provides an overview of which functions are used in
+  which endpoint.
+
 - **Monitor your test coverage:**
-The dashboard allows you to find out which endpoints are covered by unit tests, allowing also for integration with Travis for automation purposes. 
-For more information, see [this file](http://flask-monitoringdashboard.readthedocs.io/en/latest/functionality.html#test-coverage-monitoring).
+  The Dashboard allows you to find out which endpoints are covered by unit tests, allowing also for integration with Travis for automation purposes. 
+  For more information, see [this file](http://flask-monitoringdashboard.readthedocs.io/en/latest/functionality.html#test-coverage-monitoring)
+
 - **Collect extra information about outliers:**
-Outliers are requests that take much longer to process than regular requests. 
-The dashboard automatically detects that a request is an outlier and stores extra information about it (stack trace, request values, Request headers, Request environment).
-- **Visualize the collected data in a number useful graphs:**
+  Outliers are requests that take much longer to process than regular requests. 
+  The Dashboard automatically detects that a request is an outlier and stores extra information about it (stack trace, request values, Request headers, Request environment).
+
 The dashboard is automatically added to your existing Flask application.
 You can view the results by default using the default endpoint (this can be configured to another route):
 
     [/dashboard](http://localhost:5000/dashboard)
 
-For a more advanced documentation, take a look at the information on [this site](http://flask-monitoringdashboard.readthedocs.io/en/latest/functionality.html).
+For more advanced documentation, take a look at the information on [this site](http://flask-monitoringdashboard.readthedocs.io/en/latest/functionality.html).
 
 ## Installation
 To install from source, download the source code, then run this:
@@ -46,7 +54,8 @@ Adding the extension to your Flask app is simple:
     dashboard.bind(app)
 
 ## Documentation
-For a more advanced documentation, see [this site](http://flask-monitoringdashboard.readthedocs.io).
+For more advanced documentation, see [this site](http://flask-monitoringdashboard.readthedocs.io).
+If you run into trouble migrating from version 1.X.X to version 2.0.0, this site also helps you solve this.
 
 ## Screenshots
 ![Screenshot 1](/docs/img/screenshot1.png)

config.cfg

@@ -0,0 +1,23 @@
+[dashboard]
+APP_VERSION=1.0
+GIT=/<path to your project>/.git/
+CUSTOM_LINK='dashboard'
+MONITOR_LEVEL=3
+OUTLIER_DETECTION_CONSTANT=2.5
+SAMPLING_PERIOD=20
+
+[authentication]
+USERNAME='admin'
+PASSWORD='admin'
+GUEST_USERNAME='guest'
+GUEST_PASSWORD='[dashboardguest!, second_pw!]'
+SECURITY_TOKEN='cc83733cb0af8b884ff6577086b87909'
+
+[database]
+TABLE_PREFIX=''
+DATABASE=sqlite:////<path to your project>/dashboard.db
+
+[visualization]
+TIMEZONE='Europe/Amsterdam'
+COLORS={'main':'[0,97,255]',
+        'static':'[255,153,0]'}

docs/changelog.rst

@@ -9,7 +9,28 @@ Unreleased
 ----------
 Changed
 
-- Restructuring of Test-Monitoring page
+- 
+
+
+v2.0.0
+----------
+Changed
+
+- Added a configuration option to prefix a table in the database
+
+- Optimize queries, such that viewing data is faster
+
+- Updated database scheme
+
+- Implemented functionality to customize time window of graphs
+
+- Implemented a profiler for Request profiling
+
+- Implemented a profiler for Endpoint profiling
+
+- Refactored current code, which improves readability
+
+- Refactoring of Test-Monitoring page
 
 - Identify testRun by Travis build number
 

docs/conf.py

@@ -28,7 +28,7 @@
 version = constants['version']
 
 release = version
-copyright = '{}, {}'.format(datetime.datetime.utcnow().year, author)
+copyright = '{}, {}. Version {}'.format(datetime.datetime.utcnow().year, author, version)
 
 # -- General configuration ---------------------------------------------------
 

docs/configuration.rst

@@ -1,6 +1,6 @@
 Configuration
 =============
-Once you have successfully installed the Flask Monitoring Dashboard with information from 
+Once you have successfully installed the Flask-MonitoringDashboard using the instructions from 
 `this page <installation.html>`_, you can use the advanced features by correctly configuring the Dashboard.
 
 Using a configuration file
@@ -33,8 +33,8 @@ Thus, it becomes:
    if __name__ == '__main__':
      app.run(debug=True)
 
-Instead of having a hard-coded string containing the location of the config file in the code above, it is also possible
-to define an environment variable that specifies the location of this config file.
+Instead of having a hard-coded string containing the location of the config file in the code above, 
+it is also possible to define an environment variable that specifies the location of this config file.
 The line should then be:
 
 .. code-block:: python
@@ -45,75 +45,95 @@ This will configure the Dashboard based on the file provided in the environment
 
 The content of the configuration file
 -------------------------------------
-Once the setup is complete, a configuration file (e.g. 'config.cfg') should be set next to the python file that 
-contains the entry point of the app. The following things can be configured:
+Once the setup is complete, a `configuration file`_ (e.g. 'config.cfg') should be set next to the python 
+file that contains the entry point of the app. The following properties can be configured:
+
+.. _`configuration file`: https://github.com/flask-dashboard/Flask-MonitoringDashboard/tree/master/config.cfg
 
 .. code-block:: python
 
    [dashboard]
    APP_VERSION=1.0
-   CUSTOM_LINK=dashboard
-   DATABASE=sqlite:////<path to your project>/dashboard.db
-   DEFAULT_MONITOR=True
-   TIMEZONE='Europe/Amsterdam'
-   USERNAME=admin
-   PASSWORD=admin
-   GUEST_USERNAME=guest
-   GUEST_PASSWORD=['dashboardguest!', 'second_pw!']
    GIT=/<path to your project>/.git/
+   CUSTOM_LINK='dashboard'
+   MONITOR_LEVEL=3
    OUTLIER_DETECTION_CONSTANT=2.5
-   OUTLIERS_ENABLED=True
+   SAMPLING_PERIOD=20
+
+   [authentication]
+   USERNAME='admin'
+   PASSWORD='admin'
+   GUEST_USERNAME='guest'
+   GUEST_PASSWORD='[dashboardguest!, second_pw!]'
    SECURITY_TOKEN='cc83733cb0af8b884ff6577086b87909'
-   TEST_DIR=/<path to your project>/tests/
+
+   [database]
+   TABLE_PREFIX=''
+   DATABASE=sqlite:////<path to your project>/dashboard.db
+
+   [visualization]
+   TIMEZONE='Europe/Amsterdam'
    COLORS={'main':'[0,97,255]',
            'static':'[255,153,0]'}
 
-This might look a bit overwhelming, but the following list explains everything in detail:
 
-- **APP_VERSION:** The version of the application that you use.
-  Updating the version helps in showing differences in execution times of a function over a period of time.
+As can be seen above, the configuration is split into 4 headers:
 
-- **CUSTOM_LINK:** The Dashboard can be visited at localhost:5000/{{CUSTOM_LINK}}.
+Dashboard
+~~~~~~~~~
 
-- **DATABASE:** Suppose you have multiple projects where you're working on and want to separate the results.
-  Then you can specify different database_names, such that the result of each project is stored in its own database.
+- **APP_VERSION:** The version of the application that you use.
+  Updating the version allows seeing the changes in the execution time of requests over multiple versions.
 
-- **DEFAULT_MONITOR:** When this configuration is set to True, new endpoints are automatically monitored by the Dashboard.
+- **GIT:** Since updating the version in the configuration-file when updating code isn't very convenient,
+  another way is to provide the location of the git-folder. From the git-folder,
+  the version is automatically retrieved by reading the commit-id (hashed value).
+  The specified value is the location to the git-folder. This is relative to the configuration-file.
 
-- **TIMEZONE:** The timezone for converting a UTC timestamp to a local timestamp. For a list of all
-  timezones, use the following:
+- **CUSTOM_LINK:** The Dashboard can be visited at localhost:5000/{{CUSTOM_LINK}}.
 
-  .. code-block:: python
+- **MONITOR_LEVEL**: The level for monitoring your endpoints. The default value is 3. For more information, see the
+  Rules page.
 
-     import pytz  # pip install pytz
-     print(pytz.all_timezones)
+- **OUTLIER_DETECTION_CONSTANT:** When the execution time is greater than :math:`constant * average`,
+  extra information is logged into the database. A default value for this variable is :math:`2.5`.
 
-  The dashboard saves the time of every request by default in a UTC-timestamp. However, if you want to display
-  it in a local timestamp, you need this property.
+- **SAMPLING_PERIOD:** Time between two profiler-samples. The time must be specified in ms.
+  If this value is not set, the profiler monitors continuously.
+
+Authentication
+~~~~~~~~~~~~~~
 
 - **USERNAME** and **PASSWORD:** Must be used for logging into the Dashboard.
   Thus both are required.
 
 - **GUEST_USERNAME** and **GUEST_PASSWORD:** A guest can only see the results, but cannot configure/download any data.
 
-- **GIT:** Since updating the version in the configuration-file when updating code isn't very useful,
-  it is a better idea to provide the location of the git-folder.
-  From the git-folder,
-  The version is automatically retrieved by reading the commit-id (hashed value).
-  The location is relative to the configuration-file.
+- **SECURITY_TOKEN:** The token that is used for exporting the data to other services. If you leave this unchanged,
+  any service is able to retrieve the data from the database.
 
-- **OUTLIER_DETECTION_CONSTANT:** When the execution time is more than this :math:`constant * average`,
-  extra information is logged into the database.
-  A default value for this variable is :math:`2.5`.
+Database
+~~~~~~~~
 
-- **OUTLIERS_ENABLED:** Whether you want to collect information about outliers. If you set this to true,
-  the expected overhead of the Dashboard is a bit larger, as you can find
-  `here <https://github.com/flask-dashboard/Testing-Dashboard-Overhead>`_.
+- **TABLE_PREFIX:** A prefix to every table that the Flask-MonitoringDashboard uses, to ensure that there are no
+  conflicts with the other tables, that are specified by the user of the dashboard.
 
-- **SECURITY_TOKEN:** The token that is used for exporting the data to other services. If you leave this unchanged,
-  any service is able to retrieve the data from the database.
+- **DATABASE:** Suppose you have multiple projects that you're working on and want to separate the results.
+  Then you can specify different database_names, such that the result of each project is stored in its own database.
+
+Visualization
+~~~~~~~~~~~~~
+
+- **TIMEZONE:** The timezone for converting a UTC timestamp to a local timestamp. For a list of all
+  timezones, use the following:
 
-- **TEST_DIR:** Specifies where the unit tests reside. This will show up in the configuration in the Dashboard.
+  .. code-block:: python
+
+     import pytz  # pip install pytz
+     print(pytz.all_timezones)
+
+  The dashboard saves the time of every request by default in a UTC-timestamp. However, if you want to display
+  it in a local timestamp, you need this property.
 
 - **COLORS:** The endpoints are automatically hashed into a color.
   However, if you want to specify a different color for an endpoint, you can set this variable.

docs/contact.rst

@@ -12,26 +12,26 @@ Currently, the team consists of three active developers:
    +------------+----------------------------------------------------------------------+
    | |Picture1| | | **Patrick Vogel**                                                  |
    |            | |                                                                    |
-   |            | | Core developer                                                     |
+   |            | | Project Leader                                                     |
    |            | |                                                                    |
    |            | | **E-mail:** `patrickvogel@live.nl <mailto:patrickvogel@live.nl>`_. |
    +------------+----------------------------------------------------------------------+
-   | |Picture2| | | **Thijs Klooster**                                                 |
+   | |Picture2| | | **Bogdan Petre**                                                   |
    |            | |                                                                    |
-   |            | | Test Monitor Specialist                                            |
+   |            | | Core Developer                                                     |
    +------------+----------------------------------------------------------------------+
-   | |Picture3| | | **Bogdan Petre**                                                   |
+   | |Picture3| | | **Thijs Klooster**                                                 |
    |            | |                                                                    |
-   |            | | Outlier Specialist                                                 |
+   |            | | Test Monitor Specialist                                            |
    +------------+----------------------------------------------------------------------+
 
 .. |Picture1| image:: https://avatars2.githubusercontent.com/u/17162650?s=460&v=4
               ..:width: 100px
 
-.. |Picture2| image:: https://avatars3.githubusercontent.com/u/17165311?s=400&v=4
+.. |Picture2| image:: https://avatars2.githubusercontent.com/u/7281856?s=400&v=4
               ..:width: 100px
 
-.. |Picture3| image:: https://avatars2.githubusercontent.com/u/7281856?s=400&v=4
+.. |Picture3| image:: https://avatars3.githubusercontent.com/u/17165311?s=400&v=4
               ..:width: 100px
 
 

docs/developing.rst

@@ -1,18 +1,18 @@
 Developing
 ==========
-This page provides information about contributing to the Flask Monitoring Dashboard.
+This page provides information about contributing to the Flask-MonitoringDashboard.
 Furthermore, a number of useful tools for improving the quality of the code are discussed.
 
 
 Implementation
 --------------
-The Dashboard is implemented in the following 6 folders: core, database, static, templates, test 
+The Dashboard is implemented in the following 6 directories: core, database, static, templates, test 
 and views. Together this forms a Model-View-Controller-pattern:
 
 - **Model**: The model consists of the database-code. To be more specific, it is defined in
   'Flask-MonitoringDashboard/database.__init__.py'.
 
-- **View**: The view is a combination of the following three folders:
+- **View**: The view is a combination of the following three directories:
 
   - **static**: contains some CSS and JS files.
 
@@ -27,13 +27,15 @@ and views. Together this forms a Model-View-Controller-pattern:
        fmd_base.html
        ├──fmd_dashboard/overview.html
        │  └──fmd_dashboard/graph.html
-       │     └──fmd_dashboard/graph-details.html
-       │        └──fmd_dashboard/outliers.html
+       │     ├──fmd_dashboard/graph-details.html
+       │     │  └──fmd_dashboard/outliers.html
+       │     ├──fmd_dashboard/profiler.html
+       │     │  └──fmd_dashboard/grouped_profiler.html
+       │     └──fmd_testmonitor/endpoint.html
        ├──fmd_testmonitor/testmonitor.html
-       ├──fmd_testmonitor/testresult.html
        ├──fmd_config.html
        ├──fmd_login.html
-       └──fmd_urles.html
+       └──fmd_rules.html
        fmd_export-data.html
 
 
@@ -61,7 +63,7 @@ and views. Together this forms a Model-View-Controller-pattern:
 
     - **fmd_testmonitor/testmonitor.html**: For rendering the `Testmonitor-page`_.
 
-    - **fmd_testmonitor/testresult.html**: For rendering the results of the Testmonitor.
+    - **fmd_testmonitor/endpoint.html**: For rendering the results of the Testmonitor.
 
     .. _`Configuration-page`: http://localhost:5000/dashboard/configuration
     .. _`Login-page`: http://localhost:5000/dashboard/login
@@ -136,6 +138,22 @@ The following tools are used for helping the development of the Dashboard:
   .. _Sphinx: www.sphinx-doc.org
   .. _ReadTheDocs: http://flask-monitoringdashboard.readthedocs.io
 
+Database Scheme
+---------------
+If you're interested in the data that the Flask-MonitoringDashboard stores, have a look at the database scheme below.
+
+Note the following:
+
+  - A key represents the Primary Key of the corresponding table. In the StackLine-table, the Primary Key consists 
+    of a combination of two fields (request_id and position).
+
+  - The blue arrow points to the Foreign Key that is used to combine the results of multiple tables. 
+
+.. figure :: img/database_scheme.png
+   :width: 100%
+
+
+
 Versions
 --------
 The Dashboard uses `Semantic-versioning`_. Therefore, it is specified in a **Major** . **Minor** . **Patch** -format: