Persistent Storage Support (Apprise v1.9.0+) Added

Author: caronc

Dockerfile

@@ -47,17 +47,17 @@ FROM base AS runtime
 COPY ./requirements.txt /etc/requirements.txt
 COPY --from=builder /build/*.whl ./
 RUN set -eux && \
-    echo "Installing cryptography" && \
-        pip3 install *.whl && \
-    echo "Installing python requirements" && \
-        pip3 install --no-cache-dir -q -r /etc/requirements.txt gunicorn supervisor && \
     echo "Installing nginx" && \
         apt-get update -qq && \
         apt-get install -y -qq \
             nginx && \
+    echo "Installing cryptography" && \
+        pip3 install *.whl && \
     echo "Installing tools" && \
         apt-get install -y -qq \
-            curl sed && \
+            curl sed git && \
+    echo "Installing python requirements" && \
+        pip3 install --no-cache-dir -q -r /etc/requirements.txt gunicorn supervisor && \
     echo "Cleaning up" && \
         apt-get --yes autoremove --purge && \
         apt-get clean --yes && \

README.md

@@ -43,8 +43,10 @@ Using [dockerhub](https://hub.docker.com/r/caronc/apprise) you can do the follow
 docker pull caronc/apprise:latest
 
 # Start it up:
-# /config is used for a persistent store, you do not have to mount
-#         this if you don't intend to use it.
+# /config/store  is used for a persistent store, you do not have to mount
+#                this if you don't intend to use it.
+# /config is used for a spot to write all of the configuration files
+#         generated through the API
 # /plugin is used for a location you can add your own custom apprise plugins.
 #         You do not have to mount this if you don't intend to use it.
 # /attach is used for file attachments
@@ -90,6 +92,18 @@ docker run --name apprise \
    -e APPRISE_WORKER_COUNT=1 \
    -v /etc/apprise:/config \
    -d apprise/local:latest
+
+# Change your paths to what you want them to be, you may also wish to
+# just do the following:
+mkdir -p config store
+docker run --name apprise \
+   -p 8000:8000 \
+   -e PUID=$(id -u) \
+   -e PGID=$(id -g) \
+   -e APPRISE_STATEFUL_MODE=simple \
+   -e APPRISE_WORKER_COUNT=1 \
+   -v ./config:/config \
+   -d apprise/local:latest
 ```
 A `docker-compose.yml` file is already set up to grant you an instant production ready simulated environment:
 
@@ -125,6 +139,7 @@ Below is a sample of just a simple text response:
 # one or more of the following separated by a comma:
 #  - ATTACH_PERMISSION_ISSUE: Can not write attachments (likely a permission issue)
 #  - CONFIG_PERMISSION_ISSUE: Can not write configuration (likely a permission issue)
+#  - STORE_PERMISSION_ISSUE: Can not write to persistent storage (likely a permission issue)
 curl -X GET http://localhost:8000/status
 ```
 
@@ -383,6 +398,9 @@ The use of environment variables allow you to provide over-rides to default sett
 | `APPRISE_DEFAULT_THEME` | Can be set to `light` or `dark`; it defaults to `light` if not otherwise provided.  The theme can be toggled from within the website as well.
 | `APPRISE_DEFAULT_CONFIG_ID` | Defaults to `apprise`.   This is the presumed configuration ID you always default to when accessing the configuration manager via the website.
 | `APPRISE_CONFIG_DIR` | Defines an (optional) persistent store location of all configuration files saved. By default:<br/> - Configuration is written to the `apprise_api/var/config` directory when just using the _Django_ `manage runserver` script. However for the path for the container is `/config`.
+| `APPRISE_STORAGE_DIR` | Defines an (optional) persistent store location of all cache files saved. By default persistent storage is written into the `<APPRISE_CONFIG_DIR>/store`.
+| `APPRISE_STORAGE_MODE` | Defines the storage mode to use.  If no `APPRISE_STORGE_DIR` is identified, then this is set to `memory` in all circumtances reguardless what it might otherwise be set to. The possible options are:<br/>📌 **auto**: This is also the default. Writes cache files on demand only. <br/>📌 **memory**: Persistent storage is disabled; local memory is used for simple internal references. This is effectively the behavior of Apprise of versions 1.8.1 and earlier.<br/>📌 **flush**: A bit more i/o intensive then `auto`.  Content is written to disk constantly if changed in anyway. This mode is still experimental.
+| `APPRISE_STORAGE_UID_LENGTH` | Defines the unique key lengths used to identify an Apprise URL.  By default this is set to `8`.  Value can not be set to a smaller value then `2` or larger then `64`.
 | `APPRISE_ATTACH_DIR` | The directory the uploaded attachments are placed in. By default:<br/> - Attachments are written to the `apprise_api/var/attach` directory when just using the _Django_ `manage runserver` script. However for the path for the container is `/attach`.
 | `APPRISE_ATTACH_SIZE` | Over-ride the attachment size (defined in MB). By default it is set to `200` (Megabytes). You can set this up to a maximum value of `500` which is the restriction in place for NginX (internal hosting ervice) at this time.  If you set this to zero (`0`) then attachments will not be passed along even if provided.
 | `APPRISE_UPLOAD_MAX_MEMORY_SIZE` | Over-ride the in-memory accepted payload size (defined in MB). By default it is set to `3` (Megabytes). There is no reason the HTTP payload (excluding attachments) should exceed this limit.  This value is only configurable for those who have edge cases where there are exceptions to this rule.

apprise_api/api/management/__init__.py

@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2023 Chris Caron <lead2gold@gmail.com>
+# All rights reserved.
+#
+# This code is licensed under the MIT License.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files(the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and / or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions :
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+from django.core.management.base import BaseCommand
+from django.conf import settings
+import apprise
+
+
+class Command(BaseCommand):
+    help = f"Prune all persistent content older then {settings.APPRISE_STORAGE_PRUNE_DAYS} days()"
+
+    def add_arguments(self, parser):
+        parser.add_argument("-d", "--days", type=int, default=settings.APPRISE_STORAGE_PRUNE_DAYS)
+
+    def handle(self, *args, **options):
+        # Persistent Storage cleanup
+        apprise.PersistentStore.disk_prune(
+            path=settings.APPRISE_STORAGE_DIR,
+            expires=options["days"] * 86400, action=True,
+        )
+        self.stdout.write(
+            self.style.SUCCESS('Successfully pruned persistent storeage (days: %d)' % options["days"])
+        )

apprise_api/api/management/commands/__init__.py

@@ -351,8 +351,14 @@ <h4>{% trans "Persistent Store Endpoints" %}</h4>
     let code = document.createElement('code');
     let li = document.createElement('li');
     code.textContent = entry.url;
-    li.setAttribute('class', 'card-panel');
     li.appendChild(code);
+    li.setAttribute('class', 'card-panel');
+    if (entry.id) {
+      let url_id = document.createElement('code');
+      url_id.setAttribute('class', 'url-id');
+      url_id.textContent = entry.id;
+      li.appendChild(url_id);
+    }
 
     urlList.appendChild(li);
     // Store `all` tag

apprise_api/api/management/commands/storeprune.py

@@ -0,0 +1,35 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2023 Chris Caron <lead2gold@gmail.com>
+# All rights reserved.
+#
+# This code is licensed under the MIT License.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files(the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and / or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions :
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+import io
+from django.test import SimpleTestCase
+from django.core import management
+
+
+class CommandTests(SimpleTestCase):
+
+    def test_command_style(self):
+        out = io.StringIO()
+        management.call_command('storeprune', days=40, stdout=out)

apprise_api/api/templates/config.html

@@ -66,6 +66,7 @@ def test_healthcheck_simple(self):
             'config_lock': False,
             'attach_lock': False,
             'status': {
+                'persistent_storage': True,
                 'can_write_config': True,
                 'can_write_attach': True,
                 'details': ['OK']
@@ -90,6 +91,7 @@ def test_healthcheck_simple(self):
                 'config_lock': True,
                 'attach_lock': False,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': False,
                     'can_write_attach': True,
                     'details': ['OK']
@@ -113,6 +115,7 @@ def test_healthcheck_simple(self):
                 'config_lock': False,
                 'attach_lock': False,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': False,
                     'can_write_attach': True,
                     'details': ['OK']
@@ -136,6 +139,7 @@ def test_healthcheck_simple(self):
                 'config_lock': False,
                 'attach_lock': True,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': True,
                     'can_write_attach': False,
                     'details': ['OK']
@@ -159,6 +163,7 @@ def test_healthcheck_simple(self):
                 'config_lock': False,
                 'attach_lock': False,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': True,
                     'can_write_attach': True,
                     'details': ['OK']
@@ -172,6 +177,7 @@ def test_healthcheck_library(self):
 
         result = healthcheck(lazy=True)
         assert result == {
+            'persistent_storage': True,
             'can_write_config': True,
             'can_write_attach': True,
             'details': ['OK']
@@ -180,6 +186,7 @@ def test_healthcheck_library(self):
         # A Double lazy check
         result = healthcheck(lazy=True)
         assert result == {
+            'persistent_storage': True,
             'can_write_config': True,
             'can_write_attach': True,
             'details': ['OK']
@@ -192,35 +199,80 @@ def test_healthcheck_library(self):
             # We still succeed; we just don't leverage our lazy check
             # which prevents addition (unnessisary) writes
             assert result == {
+                'persistent_storage': True,
                 'can_write_config': True,
                 'can_write_attach': True,
                 'details': ['OK'],
             }
 
+        # Force a lazy check where we can't acquire the modify time
+        with mock.patch('os.path.getmtime') as mock_getmtime:
+            mock_getmtime.side_effect = OSError()
+            result = healthcheck(lazy=True)
+            # We still succeed; we just don't leverage our lazy check
+            # which prevents addition (unnessisary) writes
+            assert result == {
+                'persistent_storage': True,
+                'can_write_config': False,
+                'can_write_attach': False,
+                'details': [
+                    'CONFIG_PERMISSION_ISSUE',
+                    'ATTACH_PERMISSION_ISSUE',
+                ]}
+
         # Force a non-lazy check
         with mock.patch('os.makedirs') as mock_makedirs:
             mock_makedirs.side_effect = OSError()
             result = healthcheck(lazy=False)
             assert result == {
+                'persistent_storage': False,
                 'can_write_config': False,
                 'can_write_attach': False,
                 'details': [
                     'CONFIG_PERMISSION_ISSUE',
                     'ATTACH_PERMISSION_ISSUE',
+                    'STORE_PERMISSION_ISSUE',
                 ]}
 
-            mock_makedirs.side_effect = (None, OSError())
+            with mock.patch('os.path.getmtime') as mock_getmtime:
+                with mock.patch('os.fdopen', side_effect=OSError()):
+                    mock_getmtime.side_effect = OSError()
+                    mock_makedirs.side_effect = None
+                    result = healthcheck(lazy=False)
+                    assert result == {
+                        'persistent_storage': True,
+                        'can_write_config': False,
+                        'can_write_attach': False,
+                        'details': [
+                            'CONFIG_PERMISSION_ISSUE',
+                            'ATTACH_PERMISSION_ISSUE',
+                        ]}
+
+            with mock.patch('apprise.PersistentStore.flush', return_value=False):
+                result = healthcheck(lazy=False)
+                assert result == {
+                    'persistent_storage': False,
+                    'can_write_config': True,
+                    'can_write_attach': True,
+                    'details': [
+                        'STORE_PERMISSION_ISSUE',
+                    ]}
+
+            mock_makedirs.side_effect = (OSError(), OSError(), None, None, None, None)
             result = healthcheck(lazy=False)
             assert result == {
-                'can_write_config': True,
+                'persistent_storage': True,
+                'can_write_config': False,
                 'can_write_attach': False,
                 'details': [
+                    'CONFIG_PERMISSION_ISSUE',
                     'ATTACH_PERMISSION_ISSUE',
                 ]}
 
-            mock_makedirs.side_effect = (OSError(), None)
+            mock_makedirs.side_effect = (OSError(), None, None, None, None)
             result = healthcheck(lazy=False)
             assert result == {
+                'persistent_storage': True,
                 'can_write_config': False,
                 'can_write_attach': True,
                 'details': [

apprise_api/api/tests/test_cli.py

@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2024 Chris Caron <lead2gold@gmail.com>
+# All rights reserved.
+#
+# This code is licensed under the MIT License.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files(the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and / or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions :
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+import os
+import mock
+import tempfile
+from django.test import SimpleTestCase
+from .. import utils
+
+
+class UtilsTests(SimpleTestCase):
+
+    def test_touchdir(self):
+        """
+        Test touchdir()
+        """
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with mock.patch('os.makedirs', side_effect=OSError()):
+                assert utils.touchdir(os.path.join(tmpdir, 'tmp-file')) is False
+
+            with mock.patch('os.makedirs', side_effect=FileExistsError()):
+                # Dir doesn't exist
+                assert utils.touchdir(os.path.join(tmpdir, 'tmp-file')) is False
+
+            assert utils.touchdir(os.path.join(tmpdir, 'tmp-file')) is True
+
+            # Date is updated
+            assert utils.touchdir(os.path.join(tmpdir, 'tmp-file')) is True
+
+            with mock.patch('os.utime', side_effect=OSError()):
+                # Fails to update file
+                assert utils.touchdir(os.path.join(tmpdir, 'tmp-file')) is False
+
+    def test_touch(self):
+        """
+        Test touch()
+        """
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with mock.patch('os.fdopen', side_effect=OSError()):
+                assert utils.touch(os.path.join(tmpdir, 'tmp-file')) is False