Why? What? Who? How? #1
Description
Why?
Having an append-only log is incredibly useful in way more situations than most people realise.
Anywhere you would need accountability in data is an excellent candidate for immutability.
- CRM - where customer data is updated and can be incorrectly altered, having the complete history of a record and being able to "time travel" through the change log is a really good idea.
- CMS/Blog - being able to "roll back" content means you can invite your trusted readers / stakeholders to edit/improve your content without "fear" of it decaying.
- E-Commerce - both for journey/story tracking and transaction reliability. Also, same applies for the Product catalog (which is a specific type of CMS); having version history dramatically increase confidence in the site both from an internal/vendor perspective and from end-users.
- This is especially useful for the Reviews on e-commerce sites/apps where we want to be able to detect/see where people have updated their review following extended usage. e.g: was did the product disintegrate after a short period of time? did the user give an initially unfaforable e.g 3/5 stars review and with time come to realise that the product is actually exceptionally durable, well-designed and great value-for-money because it has lasted twice a long as any previous product they purchased to perform the same "job to be done".
- Chat - a Chat system should allow editing of previously sent messages for typos/inaccuracies.
but that edit/revision history should be transparent not "message edited" (with no visibility of what changed) and if a person deletes the a message they should have to provide a comment indicating why they are "breaking" the chain. (more on this later). - Most other Consumer Web/Mobile Applications - you name the app, I can illustrate exactly how an append-only log is applicable/useful/essential to the reliability/confidence in that app.
- Forums - any sort of user-generate content where
- Social Networking - not allowing people to delete a message without leaving a clarifying comment promote accountability for what people write and in many cases avoids most hate speech.
When a system/db does not have (field/record level) "version control" each update over-writes the state of the record so it's impossible to retrieve it without having to go digging through a backup which is often a multi-day process, cost/time prohibitive or simply unavailable.
We propose that all apps should be built with an Append Only Log at the core by default.
This is not a "new" idea. Several large companies have used the "Lambda" or "Kappa" architecture in production with superb results for reliability and scalability.
see: http://milinda.pathirage.org/kappa-architecture.com
What?
Instead of using Ecto's standard "CRUD" which allows overwriting and deleting data without "rollback" or "recoverability", we propose writing a thin "proxy" layer between the application code and the PostgreSQL database
Who?
All developers who have basic understanding of web development where data is stored in a DB
and want to "level up" their knowledge/skills and the reliability of the product they are building
with a view to understanding more ("advanced") "distributed" application architecture
including the ability to (optionally/incrementally) use IPFS and Blockchain.
How?
The purpose of this tutorial is to:
- Make it easy for anyone to build a Phoenix/Ecto based app with an append-only log at it it's core.
- Write a step-by-step guide that shows how to:
- create a content type using standard Phoenix generators
- Take your pick of what you think is the simplest/most beginner friendly content type. e.g:
- Address Book is easy to show value of tracking updates.
- Take your pick of what you think is the simplest/most beginner friendly content type. e.g:
- create a content type using standard Phoenix generators
- no additional DB plugins/"add-ons" should be required to make this work,
just "stock" PostgreSQL as downloaded or available through a DB-as-a-service provider. e.g. Heroku.
- Add our tutorial to this thread once it's "ready": https://elixirforum.com/t/append-only-db/13355
- Add link to tutorial to: https://stackoverflow.com/questions/35405671/append-only-data-in-phoenix-ecto-and-postgres
Open Questions:
- How to reference the previous version of a record from the latest one. (keen to hear feedback on this) Happy to explore using hash of data as "parent id" thus we would have a "merkle tree" structure for all data. i.e. "Blockchain" but without the "proof of work" (factor), just a chain with the history of a record for accountability/rollback purposes no need to waste CPU cycles.
- How to
deletedata (mark an item as deleted) without "destroying" the data. - Note: we would still have to have a "batch process" that is able to "really delete" data to comply with GDPR/"Right to be forgotten". But from the User's perspective we only need to "unlink" the data in the UI and then the batch process will delete it after a specified expiry. Similar to the recycling bin on a Desktop OS.
Similar to: (please use these a reference when writing the doc(s))
- https://github.com/dwyl/phoenix-ecto-encryption-example [Quite Technical]
- https://github.com/dwyl/phoenix-chat-example [more Beginner Friendly]