-
Notifications
You must be signed in to change notification settings - Fork 24
The "Life Well‐Lived" Test
"Do things that don't scale"
There are numerous ways to test and evaluate AI models. Some are performance related (e.g. tokens, context that can be absorbed in a single query) and some are related to accuracy of responses, willingness to provide incorrect or dangerous prompt responses and guidelines. The evaluation of a model and AI application is contextual, and for something like Memory Cache, exploring what "good" performance looks like is a to-be-addressed research question that incorporates elements of synthesis, ideation, brainstorming, and connecting divergent thinking threads together.
One characteristic of human neurodivergence is "going from A to D"- a phrase used to describe the process of linking together seemingly unrelated topics, skipping the steps in-between that make it easier to understand how two seemingly distinct topics may be related. In neuroscience, the corresponding phenomenon is referred to as "synaptogenesis", the process through which discrete neurons form new neural pathways. At its core, Memory Cache aims to be a tool for facilitating personal synthesis of ideas, which requires a deep understanding of what makes a new insight powerful and useful.
In the future, this project will explore the ways of testing and evaluating that in more detail. For now, I've decided to stick with what I call the "Life Well-Lived" test, which is an observational, subjective, and qualitative evaluation of the current state of Memory Cache's software by how the system answers the prompt:
What is the meaning of a life well-lived?
The benchmark for this question is how well it reflects a unique perspective that references the files used to augment a given model and RAG implementation. At some point in the future, I will attempt to create a personal scoring mechanism that quantifies the responses.
The answer to this prompt, tested on an augmented version of the model and a non-augmented version, all other factors (hardware, development environment, prompt, embeddings, model) held consistent, can be seen below:
Augmented on Liv’s “memory” (as of Nov 11, 2023, 43.4MB of browser history and personal journal entries): A life well-lived involves authentic living in community with others through storytelling and technology use. It also emphasizes the importance of relationships such as love, friendship, and family members to build a meaningful existence.
Non-Augmented model: A life well-lived means living in accordance with one's values and principles while enjoying personal growth and fulfillment through experiences such as love, friendship, work, travel, learning new skills or hobbies, helping others, making positive contributions to society, etc. It is about finding joy in the journey of life rather than just focusing on achieving success at any cost.