Evaluation so far
Recently, I managed to experiment with many different formats of psychological evaluation. I took an interest in leveraging LLMs (namely Opus 4.5 due it’s systematic approach and chat wide context mapping) to interpret my BFI scores, provide me with a journaling framework which would extract higher-dimensionality insights and then leveraging those responses in new LLM instances for separate BFI scoring.
The results showed that the contextually derived responses (journal questionnaire) contained richer data which LLMs were able to leverage in their BFI scoring to provide more accurate scores (according to my interpretation) and crucially, be able to give the broader project (BASAIRA) directional queues which made more sense for my psychological profile - i.e. next steps and directions which are aligned with my long-term goals and philosophies. The additional layered insights which are embedded into the journal data provides richer insights which give deeper context for direction.
Thoughts and next steps
Considering the results and the promise in the ability of the smartest models to interpret ‘psycho-linguistic’ data such as journal entries, I would like to continue with my original plan of building a psycho-platform for which the rest of the BASAIRA learning framework could sit - as well as directional context on how best to design it.
Another remark I made was the difference in scoring styles and slight differences in verbiage and suggestions between models. I have not added structured frameworks for the models to follow in the form of skills or tools. I also left the temperature to 1 (in the gemini models where I could control it in the google-ai-studio interface). There were distinct traits and pattens in specific models, as well as model families. It is important to establish understandings of both model behaviours as well as desired model behaviours when interpreting qualitative data which becomes a foundation for decision making.
Analysing model psychology through similar (ANIMUS) frameworks also interests me and could provide novel insight.
How this affects and directs the BASAIRA project
The Basaira ethos and intentionality includes attempting to create a self-circulating learning system.
The ways in which I have been implementing ‘Basaira-like’ learning practices is as follows : I’ve been journaling, writing and documenting areas of my life and my projects as and when they progress and/or I discover new ideas. I tend to structure my workflow, projects and career generally - in a broad, unstructured, directionally-guided format in order to suit my observed learning patterns and corresponding states. Viewing myself as the primary driving system and orienting my external inputs and practises with the long-term goals in mind has also been critical. For example, retaining autonomy over the quality, type and format of content I ingest through eliminating algorithmic content suppliers such as Instagram reels and the YouTube homepage. This practice has allowed me to control the content I ingest and hence have more directional control over the thoughts I have and keep my mind focussed within the domains of interest to me. This has also meant that I’m having to provide myself with ‘balance’. Prioritising exercise, creative outputs and socialising has been crucial to prevent burn-out states. There also exists other key factors such as diet and supplements which are important, but I will not be visiting these concepts in depth in this article.
I want to expand on the functionality available to the interface layer of writing and spoken words, and conversation. From my experience, deepening understanding of a concept or area of exploration through written mediums such as ‘journal-like’ reflection or structured, intentional conversation has been very beneficial. I’ve been able to visualise and manipulate new learnings such that I strengthen the dimensionality of the new information in my mind.
LLMs have a few differentiating functionalities which make them a novel and new technology. One of these traits is their ability to understand semantic data, information which can only be ‘read from between the lines’. Previously, search engines like Google would encode semantic value through associative language, keywords, phrases etc. which link concepts through numerically prevalent means (if a word is frequently associated with a topic, internal algorithms would form links between the two). The advent of attention mechanisms and the Transformer architecture (the architecture of LLMs) has allowed semantic understanding to be encoded into the weights of the models (LLMs) themselves. All this to say that they can actually understand shit.
Most people intuitively pick up on this and smart users consciously architect the context window (the ‘chat’ - the sequence of messages between user and model (+ files etc.)). There is much research, deliberation and interest in optimising LLM performance by augmenting and manipulating the context from which the models generate their outputs. RAG (Retrieval Augmented Generation) and (hybrid) in-context based methods like Claude skills and MCP (Model Context Protocol) have come to be to boost the scope and depth of functionality of LLM agents in ‘real-world applications’. One of the most exciting and successful instances of intelligent use of context (and tools) are coding agents like CLI-based agents (i.e. Claude code), where the objective (‘design / build X according to Y spec’) is achieved through the agent’s navigation of what is available to it - navigating and adding data to it’s context, leveraging tools etc.
In order to attempt to build a system which enriches and supports learning experiences for user’s (through effective leverage of LLM agents), it is first important to understand user’s specific learning methods and general psychological patterns. The most effective method of increasing meaningful productive output for people is usually through implementing healthier habits and patterns - not a specific tool.
ANIMUS aims to create baseline and evolving user (psychological / habitual) profiles which can be used to direct model outputs and direct learning systems. Understanding how the user operates, what internal conflicts and ruminations they are currently having, as well as patterns which present themselves over time, can greatly improve the efficacy of a truly adaptive learning system.
Key challenges / obstacles in this approach
Psychological data encoding
Encoding psychological data is difficult in-of-in itself. The standard framework is the Big Five Index (ANIMUS leverages BFI2 - with further trait breakdowns) which scores the user across 5 broad traits (0-100) : Openness, Consensuses, Extraversion, Agreeableness, Neuroticism. See ANIMUS Methods for further project related breakdown of this framework and the OCEAN paper for the original paper detailing the framework.
So far, I’ve taken a version of the widely accepted BFI questionnaire, which asks the user to score themselves on various questions about themselves from Highly Disagree to Highly Agree, and I’ve also designed and utilised open-journaling questionnaires + LLM scoring to give BFI scores. See BFI self-evaluation for further insights into my baseline (BFI questionnaire) and ‘ANIMUS_v1’ (Agent) methods and scores.
In both methods, the original data: psychological profile, habits, thoughts, patterns, subconscious tendencies, goes through a self-evaluated rubric and is compressed to numerical values. The main value in ANIMUS, in leveraging open-ended reflective questioning is to retain and highlight the data-points which indicate those scores. These data-points are things such as habits, self-observed patterns and tendencies, ties of current personality to childhood events, all of which can be leveraged to predict future-behaviours.
Whilst filling out the open-ended evaluative questionnaire Baseline depth questionnaire, I naturally expressed evidence and/or explanation for my answers. For example :
- What topics or skills have you learned fastest? What made those different?
I learn and understand topics much better and faster than skills as a whole. I can injest and understand, deeply-understand nuanced areas of research and abstact concepts very quickly. When it comes to skills, I find it difficult to stay disciplined with the inputs sometimes and such make progress slower. There is sometimes a frustration associated with the lack of pace towards the desired level of profficiency. The gap between now and the future me appears ‘too-long’.
The question above asks for a somewhat objective data-point with relation to how I’ve learning in the past. Answering the question in a conversational tone with a low (ideally zero) filter results in a natural disclosing of explanations for certain behaviours. The section of answer above provides evidence to inform multiple facets of BFI indicators. From the question alone, if it were posed on a scale basis (as in the traditional BFI questionnaire), and phrased as such ‘I learn skills faster than topics. - Highly Disagree ←> Highly Agree, the question would likely correlate to my Openness to experience in some form of numerical weighting (in conjunction with other answer scores). The additional context I naturally provide : ‘There is sometimes a frustration associated with the lack of pace..’ gives contextual evidence which can also be used to score Neurotiscm (Negative Emotionality).
The broader interpretation of questionnaire data in the above format gives LLM agents more practical evidence for scoring. Removing the scoring altogether and encoding or adding this data to an agent’s context with a prompt related to aiding the user’s learning would likely increase the results of the assisted ‘learning session’.
Context Augmentation (/Limits)
Thousands of Tokens in the form of journal-like questionnaires cannot be provided for every query or ‘learning-session’ for efficiency and practicality reasons. A more practical approach would be to have a separate ‘weighting’ / model of the user profile which is still a compressed representation of their psychology (like the BFI scoring matrix), whilst encoding this additional semantic data which the LLMs could leverage. Similar to how RAG systems selectively pull semantically encoded data from Vector Databases (databases where data is mapped into something of a 3D space according to the underlying meaning). Leveraging the user psychological profile model selectively and accordingly, creating a layer between high-level, interface touchpoints for the user and low-level data stores (i.e. project data, notes, plans / goals) could be beneficial.
Despite all this speculation, much of what I detailed is either highly speculative and difficult to execute in practice, or may not provide much tangible benefit in practice. There also exists many different layers of the Basaira system and many interpretations of how to design it, each with different specific implications for the overall functionality.
The only logical way to approach the continuation of exploration of these topics and development of this (these) systems is to conduct structured experiments, document extensively and attempt various approaches with objective benchmarking. I will continue to research and experiment with different formats of designing data-collection systems to represent rich psychological user data, as well as ways in which compress this data whilst retaining the semantic information.
Approach Validation
As mentioned before, there must be a defined, systematic approach taken when moving forwards with experimenting with and validating the psychological profiling - as well as the broader learning system Basaira.
The use-case for ANIMUS is in this context it’s application as a layer to the Basaira learning system, hence, the ‘psychological profiling’ must be somewhat focussed on traits, habits and patterns in the user related to work and learning specifically. In the case of the Baseline depth questionnaire, I ideated with Claude Opus to ascertain a specific question rubric which would give insights for the user’s broader profile (mappable BFI values) as well as specifically in relation to uncovering reasoning and traits in relation to learning specifically.
When asking LLMs to interpret my responses to that set of questions, there was generally high level of convergence between LLMs on certain BFI facet scores, whilst other areas had higher levels of deviation between models. From Baseline depth questionnaire, the LLMs (two chats per model across Claude Opus 4.5, Gemini 3-Pro and 3-Flash previews (further details and data can be found in BFI self-evaluation)) managed to all converge on scores for intellectual curiosity with an standard deviation of 3.95 and a score range of 10. All 6 LLM analyses scored 92-98 with the exception of one outlier at 88. Similarly, Anxiety was scored consistently a standard deviation of 4.19 and a range of 10. Areas such as Responsibility, Respectfulness and Organisation were scored differently across models with Standard Deviations of 11.13, 9.57 and 9.23 respectively. Other interesting phenomena occurred from using this Baseline questionnaire + LLM method. Negative emotionality and Agreeableness were scored consistently above the standard BFI questionnaire and below the standard BFI questionnaire scores (respectively). From my anecdotal (subjective) perspective, the scores given for these areas by the LLM methods were more accurate to my psychological profile.
These patterns in the data demonstrate that the design of the question (questionnaire) prompts has direct implications into the accuracy of the scoring. And further, that the question set can be structured in such a way as to optimised to increase depth and quality of data related to certain areas. This is a useful insight as it means that there must be structured experimentation done into determining a method for designing the question set, with the specific outcome of highly dimensional data in relation to user learning behaviours in mind (optimising the broader Basaira system). In a later version of the ANIMUS framework, a layered question approach may be taken. Such framework would include an initial question set which would be answered by the user, and processed (and scored) by multiple instances of LLMs, followed by the systematic creation of a subsequent question set - focussing on areas of high standard deviation. Such a method may more reliably lead to a constant psychological profile, with low standard deviation in scoring across the various sub-facets.
Another interesting emergent behaviour from the data which became apparent was the differences in interpretation of the question set (Baseline depth questionnaire) across models and more tellingly, model providers. For this initial experimentation phase, I only ran this scoring / BFI evaluation phase (BFI self-evaluation) with 3 models : Claude Opus 4.5, Gemini 3-Pro-preview and Gemini-3-Pro-Flash. I ran each test in a blank context (new chat) with the entire Baseline depth questionnaire document pasted, followed by this (user) prompt :
Based on the user’s responses to the above questions, score the user in the BFI trait matrix with the following dimensions (0-100): Extraversion Sociability Assertiveness Energy Levels Agreeableness Compassion Respectfullness Trust Conscientiousness Organisation Productiveness Responsibility Negative Emotionality (/ Neurotiscm) Anxiety Depression Emotional Volatility Open-Mindedness Intellectual Curiosity Aesthetic Sensitivity Creative Imagination. Feel free to use any methods you wish (including computational methods if you need)
Opus 4.5 exhibited the most consistency across it’s scoring with an average points difference of 4.45 (taken across all facets between the two chats), followed by Gemini-3-Flash with an average points spread of 5.5 points and lastly 3-Pro with a points spread of 8. Further, more detailed analysis of certain Model and model-specific patterns and behaviours are explored and detailed in Inter-Model_Analysis. Another key pattern which developed was the similarity in scoring patterns between the two Gemini family models. Both models averaged a difference in scoring of 4.4 points across dimensions. The Gemini models both consistently adhered to a different scoring style to Opus 4.5. The scores for open-mindedness was consistently higher from the Gemini models than Opus 4.5. The differences in model ethos between model providers can be seen in the conversational responses provided after the BFI scores for each model. It is important to note that all experimentation was done within chat interfaces (Claude desktop and Google AI studio) with no pre-set System prompt and output structure.
Opus consistently used a neutral-to-positive framing. Language was generally data-centred and measured: “Emotionally self-aware and has built systems to manage this”, “the issue isn’t caring, it’s execution”. The model refrains from holding the user’s hand and pathologising. In the case of Gemini-3-Pro, the model has a more dramatic tone. The model assigned profile labels to the user such as ‘visionary / explorer’, “The Chaos Pilot”, “The Philosopher-Artist”. Flash-3 was consistently more technical, framework heavy (utilising and referencing psychological frameworks) and generally more clinical in it’s language. Flash-3 used language such as : “psychometric analysis”, “classic example of High Openness combined with High-Volatility Conscientiousness”, “Common signature for neurodivergent (ADHD) high-performers” and “Cognitive exoskeleton”. Flash 3’ and Opus 4.5’s consistent reference of terms related to clinical psychology as well as a consistent ‘technical-tone’, in comparison to 3-Pro’s more dramatic conversational tone implies that those model’s approached the task with a more structured intention and were more likely to produce more objective, evidence-based BFI scores.
These insights into the observed differences in model behaviour is important to consider when designing future psychological frameworks. Further analysis into specific model patterns must be done in order to try to establish a reliable scientific method. Another key facet of exploration is the effects of alternate model Temperature values (adherence to probability matrices (randomness)) as well as the use of crafted system prompts and tooling. The preliminary experimentation carried out only tested 3 models from 2 model providers. The companies’ intentions for the model and the model ethos is evident in their approach. Claude Opus is a ‘deep-thinking, systematic programming-focussed’ model, Gemini-3-Pro is Google’s flagship (deep-research, conversational) consumer facing model and Flash-3 is aggressively priced in the api and is likely positioned to take market share in the agentic application layer, where cost, speed and efficiency (in relation to intelligence) are all features of high value. Exploring models across various model providers, model sizes (and countries) is the natural next phase - in combination with a more systematic, programmatic testing approach (and larger datasets).
Approach to Basaira
The above ‘article’ has detailed and explained some of the intentionality and explanation of the initial experimentation phase of an early ANIMUS framework, as well as some loose ties and explanations of how it may potentially link to (and serve) the BASAIRA project in the future. I want to dedicate the following section to explaining the ethos and rough planning framework for the Basaira project.
For some of my unfiltered thoughts, read BASAIRA - Brain dump 16.12.25. It is an unpolished, brain-dump journal entry where I look into areas of my understandings of myself and how they manifest in the Basaira The philosophy of Basaira as a (learning-enabler) project is as follows: Learning (for me at least) should be a curiosity-driven endeavour where understanding can be developed through the forming, sculpting and re-moulding of concepts and ideas. Supporting curiosity-driven learning (particularly in ADHD type psych profiles) can be difficult to the inherent sporadic, unstructured nature of the pathways these individuals take when learning new things. I’ve found that I can understand topics quickly and deeply but struggle to find the discipline to develop skills due to emotional dysregulation. During periods where I am able to pursue ‘free-learning’, with limited pressures from external sources, I will progress multiple projects sporadically (cycling between them) and over time make general progress in all said endeavours.
Basaira aims to provide some underlying structure and directional cues for the user, whilst simultaneously logging, tracking and interpreting ‘progress’. This can be done through the use of Agents across a vault of documentation. Obsidian is a note-taking software which allows users to make cross-document connections.
How an agent would actually use ANIMUS profiles
Say you ask a learning agent to help you get deeper into ML fundamentals.
Without profile context, it might give you a structured curriculum with weekly milestones and progress tracking.
With profile context, it knows:
- You need the WHY first - so it explains what understanding ML fundamentals unlocks for your agentic applications work
- You’ll bounce if the gap feels too long - so it identifies the minimum viable depth that’s actually useful, not comprehensive mastery
- You resist rigid structure - so it offers a loose map of concepts rather than a schedule
- You learn through cycling - so it’s prepared for you to drop it and return, and frames that as normal rather than failure
- Your output is state-dependent - so if you mention you’ve been off-rhythm, it might suggest inputs before content
The profile doesn’t constrain the agent; it calibrates it.