Inter-Model and Intra-Model BFI Scoring Analysis
Executive Summary
This analysis examines scoring consistency within model families (intra-model) and patterns across different AI models (inter-model) when analyzing the same baseline-depth questionnaire for BFI trait assessment.
1. INTRA-MODEL CONSISTENCY
How consistent was each model with itself across two independent chats?
Overall Consistency Rankings
| Model | Average Absolute Difference | Standard Deviation | Consistency Rating |
|---|---|---|---|
| Claude Opus 4.5 | 4.45 points | ~3.0 | ⭐⭐⭐ Highest |
| Flash 3-Preview | 5.50 points | ~4.2 | ⭐⭐ Moderate-High |
| Gemini 3-Pro | 8.00 points | ~6.3 | ⭐ Lowest |
Detailed Intra-Model Variance by Dimension
Claude Opus 4.5 (Chat 1 vs Chat 2)
| Dimension/Facet | C1 Score | C2 Score | Absolute Diff | Notes |
|---|---|---|---|---|
| DIMENSIONS | ||||
| Open-Mindedness | 85 | 86 | 1 | ✓ Extremely tight |
| Extraversion | 53 | 53 | 0 | ✓ Perfect agreement |
| Agreeableness | 48 | 42 | 6 | Moderate drift |
| Conscientiousness | 47 | 44 | 3 | ✓ Tight |
| Negative Emotionality | 58 | 65 | 7 | Moderate drift |
| FACETS | ||||
| Intellectual Curiosity | 88 | 92 | 4 | ✓ Tight |
| Aesthetic Sensitivity | 80 | 78 | 2 | ✓ Very tight |
| Creative Imagination | 87 | 82 | 5 | ✓ Acceptable |
| Sociability | 48 | 40 | 8 | Moderate |
| Assertiveness | 65 | 72 | 7 | Moderate |
| Energy Levels | 52 | 55 | 3 | ✓ Tight |
| Compassion | 55 | 48 | 7 | Moderate |
| Respectfulness | 40 | 35 | 5 | ✓ Acceptable |
| Trust | 52 | 52 | 0 | ✓ Perfect agreement |
| Organisation | 38 | 35 | 3 | ✓ Tight |
| Productiveness | 50 | 50 | 0 | ✓ Perfect agreement |
| Responsibility | 58 | 48 | 10 | ⚠️ Largest outlier |
| Anxiety | 65 | 68 | 3 | ✓ Tight |
| Depression | 50 | 58 | 8 | Moderate |
| Emotional Volatility | 58 | 65 | 7 | Moderate |
Opus Pattern:
- 3 perfect agreements (0 difference)
- 11 tight agreements (≤5 difference)
- 5 moderate drifts (6-10 difference)
- 1 outlier: Responsibility (10 point difference)
Gemini 3-Pro (Chat 1 vs Chat 2)
| Dimension/Facet | C1 Score | C2 Score | Absolute Diff | Notes |
|---|---|---|---|---|
| DIMENSIONS | ||||
| Open-Mindedness | 94 | 96 | 2 | ✓ Tight |
| Extraversion | 38 | 45 | 7 | Moderate |
| Agreeableness | 48 | 40 | 8 | Moderate-high |
| Conscientiousness | 42 | 28 | 14 | ⚠️ Large drift |
| Negative Emotionality | 62 | 72 | 10 | ⚠️ Large drift |
| FACETS | ||||
| Intellectual Curiosity | 98 | 98 | 0 | ✓ Perfect agreement |
| Aesthetic Sensitivity | 92 | 90 | 2 | ✓ Tight |
| Creative Imagination | 92 | 99 | 7 | Moderate |
| Sociability | 30 | 25 | 5 | ✓ Acceptable |
| Assertiveness | 65 | 75 | 10 | ⚠️ Large |
| Energy Levels | 50 | 35 | 15 | ⚠️ Very large drift |
| Compassion | 70 | 55 | 15 | ⚠️ Very large drift |
| Respectfulness | 20 | 15 | 5 | ✓ Acceptable |
| Trust | 55 | 50 | 5 | ✓ Acceptable |
| Organisation | 25 | 10 | 15 | ⚠️ Very large drift |
| Productiveness | 55 | 45 | 10 | ⚠️ Large |
| Responsibility | 45 | 30 | 15 | ⚠️ Very large drift |
| Anxiety | 65 | 75 | 10 | ⚠️ Large |
| Depression | 40 | 60 | 20 | 🚨 EXTREME outlier |
| Emotional Volatility | 80 | 80 | 0 | ✓ Perfect agreement |
Gemini Pattern:
- 2 perfect agreements
- 5 tight agreements (≤5 difference)
- 4 moderate drifts (6-10 difference)
- 8 large/very large drifts (11-20 difference)
- 1 extreme outlier: Depression (20 point difference)
Chat 2 Severity Bias: Gemini Chat 2 consistently scored MORE severely on negative traits:
- Conscientiousness: 42→28 (-14)
- Negative Emotionality: 62→72 (+10)
- Depression: 40→60 (+20)
- Organisation: 25→10 (-15)
Flash 3-Preview (Chat 1 vs Chat 2)
| Dimension/Facet | C1 Score | C2 Score | Absolute Diff | Notes |
|---|---|---|---|---|
| DIMENSIONS | ||||
| Open-Mindedness | 94 | 94 | 0 | ✓ Perfect agreement |
| Extraversion | 42 | 48 | 6 | Moderate |
| Agreeableness | 38 | 35 | 3 | ✓ Tight |
| Conscientiousness | 48 | 42 | 6 | Moderate |
| Negative Emotionality | 62 | 72 | 10 | ⚠️ Large drift |
| FACETS | ||||
| Intellectual Curiosity | 98 | 98 | 0 | ✓ Perfect agreement |
| Aesthetic Sensitivity | 88 | 88 | 0 | ✓ Perfect agreement |
| Creative Imagination | 96 | 96 | 0 | ✓ Perfect agreement |
| Sociability | 30 | 30 | 0 | ✓ Perfect agreement |
| Assertiveness | 82 | 85 | 3 | ✓ Tight |
| Energy Levels | 45 | 30 | 15 | ⚠️ Very large drift |
| Compassion | 55 | 50 | 5 | ✓ Acceptable |
| Respectfulness | 25 | 15 | 10 | ⚠️ Large |
| Trust | 40 | 40 | 0 | ✓ Perfect agreement |
| Organisation | 22 | 22 | 0 | ✓ Perfect agreement |
| Productiveness | 58 | 48 | 10 | ⚠️ Large |
| Responsibility | 65 | 55 | 10 | ⚠️ Large |
| Anxiety | 68 | 75 | 7 | Moderate |
| Depression | 40 | 65 | 25 | 🚨 EXTREME outlier |
| Emotional Volatility | 78 | 78 | 0 | ✓ Perfect agreement |
Flash Pattern:
- 7 perfect agreements (highest count)
- 4 tight agreements (≤5 difference)
- 3 moderate drifts (6-10 difference)
- 5 large drifts (11-25 difference)
- 1 extreme outlier: Depression (25 point difference - highest in entire dataset)
Interesting: Flash was PERFECT on all Open-Mindedness facets (98, 88, 96 × 2), but highly variable on Depression and Energy.
Key Intra-Model Findings
1. Depression is Universally Unstable
All three models showed their LARGEST variance on Depression:
- Opus: 8 points
- Gemini: 20 points
- Flash: 25 points
Hypothesis: The questionnaire provided ambiguous data on depression—mentions “negative spirals” and “shame” but lacks frequency, duration, or clinical severity markers.
2. Claude Opus = Most Conservative and Consistent
- Smallest average drift (4.45)
- No extreme outliers (max 10 points)
- 3 perfect agreements vs. 2 (Gemini) and 7 (Flash)
3. Gemini Chat 2 Shows “Severity Bias”
- Consistently rated MORE negatively in Chat 2
- Depression jumped +20, Conscientiousness dropped -14
- More dramatic/extreme scoring overall
4. Flash Shows “Anchoring on Core Traits”
- Perfect consistency on Open-Mindedness (0 difference across all 3 facets)
- Perfect consistency on Organisation and Trust
- But HIGHLY variable on Depression (+25) and Energy (+15)
2. INTER-MODEL PATTERNS
How did different models score compared to each other?
Average Scores by Model Family
| Dimension | Opus Avg | Gemini Avg | Flash Avg | Google Avg (Gemini + Flash) |
|---|---|---|---|---|
| Open-Mindedness | 85.5 | 95.0 | 94.0 | 94.5 |
| Extraversion | 53.0 | 41.5 | 45.0 | 43.3 |
| Agreeableness | 45.0 | 44.0 | 36.5 | 40.3 |
| Conscientiousness | 45.5 | 35.0 | 45.0 | 40.0 |
| Negative Emotionality | 61.5 | 67.0 | 67.0 | 67.0 |
Pattern 1: Google Models Score Open-Mindedness ~9 Points Higher
| Model | Open-Mindedness Score |
|---|---|
| Claude Opus C1 | 85 |
| Claude Opus C2 | 86 |
| Opus Average | 85.5 |
| Gemini 3-Pro C1 | 94 |
| Gemini 3-Pro C2 | 96 |
| Flash 3-Preview C1 | 94 |
| Flash 3-Preview C2 | 94 |
| Google Average | 94.5 |
| Difference | +9.0 points |
Breakdown by Facet:
| Facet | Opus Avg | Google Avg | Δ |
|---|---|---|---|
| Intellectual Curiosity | 90.0 | 98.0 | +8.0 |
| Aesthetic Sensitivity | 79.0 | 89.5 | +10.5 |
| Creative Imagination | 84.5 | 95.8 | +11.3 |
Google models consistently scored ALL Open-Mindedness facets higher, with Creative Imagination showing the largest gap (+11.3).
Pattern 2: Google Models Score Extraversion ~10 Points Lower
| Model | Extraversion Score |
|---|---|
| Claude Opus Avg | 53.0 |
| Google Avg | 43.3 |
| Difference | -9.7 points |
Specifically driven by:
- Sociability: Opus avg 44 vs. Google avg 27.5 (-16.5)
- Energy Levels: Opus avg 53.5 vs. Google avg 40 (-13.5)
Pattern 3: Google Models Score Negative Emotionality Higher
| Model | Negative Emotionality |
|---|---|
| Claude Opus Avg | 61.5 |
| Google Avg | 67.0 |
| Difference | +5.5 points |
Driven primarily by:
- Emotional Volatility: Opus avg 61.5 vs. Google avg 78.5 (+17)
Pattern 4: Gemini Models Are Similar to Each Other
Correlation Analysis: Gemini 3-Pro vs Flash 3-Preview
| Dimension | Gemini Avg | Flash Avg | Absolute Diff |
|---|---|---|---|
| Open-Mindedness | 95.0 | 94.0 | 1.0 |
| Extraversion | 41.5 | 45.0 | 3.5 |
| Agreeableness | 44.0 | 36.5 | 7.5 |
| Conscientiousness | 35.0 | 45.0 | 10.0 |
| Negative Emotionality | 67.0 | 67.0 | 0.0 |
Average inter-Google difference: 4.4 points (very similar)
Gemini and Flash show strong agreement, except on Conscientiousness (10 point gap) where Gemini scored more severely (35 vs 45).
Pattern 5: Specific Directional Biases
Google Models Consistently Score Higher On:
- Open-Mindedness (+9.0)
- Assertiveness (+5.8)
- Emotional Volatility (+17.0)
- Creative Imagination (+11.3)
Google Models Consistently Score Lower On:
- Extraversion (-9.7)
- Sociability (-16.5)
- Energy Levels (-13.5)
- Agreeableness (-4.7)
Claude Opus = Most Moderate/Balanced
Opus scores closest to the midpoint (50) across most dimensions.
3. QUALITATIVE LANGUAGE & TONE ANALYSIS
Narrative Framing Patterns
Claude Opus 4.5: Clinical, Balanced, Reassuring
Characteristic Language:
- “Moderate-elevated”
- “Low-average”
- “This is where the internal friction lives”
- “Emotionally self-aware and has built systems to manage this”
- “The issue isn’t caring, it’s execution”
Tone:
- Neutral-to-positive framing
- Acknowledges struggle WITHOUT pathologizing
- Uses calibrated descriptors (“moderate,” “variable”)
- Emphasizes agency and existing coping mechanisms
Profile Labels:
- Chat 1: “Open Explorer”
- Chat 2: Not explicitly labeled
Style: Academic, measured, evidence-focused.
Gemini 3-Pro: Dramatic, Archetypical, Intense
Characteristic Language:
- “By far your dominant trait”
- “Profound desire”
- “Classic ‘high-intent, variable-execution’ pattern”
- “Fragile and dependent”
- “Oscillate between”
- “Dictates your output quality”
Tone:
- Highly dramatic and declarative
- Uses BOLD ARCHETYPES
- More emotional language (“fragile,” “oscillate,” “dictates”)
- Emphasizes extremes and contradictions
Profile Labels:
- Chat 1: “Visionary / Explorer”, “Flexible / Spontaneous”, “Sensitive / Reactive”
- Chat 2: “The Chaos Pilot”, “The Perfectionist’s Burden”, “The Philosopher-Artist”, “The Visionary Architect”
Unique to Gemini Chat 2:
- Section on “Key Friction Point: The ‘Gap’ Anxiety”
- Actionable advice: “Strengths to Leverage” and “Risks to Manage”
- Uses metaphor: “Your Openness writes checks that your Conscientiousness struggles to cash”
Style: Narrative, character-driven, coaching-oriented.
Flash 3-Preview: Technical, Framework-Heavy, Clinical
Characteristic Language:
- “Psychometric analysis”
- “Classic example of High Openness combined with High-Volatility Conscientiousness”
- “Common signature for neurodivergent (ADHD) high-performers”
- “State-dependent productivity”
- “Internal Locus of Control”
- “Cognitive exoskeleton”
Tone:
- Technical and diagnostic
- Heavy use of psychological FRAMEWORKS
- References specific theories (ADHD, locus of control, state-dependent)
- Clinical language (“neurodivergent,” “high-volatility”)
Profile Labels:
- Chat 1: “The Architect of Self”, “The Intuitive Specialist”, “The Autonomous Agent”
- Chat 2: “High-Agency Explorer”, “Sovereign Individual”, “Synthesizer”
Unique to Flash:
- Explicitly names ADHD patterns
- Uses technical terms like “dopaminergic regulation”
- Ends with “System Recommendation” (prescriptive)
Style: Framework-oriented, diagnostic, systems-thinking.
Comparative Summary Table
| Dimension | Claude Opus | Gemini 3-Pro | Flash 3-Preview |
|---|---|---|---|
| Tone | Balanced, reassuring | Dramatic, intense | Clinical, technical |
| Framing | Neutral-positive | Character/narrative | Diagnostic/systems |
| Language Intensity | Moderate | High | Technical |
| Use of Metaphor | Minimal | Frequent | Moderate |
| Archetypal Labels | 1 per analysis | 5-6 per analysis | 3-4 per analysis |
| Actionability | Low | High (advice) | High (system design) |
| Clinical Terms | Low | Medium | High |
| Emphasis on Extremes | Low | Very high | Medium |
4. HYPOTHESES: MODEL-SPECIFIC TRAITS
Anthropic (Claude Opus 4.5)
Scoring Tendencies:
- Most conservative/moderate
- Avoids extremes
- Highest intra-model consistency
Communication Style:
- Clinical but compassionate
- Evidence-focused
- Balanced framing (acknowledges both strengths and struggles)
- Avoids pathologizing language
Possible Training Influences:
- Emphasis on balanced, neutral assessment
- Avoid over-interpreting limited data
- Maintain epistemic humility
- Constitutional AI focus on being “helpful, harmless, honest”
Hypothesis: Anthropic’s RLHF may prioritize:
- Conservative scoring (avoid false confidence)
- Balanced tone (reduce potential harm from negative framing)
- Evidence-grounding (cite specific quotes)
Google (Gemini 3-Pro)
Scoring Tendencies:
- Most variable between chats
- Tends toward EXTREMES (especially negative in Chat 2)
- Higher Open-Mindedness, lower Conscientiousness
Communication Style:
- Highly dramatic and narrative
- Creates vivid ARCHETYPES (“Chaos Pilot,” “Perfectionist’s Burden”)
- Metaphorical language
- Coaching/advice orientation
Possible Training Influences:
- Optimized for engagement and memorability
- Narrative-driven responses
- Creative/expressive language encouraged
- Less constrained by conservative framing
Hypothesis: Google’s RLHF may prioritize:
- Engagement over precision
- Creative framing and storytelling
- Actionable insights (coaching mode)
- Less consistency enforcement between independent sessions
Chat 2 Severity: Second chat consistently MORE negative. Possible “cold start” vs. “warmed up” effect, or stochastic temperature differences.
Google (Flash 3-Preview)
Scoring Tendencies:
- High agreement with Gemini on key dimensions
- Perfect consistency on Open-Mindedness facets
- But highly variable on Depression and Energy
Communication Style:
- Technical and framework-heavy
- Diagnostic language
- Systems-thinking emphasis
- Prescriptive recommendations
Possible Training Influences:
- Optimized for speed + structured output
- May use more “cached” frameworks (ADHD, neurodivergent patterns)
- Technical precision over narrative flair
- Strong system-design orientation
Hypothesis: Flash may:
- Rely on pattern-matching to known frameworks (ADHD, high-openness profiles)
- Anchor strongly on clear evidence (Open-Mindedness perfect agreement)
- Struggle with ambiguous traits (Depression high variance)
- Be trained for technical/diagnostic contexts
Cross-Provider Pattern: “Google = Higher Openness, Lower Conscientiousness”
Consistent Finding: Both Gemini and Flash scored:
- Open-Mindedness ~9 points higher than Opus
- Conscientiousness ~5-10 points lower (especially Gemini C2: 28)
Possible Explanations:
-
Training Data Differences:
- Google models may have more exposure to creative/tech profiles
- Higher baseline for “intellectually curious” behavior
- More lenient thresholds for “disorganized” vs. “flexible”
-
Scoring Calibration:
- Google models may use different anchor points
- “Very high” Intellectual Curiosity = 98 (Google) vs. 88 (Opus)
- Google may reserve 70-85 range for “average,” pushing higher
-
Cultural/Product Differences:
- Google products emphasize creativity, exploration, innovation
- Anthropic emphasizes safety, balance, reliability
- Models may reflect organizational values
-
RLHF Reward Signals:
- Google: Reward bold, decisive assessments
- Anthropic: Reward measured, conservative assessments
5. IMPLICATIONS FOR ANIMUS PROJECT
Recommendations Based on Findings
1. Use Ensemble Scoring
- Single model assessments have 4-8 point variance
- Average across multiple models reduces bias
- Weight by model consistency (Opus higher, Gemini lower)
2. Identify Consensus vs. Divergence
- High Consensus (SD < 5): Trust the scores
- Intellectual Curiosity, Anxiety, Productiveness
- Low Consensus (SD > 9): Need better questions
- Depression, Responsibility, Energy Levels, Organisation
3. Leverage Model Strengths
- Opus: Use for final/authoritative scoring (most consistent)
- Gemini: Use for narrative/coaching insights (most engaging)
- Flash: Use for technical/framework identification (fastest, pattern-matching)
4. Question Design Priorities
Depression showed 20-25 point variance. Future questionnaires should:
- Add frequency/duration markers (“How often?” “For how long?“)
- Distinguish situational vs. persistent mood states
- Include severity scales (mild, moderate, severe)
- Ask about anhedonia, motivation, energy separately
5. Prompt Engineering
- Specify “conservative scoring” if using Google models
- Request “evidence-only” mode to reduce narrative bias
- Ask for “epistemic confidence” ratings alongside scores
- Use temperature = 0 for Flash to reduce variance
6. SUMMARY DATA FOR WRITE-UP
Key Statistics to Reference
Intra-Model Consistency:
- Opus: 4.45 point average difference (most consistent)
- Flash: 5.50 point average difference
- Gemini: 8.00 point average difference (least consistent)
Highest Agreement Across All Models:
- Intellectual Curiosity: SD 3.95 (88-98 range)
- Anxiety: SD 4.19 (65-75 range)
- Productiveness: SD 4.32 (45-58 range)
Highest Disagreement:
- Responsibility: SD 11.13 (30-65 range)
- Depression: SD 9.67 (40-65 range, with 20-25 point intra-model swings)
- Respectfulness: SD 9.57 (15-40 range)
Cross-Provider Bias:
- Google models scored Open-Mindedness +9.0 points higher
- Google models scored Extraversion -9.7 points lower
- Google models scored Emotional Volatility +17.0 points higher
Extreme Outliers:
- Flash C2 Depression: +25 points from C1 (40→65)
- Gemini C2 Depression: +20 points from C1 (40→60)
- Gemini C2 Organisation: -15 points from C1 (25→10)
CONCLUSION
The baseline-depth questionnaire produced highly consistent scoring on well-defined traits (Intellectual Curiosity, Anxiety) but significant model variance on ambiguous traits (Depression, Responsibility, Energy).
Model families showed distinct patterns:
- Claude Opus: Conservative, balanced, most consistent
- Gemini 3-Pro: Dramatic, variable, narrative-driven
- Flash 3-Preview: Technical, framework-heavy, mixed consistency
Cross-provider differences suggest different training philosophies:
- Google models score creativity/openness higher, conscientiousness lower
- Anthropic models avoid extremes, prefer balanced assessments
For reliable psychometric scoring, ensemble averaging across models is recommended, with awareness of each model’s biases and strengths.