Learn what confidence scores mean in Arcus classifications and when to request expert review.
What Are Confidence Scores?
When Arcus classifies your AI system, it provides a confidence score from 0-100% indicating how certain the AI is about the classification.
This transparency helps you understand:
- How reliable the classification is
- Whether additional review is needed
- When to provide more information
- If legal consultation is warranted
The Three Confidence Levels
High Confidence (80-100%) ✅
What It Means:
The classification is straightforward with clear regulatory guidance.
Characteristics:
- System description matches well-defined category
- Regulatory precedent exists
- Minimal ambiguity in use case
- Comprehensive information provided
Example:
System: Employee resume screening tool
Confidence: 95%
Classification: High-Risk (Employment)
Why High Confidence: This clearly matches Annex III, Section 4(a) of the EU AI Act - AI systems used for recruitment. Strong regulatory precedent exists.
Should You Request Review?
Generally not necessary, but you may still want legal consultation for implementation details.
Moderate Confidence (60-79%) ⚠️
What It Means:
The classification requires some interpretation or could affect multiple categories.
Characteristics:
- Some ambiguity in system description
- Could fall into multiple risk categories
- Limited regulatory precedent
- Novel or emerging use case
Example:
System: Customer sentiment analysis for product feedback
Confidence: 72%
Classification: Limited Risk (Emotion Recognition)
Why Moderate Confidence: While it analyzes sentiment, it's unclear if this constitutes "emotion recognition" under the Act. Depends on specific implementation.
Should You Request Review?
Recommended, especially for critical systems or if facing audits.
Low Confidence (<60%) 🚨
What It Means:
Significant uncertainty in the classification - expert review strongly recommended.
Characteristics:
- Insufficient system information
- Borderline between categories
- Novel use case with no precedent
- Regulatory ambiguity
Example:
System: AI-powered workflow optimizer
Confidence: 45%
Classification: Minimal Risk
Why Low Confidence: Description is vague. Could be minimal risk general productivity tool, or high-risk if it makes employment decisions. Need more details.
Should You Request Review?
Yes, strongly recommended. Don't rely on this classification for compliance decisions.
Factors Affecting Confidence
Description Quality
High Impact on Confidence:
- Detailed explanation of purpose
- Clear use case scenarios
- Specific decision-making authority
- Well-defined user groups
Low Impact on Confidence:
- Vague or generic descriptions
- Unclear decision-making process
- Ambiguous use cases
- Missing key information
Regulatory Clarity
High Confidence Triggers:
- System matches specific Annex III category
- Similar systems have established precedent
- Guidance documents address use case
Low Confidence Triggers:
- Novel technology or use case
- Borderline classification
- Conflicting regulatory guidance
- Rapidly evolving regulation
Use Case Complexity
Simple = Higher Confidence:
- Single, well-defined purpose
- One primary use case
- Clear sector placement
Complex = Lower Confidence:
- Multiple interconnected purposes
- Hybrid use cases
- Cross-sector applications
How Confidence Is Calculated
Arcus AI evaluates several dimensions:
1. Criteria Match Strength
How closely does the system match EU AI Act categories?
- Perfect match → +20 points
- Strong match → +15 points
- Partial match → +10 points
- Weak match → +5 points
2. Ambiguity Presence
Are there unclear aspects in the description?
- No ambiguity → +20 points
- Minor ambiguity → +10 points
- Moderate ambiguity → +5 points
- High ambiguity → 0 points
3. Precedent Availability
Do similar systems have established classifications?
- Strong precedent → +20 points
- Some precedent → +10 points
- Limited precedent → +5 points
- No precedent → 0 points
4. Information Completeness
How complete is the system description?
- All key information → +20 points
- Most information → +15 points
- Some gaps → +10 points
- Major gaps → 0 points
5. Regulatory Guidance
Is there clear guidance for this use case?
- Explicit guidance → +20 points
- Implied guidance → +10 points
- General guidance → +5 points
- No guidance → 0 points
Total Score = Sum of all dimensions
When to Request Expert Review
Always Request Review If:
- Confidence score below 60%
- High-risk classification with any uncertainty
- System affects fundamental rights
- Facing regulatory audit
- Legal implications are significant
Consider Review If:
- Confidence 60-79% and system is high-risk
- Disagreement with classification
- Novel or unique use case
- Multiple stakeholders need alignment
- Board or investors require validation
Probably Don't Need Review If:
- Confidence 85%+ and minimal/limited risk
- Clear regulatory guidance
- Low-stakes internal tool
- Similar to previously classified systems
Improving Confidence Scores
Add Missing Information
Before:
"Machine learning model for customer service"
After:
"NLP-based chatbot that answers common customer questions about order status and returns. Escalates complex issues to human agents. Does not make autonomous decisions about refunds or account changes. Used by 50,000 monthly customers."
This added detail typically increases confidence by 15-25 points.
Clarify Decision-Making
Before:
"Helps managers with hiring"
After:
"Provides candidate rankings to hiring managers based on resume analysis. Final hiring decisions are always made by humans after interviews. System is advisory only - managers can and do override recommendations."
Clarity on human oversight often increases confidence significantly.
Specify the Sector
Before:
"AI for risk assessment"
After:
"Credit risk assessment tool for consumer loan applications in the financial services sector. Determines eligibility for loans up to €10,000."
Sector specification helps AI match to specific regulatory categories.
Provide Use Case Context
Explain:
- Where the AI fits in your workflow
- Who it serves and how
- What happens with its output
- What safeguards are in place
What Confidence Doesn't Mean
❌ Confidence ≠ Accuracy
High confidence means the AI is certain about its classification based on the information provided. It doesn't guarantee the classification is legally correct.
❌ Confidence ≠ Compliance
Even a 100% confident classification doesn't mean your system is compliant - it just means the risk level is clear. You still need to meet the requirements for that risk level.
❌ Confidence ≠ Legal Advice
Arcus provides guidance, not legal advice. Consult attorneys for compliance strategy, even with high confidence scores.
Expert Review Process
What Review Includes
When you request expert review:
- Human Analysis: Legal and technical experts examine your system
- Clarifying Questions: You may be asked for additional details
- Independent Assessment: Classification verified or corrected
- Detailed Reasoning: Written explanation of the determination
- Recommendations: Specific guidance for your situation
Typical Timeline
- Professional Plan: 3-5 business days
- Business Plan: 1-2 business days
- Urgent requests: 24-hour expedite available (additional fee)
Review Limits
Based on your subscription:
- Starter: Not available (upgrade required)
- Professional: 2 reviews per month
- Business: Unlimited reviews
Real-World Examples
Example 1: E-Commerce Recommender
Initial:
Description: "Product recommendation system"
Confidence: 55%
Classification: Limited Risk
After Adding Details:
Description: "Collaborative filtering system that suggests products based on browsing history. Not used for dynamic pricing or inventory decisions. Shows 'Customers also bought' section on product pages. Users can easily ignore suggestions."
Confidence: 88%
Classification: Limited Risk (Transparency required)
Lesson: Clarifying what the AI doesn't do is as important as what it does.
Example 2: Healthcare Assistant
Initial:
Description: "AI assistant for doctors"
Confidence: 42%
Classification: High-Risk
After Expert Review:
Description: "Administrative assistant that schedules appointments and sends medication reminders. Does NOT provide diagnostic suggestions or treatment recommendations. All clinical decisions made by licensed physicians."
Confidence: 91%
Classification: Minimal Risk
Lesson: Healthcare context triggers caution, but administrative use doesn't trigger high-risk designation.
FAQs
Q: Can confidence scores change over time?
Yes. As regulations evolve and precedent develops, Arcus may re-score classifications. You'll be notified of significant changes.
Q: What if two experts disagree?
If external review contradicts Arcus, document both opinions and consult your legal team for final determination.
Q: Can I override a low-confidence classification?
Yes, but you must document your justification thoroughly for potential audits.
Q: Do high scores guarantee no audit issues?
No. Auditors may interpret regulations differently. Always maintain strong documentation.
Learn More
- How Classification Works
- Improving Classification Accuracy
- Risk Classification Explained
Confidence scores are a tool for transparency, not a substitute for legal judgment.