← All projects
🔬
LLM Evaluation Platform
Advanced⏱ 2–3 weeks · 6 milestones
Build a platform for evaluating LLM outputs — RAGAS-style metrics, human evaluation UI, leaderboard, and automated regression testing.
Difficulty
Advanced
Duration
2–3 weeks
Milestones
6 steps
Interview Qs
4 questions
🛠 Tech stack
PythonRAGASFastAPINext.jsPostgreSQL or FirestoreDocker
✅ Prerequisites
- ● RAG fundamentals
- ● Python (advanced)
- ● Full-stack basics
Step-by-step milestones
1
Define evaluation metrics
Implement faithfulness, relevance, context precision.
2
Automated eval pipeline
Run evals programmatically against a test set.
3
Human eval UI
Build side-by-side comparison and rating UI.
4
Leaderboard
Store and display model comparison results.
5
CI integration
Run evals on every PR automatically.
6
Report generation
Export evaluation reports as PDF/HTML.
Skills you'll build
LLM EvaluationRAGMetricsSystem designFrontend + Backend
📝 Interview questions you'll face
- 1.How do you define ground truth for LLM outputs?
- 2.What is faithfulness vs relevance in RAG eval?
- 3.How do you prevent LLM judges from being biased?
- 4.How would you scale evals to 1M queries/day?
🌟 Portfolio guidance
This is a standout portfolio project. Write a technical blog post, share it on LinkedIn/X, and mention RAGAS/LLM-as-judge concepts.