Personal project · Design overview
Inference Economics Workbench
A measurement-led workbench for comparing open-source models across practical enterprise workloads, with accuracy-per-dollar as the decision lens.
How it works
The proposed workbench frames model selection as an empirical platform decision. For each workload, define the expected outcome, evaluate candidate models under recorded conditions, and relate observed quality to serving cost. The result is a decision teams can inspect and revisit as models, workloads, and constraints change.
Operating principle: recommend from measured outcomes and make the evaluation context visible. A score without its workload and conditions is not a useful platform decision.
Evaluation tracks
Track 01
Conversational analytics
Frame the workload around natural-language questions that require SQL execution. The implementation should compare returned query outcomes rather than surface similarity between generated SQL statements.
Track 02
Enterprise retrieval-augmented generation
The enterprise RAG motion uses a versioned semantic extraction engine with semantic contracts and an ontology. Provenance, confidence, and permission models are part of the platform design.
Track 03
Tool use
A further evaluation track can cover model interactions with tools. Its tasks and success criteria should be defined and measured before reporting results.
Key design decisions
Measure outcomes, do not predict them.
Compare observed task quality and serving cost under stated conditions. Keep assumptions and run provenance visible so a recommendation can be reproduced and challenged.
Keep task suites explicit.
Treat each workload as its own evaluation suite. This makes it possible to add a use case without hiding its success criteria inside a general score.
Make comparisons auditable.
Record model, configuration, workload, and measurement context with each result. The measured-results section stays open until actual runs are available.
Trust and governance
Enterprise adoption depends on more than a model score. The evaluation design should make inputs, source grounding, action constraints, and outcome provenance inspectable. For enterprise RAG, permission-aware access and the evidence behind an answer are central requirements; they should be evaluated as part of the workload rather than treated as deployment details.
Extensibility
A new model should be introduced through a recorded configuration; a new use case should arrive with its own task suite and success criteria. This keeps comparisons legible as the workbench grows and avoids silently changing what a score means.
Measured results
| Quality | Measured task success for each evaluation track |
|---|---|
| Economics | Serving cost associated with each measured configuration |
| Traceability | Workload, model, configuration, and run conditions |
Technical architecture
The design separates workload definitions, model configurations, evaluation runs, and result reporting. That boundary allows teams to add models and tasks independently while preserving the context behind each comparison. Implementation details and measured results belong here once they are verified from the project itself.
| Decision lens | Measured task quality per serving cost |
|---|---|
| Workloads | Conversational analytics, enterprise RAG, and tool use |
| Results | Pending measured runs |