AI Insights

Written by Siarhei Oshyn, Head of Data / Data & AI Architect.

Based on Godel’s experience, most AI initiatives involving large language models miss their targets for a single reason: model selection is treated as a technical preference rather than a business decision. Teams often start by comparing benchmarks or picking the “most powerful” model – without clearly defining what success actually looks like in operational terms.

In leadership, the right model is not the most popular. It is the model that clears quality gates on real workflows, meets latency and reliability requirements, stays cost-efficient at scale, and can be governed by your team.

To demonstrate this in practice, the Godel Advisory Team applied their model selection framework to a concrete scenario: a smartphone Technical Support Centre for an e-commerce business. Instead of discussing models in the abstract, the decision process is anchored in a real operational use case – classifying and routing customer emails about battery issues, display defects, charging failures, and similar problems. Each email is mapped to a structured JSON output containing fields such as problem category, severity level, resolution path, and recommended action. In this context, the LLM is not simply generating text – it is becoming part of a decision system.

The framework is built around five decision gates. The first is Data Readiness: before touching any model, you validate whether your dataset reflects real production complexity. This means combining real anonymised emails with synthetic data to cover edge cases, enforcing strict annotation standards, and obtaining cross-functional sign-off from operations, engineering, and compliance stakeholders.

The second gate is Cost Feasibility. Token price is only one part of the equation – the real unit economics depend on prompt size, output length, retry behaviour, and expected volume. Translating this into daily operational costs at 1,000 and 10,000 issues per day quickly reveals which models are viable and which should be filtered out before any quality testing begins.

The third gate covers SLA and Performance. Even a cheap model fails if it cannot meet response time requirements. Latency benchmarks across all candidate models enable elimination of non-starters before investing in deeper evaluation.

The fourth and most detailed gate is Quality and Risk. Models are evaluated on structured output fields using F1 scores, and on free-text fields using an LLM-as-judge approach measuring semantic similarity, information coverage, and hallucination rate. Critically, results are presented as a Go/No-Go scorecard per field – because not every output needs to be perfect, but only some can be trusted for automation. Fields that fall below the threshold stay in human review until the next iteration.

The fifth gate addresses Prompt Operations. Prompt quality is an operating capability, not a one-time artefact. Organisations that succeed with LLMs treat prompts as versioned, monitored production assets – with rollback discipline, change logs, and defined ownership. A key architectural choice also arises here: whether to use one large prompt per call or a multi-step pipeline. The right answer depends on delivery maturity, team capacity, and cost constraints.

Finally, all dimensions – quality, cost, and speed – are consolidated into a weighted scoring model that produces a ranked shortlist aligned with the business’s actual priorities. In the example run, Mistral’s Devstral ranked first by combining strong quality, manageable cost, and mid-pack latency. Claude 3 Haiku proved a strong alternative when cost and speed matter alongside solid output quality. Claude Sonnet 4 led on raw quality but was penalised by the highest token cost under this weighting.

The takeaway is straightforward: choosing an LLM is not a procurement exercise. It is an operating model decision – one that defines how your organisation balances cost, quality, speed, and risk in production AI systems. The five-gate framework provides a repeatable, business-grade process for making that decision with confidence rather than enthusiasm.

For the full version of this article, including detailed benchmark tables, prompt examples, scoring methodology, and field-level quality results, visit here.

Siarhei Oshyn, Head of Data / Data & AI Architect

Posted 21 May 2026

share