AIF-C01
Deep Dive
With dozens of foundation models available on Bedrock, choosing the right one for your use case requires comparing objective criteria — not just picking "the biggest" or "the newest".
Contents
There's no "best model" in the abstract — there's the model best suited to your task, your budget, and your constraints.
Cost
Pricing varies per million input/output tokens, and also depends on whether the model is large or small. A more expensive model doesn't always add value if your task is simple.
Latency
How long the model takes to respond. Critical in real-time applications (chat, voice assistants) where the user expects an almost-immediate response.
Context window size
How much information (text, conversation history, documents retrieved with RAG) the model can process at once. Tasks that analyze long documents need a wide context window.
Multimodal capability
If the use case requires processing not just text but also images, audio, or video, you need a multimodal model specifically trained for that.
Accuracy for the specific task
A model may excel at creative generation but be mediocre at mathematical reasoning, or vice versa. Accuracy should be measured on the real task, not on generic benchmarks.
Compliance and data residency requirements
Some models or regions have different availability restrictions or certifications, relevant in regulated industries.
Evaluating a generative model is harder than evaluating a traditional classification model: there isn't always a single "correct answer".
BLEU / ROUGE
Automatic metrics that compare the model's generated text against a "correct" reference text, measuring the overlap of words or phrases. Useful for tasks like translation (BLEU) or summarization (ROUGE), but limited: they don't capture semantic quality or creativity well.
BERTScore
An automatic metric that, unlike BLEU/ROUGE, doesn't compare exact words but uses embeddings to measure the SEMANTIC similarity between the generated text and the reference. It detects as "correct" answers that say the same thing with different words, something BLEU/ROUGE would penalize.
LLM-as-a-judge
Uses another foundation model as a "judge" to evaluate and score the responses of the model being evaluated, based on criteria like relevance, correctness, or tone. Scales better than human evaluation and captures more nuance than automatic metrics like BLEU/ROUGE, though it inherits the potential biases of the evaluator model.
Human evaluation
People review and score the model's responses based on criteria like relevance, coherence, helpfulness, or tone. More costly and slow, but captures nuance that automatic metrics can't measure.
Standardized benchmarks
Public test suites (for reasoning, general knowledge, code, etc.) that allow comparing relative performance across different models consistently.
Key idea for the exam
The AIF-C01 expects you to understand these metrics at a conceptual level: what they measure and which type of task they're relevant for (BLEU/ROUGE for text with a reference, human evaluation for subjective quality). You don't need to memorize mathematical formulas.
What it lets you do
Bedrock includes a model evaluation feature that lets you compare the performance of different foundation models using your own data and test prompts, before deciding which one to take to production.
Why it matters
Instead of choosing a model based only on generic public benchmarks, you can validate which one performs best specifically for your real use case, with your own data — before committing to it in production.
Compare the response quality of two foundation models with Bedrock Model Evaluation.
+1 more...
Evaluating just the foundation model in isolation isn't enough. In production, the model is one piece within a larger application (a RAG solution, an agent, a multi-step workflow), and you need to evaluate whether the full application meets the business goal.
RAG applications
It's not enough to evaluate whether the model generates fluent text: you need to measure whether the retrieval found the right fragments and whether the final response is actually grounded in those fragments (retrieval relevance + response faithfulness to the context).
Agents
You need to evaluate whether the agent chose the right tools, called them with the right parameters, and whether the sequence of steps completed the requested task — not just whether the final text "sounds good".
Multi-step workflows
When several components (models, tools, business rules) are chained together, you need to evaluate the end-to-end result, because a failure in an intermediate step may not be visible if you only look at the final output.
Beyond the model's technical metrics, the exam expects you to connect evaluation with real business impact.
Task completion rate
The percentage of requests or tasks the application effectively resolves from start to finish, without additional intervention.
User satisfaction
How satisfied real users are with the responses or results — measured with surveys, ratings, or escalation-to-human rates.
Cost per interaction
The average cost (tokens, compute, tool calls) of each full interaction with the application, key to deciding whether the solution is sustainable at scale.
Key idea for the exam
A model with excellent technical metrics (high BLEU, good LLM-as-a-judge score) can still be a bad business solution if the cost per interaction is too high or if real users aren't satisfied — technical evaluation and business evaluation are complementary, not substitutes.
Large models
Ideal for complex tasks where response quality matters more than speed or cost.
Small models
Ideal for simple, high-volume tasks, such as fast classification or simple extraction, where speed and cost are priorities.
Decision rule for the exam
If the scenario emphasizes cost and speed at large scale with relatively simple tasks, favor a smaller model. If it emphasizes quality and reasoning on complex tasks, favor a larger model — even at higher cost and latency.
Did you get this topic?
Test what you just learned
A company needs to automatically classify millions of support messages per day into simple categories (billing, technical, general) with the lowest possible cost and latency. Which type of model is most appropriate?
Put what you learned to the test with original AIF-C01 exam questions.
Start AIF-C01 mock examUnlimited mock exams with the Pro plan
Log in to track your progress.