AIF-C01
Deep Dive
Before talking about foundation models or generative AI, the AIF-C01 exam expects you to master the base concepts of Machine Learning: how a model learns, what phases an ML project has, and why a model can fail even when it "works" in tests.
Contents
Machine Learning is not a single technique — it's a family of approaches that differ in how the model receives information during training. The exam expects you to recognize which one applies to a given scenario.
Supervised
The model learns from labeled data: every training example has an input and a known correct output. The model adjusts its parameters to minimize the error between its prediction and the actual label.
Unsupervised
The model receives unlabeled data and looks for hidden structure or patterns on its own: clusters, relationships, or dimensionality reduction.
Reinforcement
An agent learns by trial and error, interacting with an environment. It receives rewards or penalties based on its actions and adjusts its strategy (policy) to maximize cumulative reward.
Common exam trap
If the scenario mentions "historical data with the correct answer already known" → supervised. If it mentions "grouping without knowing the categories beforehand" → unsupervised. If it mentions "rewards", "policies", or "an agent acting in an environment" → reinforcement.
An ML model isn't born trained. It's an iterative process with six well-defined phases — and the exam may ask which phase a specific activity occurs in (for example, "detecting drift" is monitoring, not evaluation).
Data collection
Gather relevant, sufficient, and representative data for the real problem. The quality of this phase caps the ceiling of the whole project.
Data preparation
Cleaning (null values, duplicates), transformation (normalization, encoding), and labeling if the approach is supervised.
Training
The algorithm iteratively adjusts its internal parameters to minimize error over the training set.
Evaluation
Measure performance with data the model NEVER saw during training, using metrics like accuracy, precision, recall, or F1.
Deployment
Publish the model as an inference endpoint accessible by applications, integrated into a production workflow.
Monitoring
Watch performance in production to detect degradation or "drift": when real-world data changes relative to the training data.
It's a cycle, not a straight line
Monitoring in production frequently reveals that the model needs to be retrained with new data — which restarts the cycle from data collection. This continuous process is known as MLOps.
Let SageMaker Autopilot automatically train and compare several models, and deploy the best one.
+1 more...
Labeled data
Each record includes the correct answer alongside the input. Example: an image together with the label "cat". Necessary for supervised learning, but expensive and slow to produce — often requiring human annotation.
On AWS, SageMaker Ground Truth combines human annotation with ML assistance to reduce labeling cost.
Unlabeled data
Just the input, with no known answer. Much more abundant and cheaper to obtain (internet text, logs, uncategorized images) and the foundation of unsupervised learning and foundation model pre-training.
Most generative AI foundation models are pre-trained on massive volumes of unlabeled data.
Create a labeling job and manually label uncategorized support messages using SageMaker Ground Truth.
+1 more...
Beyond labeled/unlabeled, the exam expects you to recognize other common ways of classifying the data that feeds an AI model, based on its structure and its relationship with time.
Structured data
Organized in a fixed, predictable schema: rows and columns, like a database table or a CSV file. Easy to query and process with classic ML.
Unstructured data
No predefined schema: free text, images, audio, video. They represent most of the data generated today and are the natural territory of deep learning and foundation models.
Tabular data
A particular (and very common) case of structured data: each row is an observation and each column a feature. It's the typical input format for SageMaker Autopilot and traditional scoring/prediction models.
Time-series data
Chronologically ordered observations where temporal order matters: stock prices, IoT sensor readings, web traffic. They require techniques that capture trend and seasonality (e.g. RNNs, or specialized options like SageMaker Canvas time-series forecasting, the successor to Amazon Forecast).
To honestly evaluate a model, its data is split into three independent sets. Using the same set for training and evaluation produces a false sense of accuracy.
| Set | What it's used for | Typical proportion |
|---|---|---|
| Training set | The model learns its parameters by fitting directly to this data | ~60-80% |
| Validation set | Tune hyperparameters and compare model configurations without touching the test set | ~10-20% |
| Test set | Final evaluation, done once, with data the model never saw nor influenced any decision | ~10-20% |
Why a separate test set matters
If you evaluate with data the model already saw (directly or indirectly, by tuning hyperparameters against it), you get an optimistic metric that doesn't predict real production performance. The test set should be touched only once, at the end.
The goal of training a model isn't for it to memorize the data, but to generalize: to perform well on new data it has never seen. Two ways to fail at this, in opposite directions.
Overfitting
The model is too complex for the amount of data available and ends up memorizing the training set, including its noise, instead of learning general patterns.
Symptom: very high accuracy on training, but low on validation/test.
Example: a fraud-detection model that reaches 99.9% accuracy on training but consistently fails on new transactions — it memorized specific cases, not the fraud pattern.
Underfitting
The model is too simple to capture the real patterns in the data. It doesn't even achieve good performance on the training set itself.
Symptom: low accuracy on both training and validation/test.
Example: trying to predict a home's price using only a straight line when the actual relationship between variables is clearly non-linear.
Common mitigations:
Once trained, a model is used to generate predictions on new data — that's "inference". How and when it runs depends on the use case's requirements: tolerable latency, request volume, and its pattern over time.
Batch
A large volume of data is processed at once, on a schedule, without needing an immediate response. Example: generating product recommendations for all customers every night.
Real-time
The model responds in milliseconds to each individual request through an always-on endpoint. Example: approving or declining a card transaction at the moment of purchase.
Asynchronous
The request is queued and processed when capacity is available, without blocking the client that generated it. Ideal for large inputs (long documents, videos) where processing can take seconds or minutes and instant response isn't required.
Serverless
The inference infrastructure automatically scales to zero when there's no traffic and provisions on demand. Reduces costs for intermittent or unpredictable workloads, at the cost of possible extra latency on the first request ("cold start").
In SageMaker
SageMaker offers dedicated endpoints for real-time inference, Batch Transform for batch inference, asynchronous inference queues for large payloads, and Serverless Inference for intermittent workloads without managing fixed capacity.
"Accuracy" is the most intuitive metric, but it can be deeply misleading when classes are imbalanced. The exam expects you to know when to look beyond accuracy.
Accuracy
Total percentage of correct predictions over all predictions. Simple, but misleading with imbalanced classes.
Precision
Of everything the model flagged as positive, what percentage was actually positive? Matters when the cost of a false positive is high.
Recall
Of all the actually positive cases, what percentage did the model detect? Matters when the cost of a false negative is high.
F1 score
Harmonic mean of precision and recall. Useful as a single summary metric when you need to balance both and classes are imbalanced.
Why accuracy alone is misleading — fraud-detection example
Imagine a transaction dataset where only 1% are fraudulent. A "lazy" model that ALWAYS predicts "not fraud" gets 99% accuracy — and yet it's completely useless, because it never catches a single real fraud (recall = 0%). Here precision and recall (and F1) reveal the problem that accuracy hides.
A technically excellent model can still be a bad project if it doesn't generate business value. The exam also tests whether you recognize metrics outside the model itself, used to justify (or question) an investment in ML/AI.
Cost per user
How much it costs to operate the model (compute, storage, API) divided by active users — key for deciding whether the model scales sustainably.
Development costs
Investment in data, talent, training, and experimentation needed to reach a usable production model.
Customer feedback
Satisfaction, complaints, or actual adoption rates of the AI-powered feature — the model can have great technical metrics and still not please users.
ROI (Return on Investment)
Ratio between the value generated (cost savings, additional revenue) and the total project investment — the metric that ultimately decides whether the project continues.
Model metrics (accuracy, F1...) and business metrics (ROI, cost per user...) answer different questions: the former say whether the model predicts well; the latter say whether it's worth having in production.
Did you get this topic?
Test what you just learned
A data science team trains a model that achieves 98% accuracy on the training set, but only 61% on the test set. What problem best describes this situation?
Put what you learned to the test with original AIF-C01 exam questions.
Start AIF-C01 mock examUnlimited mock exams with the Pro plan
Log in to track your progress.