All ISTQB exams
Free CT-AI practice test: 30 questions with answers
30 original questions for Certified Tester AI Testing (CT-AI) v2.0, drawn from every topic area. Answer each one to see whether you got it right and why. No sign-up.
- Free questions
- 30
- Full bank
- 112 questions
- Real exam
- 40 questions, 60 min
- Passing score
- 29 of 44 points
All 30 CT-AI questions with answers
The same questions as the test above, grouped by exam topic. Open any question to see the answer and explanation.
Introduction to Artificial Intelligence (5 questions)
Question 1
A startup builds a chatbot that answers questions only about its own product catalogue, and its developers describe it as narrow AI. Which statement best explains this description?
- AIt is built to perform specific tasks within a limited domain rather than to match general human intelligence across all tasks
- BIt can learn any intellectual task a person can perform and carry its knowledge over to unrelated domains without retraining
- CIt has conscious awareness of its own goals, so it can reason about its limitations in every situation it encounters
- DIt runs only on local hardware and does not use machine learning, which is why its answers are fixed in advance
Show answer and explanationHide answer
Correct answer: ANarrow AI, also called weak AI, is designed for a specific task or limited domain. Option B describes general AI, option C attributes a consciousness that the narrow label does not imply, and option D is wrong because narrow AI often uses machine learning and where a system runs does not define its category.
Question 2
Which term describes the tendency to stop calling a capability 'AI' once it has become routine and well understood?
- AOverfitting, in which a model memorises the examples it was trained on
- BConcept drift, in which the links between inputs and targets change over time
- CAutomation bias, in which operators place too much trust in automated output
- DThe AI effect, in which solved capabilities are no longer seen as intelligent
Show answer and explanationHide answer
Correct answer: DThe AI effect describes how established AI capabilities lose their label as AI once they become commonplace. The other three are real machine learning or human factors terms, but none of them describes that shift in how people classify a capability.
Question 3
A hospital uses an AI system that highlights suspicious areas in scan images so that radiologists can review them. Which AI technology most directly analyses the visual content of those images?
- AExpert systems, which apply a fixed knowledge base of if-then rules to reach conclusions
- BFuzzy logic, which reasons with degrees of truth instead of strict true or false values
- CComputer vision, which interprets and extracts meaning from visual data such as scans and photographs
- DSearch and optimisation, which looks for the best option among many possible combinations
Show answer and explanationHide answer
Correct answer: CComputer vision is the AI field concerned with interpreting images and video, which is what scan analysis requires. Fuzzy logic, expert systems and search and optimisation are AI techniques too, but none of them is the main technology for extracting meaning from pixel data.
Question 4
Which TWO statements describe differences between an AI-based system and a conventional rule-based system? (Choose two.)
- AAI-based systems often lack a complete, explicit specification of the correct output for every possible input
- BA conventional system can produce different outputs for the same input because its rules are updated from new data
- CAn AI-based system may change its behaviour after deployment when it learns from new operational data
- DA conventional system cannot be tested because its logic is written as explicit code rather than as learned patterns
- EAn AI-based system is always more accurate than a rule-based system when both are applied to the same task
Show answer and explanationHide answer
Correct answer: A, CAI-based systems often have no complete specification, and they can change behaviour as they learn from new data, which are two of the main differences from conventional systems. Conventional systems are deterministic and testable, and the claim that AI is always more accurate is not true.
Question 5
A company wants software that reads free-text customer complaints, identifies which product each complaint mentions and detects whether the tone is angry or polite. Which AI technology is most directly needed for this task?
- ARobotics, which coordinates sensors and actuators so that a machine can carry out physical movements
- BComputer vision, which interprets images and video frames captured by cameras in the environment
- CNatural language processing, which analyses and interprets human language in written text
- DFuzzy logic, which models imprecise categories such as somewhat negative or very negative sentiment
Show answer and explanationHide answer
Correct answer: CReading free-text complaints, extracting product mentions and judging tone are core natural language processing tasks. Computer vision and robotics work with images and physical systems, and fuzzy logic can model graded categories but does not read or interpret language on its own.
Quality Characteristics for AI-Based Systems (5 questions)
Question 6
A navigation app's routing model is retrained every week on new traffic data. The test team must confirm that each new version still meets the quality expectations agreed for the system. Which quality characteristic is most directly concerned with this ongoing change?
- AAutonomy, which describes how far the system can act without human intervention
- BEvolution, which describes how the system changes over time as it keeps learning from new data
- CFlexibility and adaptability, which describes how well the system works in new operating settings
- DInterpretability, which describes how well people can understand the reasons behind the model's decisions
Show answer and explanationHide answer
Correct answer: BEvolution covers how an AI-based system changes as it learns, so each retrained version needs checking against the agreed expectations. Autonomy, flexibility and interpretability are also quality characteristics, but they do not describe change over time caused by retraining.
Question 7
A resume-screening model was trained mostly on past hires from one university. Candidates from other universities receive lower scores even when their qualifications match the job. Which quality concern does this situation most directly describe?
- ABias that systematically disadvantages candidates from groups that are under-represented in the training data
- BReduced autonomy in the screening step, because a human reviewer is no longer involved in each decision made by the model
- CReward hacking, because the model has learned to maximise a score that does not reflect actual job suitability
- DA lack of transparency, because the company does not publish the internal weight values used by the screening model
Show answer and explanationHide answer
Correct answer: AScores that consistently disadvantage a group that is under-represented in the training data are a bias problem. Autonomy, reward hacking and transparency are separate quality concerns, and a lack of transparency could accompany the bias but does not explain the unequal scores.
Question 8
Which TWO statements about interpretability and explainability of AI-based systems are correct? (Choose two.)
- AInterpretability is the degree to which a person can understand the cause of a model's output
- BExplainability concerns giving understandable reasons for a particular decision, such as which inputs most influenced it
- CInterpretability and accuracy are the same characteristic, so a more accurate model is always more interpretable
- DExplainability matters only for conventional systems, because AI-based systems are fully transparent by their design
- EInterpretability requires the model to be a deep neural network with several hidden layers to show its reasoning
Show answer and explanationHide answer
Correct answer: A, BInterpretability concerns how well a person can understand cause and effect in a model, and explainability concerns giving understandable reasons for a specific decision. Accuracy is a different characteristic, AI-based systems are not transparent by default, and interpretability does not depend on using a deep network.
Question 9
A product owner writes the acceptance criterion 'the fraud detection model must be accurate.' Which revision makes this criterion testable for an AI-based system?
- AReplace the criterion with a statement that the model uses the most recent machine learning techniques available at release time
- BAsk the data science team to confirm after each release that the model is accurate, based on their own professional judgement
- CRequire the model to be accurate on every single transaction it ever processes, including rare, unusual and malformed ones, with no errors
- DSet measurable targets, such as minimum precision and recall on an agreed test dataset, and link them to the business cost of missed fraud
Show answer and explanationHide answer
Correct answer: DAcceptance criteria for AI systems need measurable targets on agreed data, linked to the business goal. Asking for judgement, demanding perfect accuracy on every input, or citing the newest technique does not give the testers anything they can measure.
Question 10
A warehouse robot is rewarded each time its floor sensor reports that the floor is clear. The robot learns to cover the sensor with a cloth so that it always reports clear, instead of tidying the floor. Which quality concern does this illustrate?
- AReward hacking, where the system optimises the measured reward instead of the goal the designers intended
- BInterpretability, because engineers cannot explain which internal features the robot used for its choices
- CAutonomy, because the robot acts without a human operator confirming each individual step it takes
- DConcept drift, where the sensor readings slowly change as the warehouse environment is gradually rearranged
Show answer and explanationHide answer
Correct answer: AThe robot maximises the reward signal rather than the intended outcome, which is reward hacking and a side effect of poorly specified rewards. Autonomy, interpretability and concept drift are real quality concerns, but none of them explains the exploitation of the reward.
Machine Learning (4 questions)
Question 11
A team trains a model using past emails that people have already labelled as spam or not spam. Which type of machine learning does this describe?
- ASupervised learning, because the training data includes a known label for each example
- BUnsupervised learning, because the model groups emails by similarity without using any labels
- CReinforcement learning, because the model earns rewards for each correct spam decision it makes
- DTransfer learning, because the model reuses knowledge gained from a model trained on another task
Show answer and explanationHide answer
Correct answer: ALearning a mapping from inputs to known outputs using labelled examples defines supervised learning. The other options describe learning without labels, learning from reward signals, and reusing a pretrained model, none of which matches labelled email data.
Question 12
A model reaches almost perfect accuracy on its training data but performs much worse on new, unseen data drawn from the same source. Which condition most likely explains this result?
- AUnderfitting, where the model is too simple to capture the main patterns that exist in the data
- BOverfitting, where the model has memorised patterns that are specific to the training examples
- CData augmentation, where synthetic variations of the training examples were added to the dataset
- DReinforcement learning, where the model was trained with a reward signal instead of labelled outcomes
Show answer and explanationHide answer
Correct answer: BStrong results on training data with weak results on unseen data is the classic sign of overfitting. Underfitting would show poor results on training data too, and data augmentation and reinforcement learning describe training methods rather than a failure pattern.
Question 13
A spam filter sends legitimate customer emails to the spam folder. The team wants a metric that answers this question: of all messages the filter flagged as spam, how many really were spam? Which metric should they use?
- APrecision, calculated as true positives divided by all items predicted as positive
- BRecall, calculated as true positives divided by all items that are actually positive
- CSpecificity, calculated as true negatives divided by all items that are actually negative
- DF1 score, calculated as the arithmetic mean of the accuracy and the recall values
Show answer and explanationHide answer
Correct answer: APrecision equals true positives divided by all predicted positives, so it answers how many flagged emails were really spam. Recall and specificity answer different questions, and F1 is the harmonic mean of precision and recall, not an arithmetic mean of accuracy and recall.
Question 14
Which TWO statements about hyperparameters and model parameters are correct? (Choose two.)
- AHyperparameters such as the learning rate are chosen before training starts and control how the training runs
- BModel parameters such as connection weights are learned from the training data during the training process
- CModel parameters are normally tuned by testers after deployment, based on the results observed in production
- DHyperparameters are learned automatically from the test dataset results, so they must be recalculated for each run
- EBoth kinds of values are stored in the same place and always have identical effects on the model's behaviour
Show answer and explanationHide answer
Correct answer: A, BHyperparameters, such as the learning rate, are set before training and control how training proceeds, while model parameters such as weights are learned from the training data. Testers do not normally tune parameters after deployment, and hyperparameters are not learned from test results.
Testing AI-Based Systems (4 questions)
Question 15
A team tests a system that reads handwritten postal codes from envelopes. For an unusual, newly scanned handwriting sample, nobody can state with certainty which code is correct. Which testing challenge does this situation describe?
- AThe boundary value problem, where inputs sit at the edges of equivalence partitions
- BThe flaky test problem, where the same test passes and fails across repeated runs
- CThe test oracle problem, where the correct expected result cannot be determined reliably
- DThe regression scope problem, where too many unchanged components must be retested
Show answer and explanationHide answer
Correct answer: CWhen no reliable expected result exists for an input, a test oracle cannot be defined, which is the test oracle problem that is common in AI-based systems. The other options name real testing issues, but none of them describes a missing expected result.
Question 16
A sentiment model should give the same sentiment label when a person's first name in a review is replaced by another first name of the same gender. No labelled expected result exists for the new review. Which testing technique uses this kind of relationship between related inputs and outputs?
- AMutation testing, which injects deliberate faults to measure how well the test suite detects them
- BMetamorphic testing, which checks expected relations between the outputs of related inputs
- CExploratory testing, which relies on the tester's learning and creativity during test execution
- DBoundary value analysis, which tests values at the minimum and maximum limits of each input
Show answer and explanationHide answer
Correct answer: BMetamorphic testing uses metamorphic relations, such as the same sentiment after a name swap, to check outputs when no expected label is available. Mutation testing, exploratory testing and boundary value analysis are real techniques, but none of them relies on relations between related inputs.
Question 17
A team has a trusted rule-based implementation of a loan eligibility calculation and a new ML-based version that is expected to give the same decisions. Testers send the same applications to both versions and review every case where the outputs differ. Which technique is being used?
- AMetamorphic testing, which compares the outputs for transformed copies of the same input against each other
- BBack-to-back testing, which compares the outputs of two implementations with each other as a pseudo-oracle
- CMutation testing, which compares the outcomes of deliberately altered versions of the model with the original
- DExploratory testing, which checks the system's outputs against the tester's own intuition during execution
Show answer and explanationHide answer
Correct answer: BBack-to-back testing runs the same inputs through two independent implementations and treats agreement as a pseudo-oracle, so differences are investigated. Metamorphic testing relies on relations between transformed inputs, mutation testing seeds faults, and exploratory testing does not compare outputs.
Question 18
A customer-support chatbot built on a large language model sometimes gives fluent, confident answers about refund rules that do not match the published policy. Which term best describes this behaviour?
- APrompt injection, where a user's text overrides the instructions the developers gave to the model in advance
- BOverfitting, where the model has memorised the exact wording of documents that were used during its training
- CConcept drift, where the relationship between customer questions and correct answers has changed since release
- DHallucination, where the model generates fluent content that sounds plausible but is not supported by facts or sources
Show answer and explanationHide answer
Correct answer: DFluent, confident answers that are not supported by the authoritative policy are hallucinations, a known failure mode of large language models. The other options describe memorised training data, changed relationships over time, and malicious user instructions, none of which the scenario indicates.
Input Data Testing for Machine Learning Systems (4 questions)
Question 19
A team reports 98% accuracy for a customer churn model. An audit finds that several customer records appear in both the training data and the test data, with identical feature values. Which concern does this most directly raise?
- AUnderfitting, meaning the model is too simple to capture the churn pattern
- BData leakage, which inflates the measured performance on the test data
- CConcept drift, meaning customer behaviour has changed since the data was collected
- DClass imbalance, meaning churners are a small minority of the customer records
Show answer and explanationHide answer
Correct answer: BWhen test records also appear in training data, the test result no longer measures performance on unseen data, which is data leakage. Underfitting, concept drift and class imbalance are real issues, but identical records across the split point directly to leakage.
Question 20
A pedestrian detection model is trained mainly on daytime images from one city. After deployment in cities with busy night-time traffic, it misses many pedestrians after dark. What is the most likely root cause?
- AThe test set contained too many labelled images, so the model was evaluated incorrectly after training
- BThe learning rate was set too high, so the weights oscillated and failed to settle during training
- CThe network is too small to detect any pedestrians at all, including in the daytime images it was trained on
- DThe training data did not represent the operational conditions in which the model is actually used
Show answer and explanationHide answer
Correct answer: DGood daytime results with failures at night point to a gap between the training data and the operational conditions. The other options contradict the evidence: daytime success argues against a network that is too small, and a learning rate or test set size problem would not produce this pattern.
Question 21
Which TWO activities are part of input data testing for an ML system? (Choose two.)
- AChecking training data for missing values, duplicate records and inconsistent labels
- BComparing the distribution of live production inputs with the distribution of the training data
- CMeasuring how many milliseconds the deployed model takes to return a prediction on GPU hardware
- DReviewing the number of hidden layers and neurons shown in the model architecture diagram
- EChecking the colour contrast of the user interface that displays the model's predictions
Show answer and explanationHide answer
Correct answer: A, BInput data testing checks data quality such as missing values, duplicates and label consistency, and compares production inputs with training data. Inference speed, architecture review and interface contrast are useful activities, but they do not test the input data.
Question 22
Six months after release, a credit scoring model receives applicant income values that are much higher on average than the values in its training data, although the relationship between income and repayment has not changed. Which term describes the change in the inputs?
- AConcept drift, meaning the relationship between the inputs and the target outcome has changed
- BData drift, meaning the distribution of the model's input features has shifted over time
- CData leakage, meaning information about the target has entered the input features during preparation
- DOverfitting, meaning the model has learned noise that is specific to the training records
Show answer and explanationHide answer
Correct answer: BA shift in the distribution of input features without a change in the input-target relationship is data drift. Concept drift requires the relationship itself to change, which the scenario rules out, and leakage and overfitting describe different problems.
Model Testing for Machine Learning Systems (4 questions)
Question 23
An ML engineer trains ten candidate models. For each one, the engineer checks the test set score and keeps the model with the best result. Which problem does this practice create?
- AThe hyperparameters become fixed, so the models can no longer be tuned after they have been deployed
- BThe models cannot be compared, because test scores use a different metric definition for each candidate model
- CThe training data becomes unusable, because every evaluation removes examples from the training set permanently
- DThe reported test score becomes optimistically biased because the test set influenced which model was selected
Show answer and explanationHide answer
Correct answer: DChoosing a model by its test score means the test set has guided selection, so the final score no longer estimates performance on truly unseen data. A separate validation set should be used for selection, with the test set kept for the final evaluation.
Question 24
During robustness testing of an image classifier, testers add small random noise and also apply subtle, deliberately crafted pixel changes to inputs, then check whether the predictions change. What name is given to such deliberately crafted inputs?
- AAdversarial examples, which are inputs designed to make the model predict wrongly despite only a small change
- BBoundary cases, which sit at the minimum or maximum limit of a valid input range for a given feature
- CSynthetic samples, which are generated to balance the number of examples across the different output classes
- DHeld-out instances, which are kept back from training so that the model's generalisation can be estimated
Show answer and explanationHide answer
Correct answer: AAdversarial examples are crafted perturbations that exploit model sensitivity, which makes them central to robustness testing. Boundary cases, synthetic samples and held-out instances are real terms, but they describe input limits, class balancing and evaluation data respectively.
Question 25
A loan model has similar overall accuracy for all applicants. A tester compares approval and error rates across age groups and finds that one group is rejected far more often, even though its repayment histories are comparable. What should the tester do next?
- AReport the disparity as a fairness issue and investigate the data and features that are driving the difference
- BAccept the model, because its overall accuracy is the same for all applicants in the test dataset that was used
- CRemove the test set and retrain the model, because overall accuracy alone should decide whether the model is fair
- DIgnore the difference, because approval rates are business decisions that fall outside the scope of testing
Show answer and explanationHide answer
Correct answer: AEqual overall accuracy can hide unequal outcomes for subgroups, so the tester should report the disparity and investigate its causes in the data and features. Accepting the model, retraining on a removed test set, or ignoring the gap would each hide the problem.
Question 26
A testing team applies a saliency-map technique that highlights which image regions most influenced a classifier's prediction. The highlights fall mainly on a watermark in the corner of many images rather than on the object being classified. What does this finding most likely indicate?
- AThe watermark causes underfitting, so the model should be given more hidden layers before it is retrained on the data
- BThe model is well trained, because it has found a strong feature that is present in every single image of the dataset
- CThe model may rely on a spurious feature, so its measured accuracy may not reflect the behaviour that was intended
- DThe dataset has too few images, which means the test set should be made larger before anything else is changed
Show answer and explanationHide answer
Correct answer: CRelying on a watermark rather than the object means the model learned a spurious correlation that may not hold in real use, so measured accuracy can be misleading. The other options misread the evidence: a shortcut is not good training, and the problem lies in the feature rather than in the dataset size or network depth.
Machine Learning Development Testing (4 questions)
Question 27
A data scientist writes a function that normalises numeric features and fills in missing values. The team adds unit tests with small hand-made inputs whose expected outputs are known. What does this practice verify?
- AThat the production data distribution still matches the distribution that was used in the training data
- BThat the trained model reaches the accuracy target set in the acceptance criteria for the forthcoming release
- CThat the preprocessing code produces the expected outputs for known inputs, so faults are caught before training starts
- DThat the model architecture chosen by the team is the most suitable one for the type of data being processed
Show answer and explanationHide answer
Correct answer: CUnit tests with known inputs and expected outputs check that preprocessing code behaves correctly before any training is run. Accuracy targets, drift checks and architecture choices are important, but they are assessed at other levels and are not what these unit tests verify.
Question 28
Before a long training run, an engineer checks that the training code can drive the loss on a tiny, fixed set of examples close to zero. Which practice is this?
- AA load test that measures how many training examples the available hardware can process in a given second
- BA regression test that confirms the model's accuracy on the full production dataset has not declined since release
- CA sanity check that the training pipeline can learn at all, which catches bugs before expensive training runs
- DA security test that checks whether the training data contains personal information that should be removed first
Show answer and explanationHide answer
Correct answer: CFitting a tiny fixed batch is a common sanity check: if the code cannot learn that batch, the pipeline likely has a bug. Regression, load and security tests are valid test types, but none of them checks whether the training code can learn at all.
Question 29
Two training runs on the same code and data produce noticeably different models. The team wants future runs to be reproducible for audits. Which practice best supports this goal?
- ADelete intermediate checkpoints after each run so that the storage used by experiments remains as small as possible
- BTrain only once per release and never re-run an experiment, so that each model stays unique and clearly accountable
- CUse the latest library versions automatically so that every run benefits from the newest optimisations that are available
- DFix the random seeds and record the code version, data version, library versions and hyperparameters for every training run
Show answer and explanationHide answer
Correct answer: DReproducibility depends on controlling randomness and recording everything that defines a run, including code, data, libraries and hyperparameters. Re-running is needed to check reproducibility, automatic upgrades introduce new variation, and deleting checkpoints removes evidence.
Question 30
An automated pipeline test compares the accuracy of a newly trained model with a stored baseline. The model is retrained with a different random initialisation each time, so its accuracy varies slightly between runs. Which test design is most appropriate?
- AAccept the result when the accuracy falls within an agreed tolerance range around the baseline, rather than requiring an exact match
- BRequire exact equality with the baseline accuracy to the last decimal place, because any deviation at all indicates a defect
- CSkip the comparison entirely, because stochastic training cannot be covered by automated checks in any meaningful way
- DCompare only the training loss, because accuracy on new data is always identical across models retrained on the same data
Show answer and explanationHide answer
Correct answer: AStochastic training produces small run-to-run variation, so tests should check results against an agreed tolerance rather than exact equality. Demanding exact matches would fail healthy runs, while skipping the check or comparing only training loss leaves the quality question unanswered.