SDD  ·  Revision  ·  Section E

Testing & Evaluation
Error Types, Test Data and the Four SQA Criteria

Software Design & Development revision Approx 30–40 min 8 scenarios  ·  justification practice
Recap: SDD16 names three error types — syntax (breaks Python's grammar; detected before the program runs; it never starts), execution/run-time (syntax is fine, but something goes wrong once the program is running and it crashes with an error message), and logic (the program runs to completion with no error message, but the output is wrong) — and three test data types for a test plan: normal (typical values well inside the valid range), extreme (the exact boundary values, minimum and maximum), and exceptional (values outside the range, or the wrong data type, that should be rejected). SDD17 names the four SQA evaluation criteria — fitness for purpose (does it meet the stated requirements), robustness (does it handle invalid or unexpected input without crashing), efficient use of coding constructs (e.g. an ELIF chain instead of repeated separate IFs), and readability (internal commentary, meaningful identifiers, indentation, white space).  → SDD16  ·  SDD17
How to use this drill: Each scenario names a specific situation and offers two options to choose between — sometimes two error types, sometimes two test data types, sometimes two evaluation criteria, and the two options change from scenario to scenario. Select the option that fits, then write a justification quoting the exact detail in the scenario that points to it and rules out the other option. Reveal the model answer to check your reasoning.
Your choices and justifications are saved automatically.
Section E — Testing & Evaluation

Eight scenarios, each offering a different pair of testing or evaluation concepts. Select the option that best fits, justify your choice using the scenario, then check the model answer. Scenarios 4 and 8 are marked DISCUSS — either answer can be correct if well-justified.

Scenario 1 of 8 Litter-picking rota generator won't start

Miss Whyte writes a program to allocate the weekly litter-picking rota. The instant she clicks Run in PyCharm, nothing happens at all — no output appears, and PyCharm immediately underlines for pupil in rota in red, reporting that the line is missing a colon at the end.

Expected: Syntax error Python checks the grammar of the whole program before it runs anything. The missing colon after for pupil in rota breaks that grammar, so Python stops before execution begins — nothing is printed at all. This is not an execution error: an execution error can only happen once a program has already started running, and here the program never starts.
Scenario 2 of 8 Dinner-money calculator crashes on pupil 30

A dinner-money balance calculator reads a 30-item list of pupil balances and prints each one in turn. It correctly prints balances for the first 29 pupils, then suddenly stops with the message IndexError: list index out of range on the 30th pass through the loop, because the loop condition allows index to reach 30 in a list of 30 items (valid indexes 0–29).

Expected: Execution error The program starts and runs correctly for 29 iterations, so the syntax was fine — but it then crashes mid-run with a visible Python error message, IndexError, and stops before finishing. This is not a logic error: a logic error would let the program run all the way to completion with no crash, just producing a wrong answer, whereas here the program actually halts.
Scenario 3 of 8 Quiz percentage always one question short

A ten-question quiz-marking program always finishes and always displays a percentage score, with no error message. When Mr Sinclair tests it by getting every single question right, it displays 90% instead of 100%, because the FOR loop marking each answer is written as range(9) instead of range(10), so the last question is never checked.

Expected: Logic error The program runs from start to finish every time and never crashes — there is no error message at any point — but the output, 90% instead of 100%, is wrong. This is exactly what defines a logic error: the mistake is in the programmer's reasoning (range(9) misses the last index), not in the code's grammar or something that makes Python halt. This is not an execution error, because nothing crashes.
Scenario 4 of 8 Testing the detention threshold with the number 5 Discuss

A homework-lateness checker accepts the number of days a piece of work is late, from 0 to 5 inclusive, and should print "OK" for 0–2 days, "Warning" for 3–4 days, and "Detention" for 5 or more. Mr Reid deliberately tests the program by entering exactly 5 — the value sits right on the edge both of the valid input range and of the Warning/Detention boundary. The program runs without crashing but wrongly prints "Warning" instead of "Detention", because the condition was written as days > 5 instead of days >= 5.

Expected: Either — depending on what the question actually asks If choosing Extreme test data: The value 5 sits exactly on the boundary Mr Reid deliberately chose to test — the maximum accepted value and the Warning/Detention split — which is precisely the definition of extreme (boundary) test data from SDD16.

If choosing Logic error: Once the test has been run, the defect it revealed — the program continuing without crashing but producing the wrong category — is precisely the definition of a logic error from SDD16.

Exam note: This scenario shows why command words matter. "What type of test data did Mr Reid use?" and "What type of error does this test reveal?" are two different questions about the exact same scenario, and they have two different correct answers. A weak answer picks one label and applies it to both questions. A strong answer reads the actual question stem before answering, and can explain both: the extreme test data is what caught the logic error.
Scenario 5 of 8 Testing an after-school club's age limit

An after-school coding club sign-up form only accepts pupils aged 12 to 16 inclusive. When designing the test plan, a pupil chooses to test the program by entering exactly 12 — the lowest age the form is supposed to accept.

Expected: Extreme test data 12 is the exact minimum boundary of the valid range 12–16, which is the definition of extreme (boundary) test data. This is not normal test data: normal data is a typical value well inside the range, such as 14 — it does not sit exactly on an edge where an off-by-one mistake (using > instead of >=) could wrongly reject a valid age.
Scenario 6 of 8 Three separate IFs instead of one chain

A pupil evaluating a grade-band program writes: "Lines 4 to 15 use three separate IF statements, one after another, to test the same score variable, even though only one of the three conditions can ever be true for a given score." Every variable in those lines is clearly named and there is a comment above each block.

Expected: Efficient use of coding constructs Using three separate IF statements where only one condition can ever be true means Python evaluates every remaining condition even after a match has already been found — an ELIF chain would stop checking as soon as one condition is met. This is a construct-choice issue, not a readability one: the scenario explicitly says the identifiers are clear and the comments are present, which rules out readability.
Scenario 7 of 8 Correct average, unreadable code

A program is required to calculate and display the average of five pupils' spelling scores. Testing confirms it does exactly this, correctly, every time. However, every variable in the program is named a, b, c, i or t, and there is not a single comment anywhere in the script.

Expected: Readability The program's meaningful-identifier and internal-commentary sub-criteria both fail — a, b, c, i and t give no clue what they hold, and there are no comments at all. This is not a fitness for purpose issue: the scenario states the program correctly calculates and displays the required average every time, so the stated requirement is fully met — the weakness is entirely in how understandable the code is to another programmer.
Scenario 8 of 8 Fixture sign-up accepts "banana" as a time Discuss

An interhouse fixtures sign-up program's only stated requirements are to: (a) accept a house name, (b) accept a start time, and (c) display a confirmation message. There is no requirement to validate the format of the start time. When Mrs Adams tests it by typing "banana" as the start time, the program does not crash — it happily displays "Fixture confirmed for banana" and saves the entry.

Expected: Either — but a strong answer names both If choosing Fitness for purpose: The program does exactly what the three stated requirements ask — it accepts a house name, accepts a start time (of any text), and displays a confirmation — so judged strictly against the stated requirements, it is fit for purpose.

If choosing Robustness: Robustness is about whether a program handles invalid or nonsensical input sensibly, whether or not that was explicitly written into the requirements — and saving "banana" as a fixture time without any check is exactly the kind of unhandled invalid input SDD17 defines robustness around.

Exam note: This is the sharpest DISCUSS case, and it is the single most common confusion the SQA penalises: a program can be fully fit for purpose (it meets every stated requirement) while still not being robust (it doesn't cope sensibly with input nobody thought to rule out). A weak answer treats the two criteria as interchangeable and only names one. A strong answer explains that they are judging different things here, and that both verdicts can be true of the same program at once.
Marker's eye view — why candidates lose marks

For error types, the deciding question is always: did the program ever start, and did it ever crash? Never started at all → syntax error. Started, ran for a while, then crashed with a visible Python error message → execution (run-time) error. Ran all the way to the end with no crash, but gave the wrong answer → logic error. Naming the error type without saying which of these happened — "it's an error because the code is wrong" — scores nothing.

For test data, the deciding question is: where does this value sit relative to the valid range? Comfortably inside it → normal. Exactly on the edge (the minimum or maximum accepted value) → extreme. Outside it, or the wrong data type altogether → exceptional. The extreme values are the ones most likely to expose an off-by-one logic error, because > versus >= only makes a difference right at the boundary.

For evaluation, always name, judge, and justify using the exact SQA wording — "efficient use of coding constructs", not "good code"; and for readability, cover whichever of the four sub-criteria (commentary, meaningful identifiers, indentation, white space) the evidence actually supports, not just one of them. The single most common mix-up is fitness for purpose versus robustness: fitness for purpose asks whether the stated requirements are met, while robustness asks whether invalid input is handled sensibly — a program can pass one and fail the other, as in Scenario 8.