Project: Simple Classification Experiment

One-sentence summary

We will build a simple rule-based (threshold) classifier on a small dataset, test it on training and test data, measure its accuracy, and honestly discuss the limits of its results.

Why does it matter?

Classification means placing an example into the correct group: sorting an email into "spam" or "not spam," or labelling a fruit as "apple" or "orange." Most artificial-intelligence tools do some kind of grouping like this behind the scenes.

In this lesson we will build a working classifier with our own hands. But let us be honest from the start: we are not training a real AI model here. We will write the rules ourselves. This is the best way to see that AI is not mysterious magic; in the end it rests on simple decisions that catch a pattern in the data. When you write the rule yourself, you also see with your own eyes where a machine can go wrong.

What is classification?

Example and label

Classification involves two things:

• Example: The thing we decide about (a fruit, a message, a weather reading).
• Label: The group the example belongs to (apple/orange, spam/normal).

Our goal is to find a rule that looks at an example and predicts the correct label.

Two everyday examples

Example 1 — Fruit basket. A basket holds apples and oranges. Even with our eyes closed, weight alone often lets us tell them apart: oranges are usually heavier. A rule like "if the weight is above 150 grams it is an orange, otherwise an apple" gets most of the job right.

Example 2 — Message inbox. If a message contains words like "free," "click now," or "you won," it is probably an unwanted (spam) message. The rule "if any of these keywords appear, it is spam" is simple but useful.

In both cases we do the same thing: we look at one feature (weight, a keyword) and decide based on a threshold or rule.

A threshold-based classifier

The idea

Suppose we have each fruit's weight (in grams) and its true label. We pick a threshold: examples above it are oranges, examples below it are apples. Finding a good threshold is what "building" the classifier means.

Why split training and test data?

We divide the data into two parts:

• Training data: The examples we look at while choosing the threshold.
• Test data: Examples we have never seen before, used to check the classifier after the threshold is chosen.

Why? Because we can only tell whether a rule is truly good by trying it on examples it has not seen. It is easy to look good on examples whose answers we already know; the real question is whether it gets a new fruit right.

Code: create the data and split it

# Each example: (weight_grams, true_label)
data = [
    (120, "apple"), (130, "apple"), (140, "apple"),
    (135, "apple"), (125, "apple"), (110, "apple"),
    (160, "orange"), (170, "orange"), (180, "orange"),
    (155, "orange"), (175, "orange"), (165, "orange"),
]

# First half for training, second half for testing
training = data[:6] + data[6:9]   # some apples + some oranges
test = data[3:6] + data[9:]       # the remaining examples

Code: predict from the threshold

def predict(weight, threshold):
    # Above the threshold is orange, otherwise apple
    if weight > threshold:
        return "orange"
    else:
        return "apple"

Code: measure the accuracy

Accuracy is the number of examples we get right divided by the total number of examples.

def accuracy(dataset, threshold):
    correct = 0
    for weight, actual in dataset:
        if predict(weight, threshold) == actual:
            correct += 1
    return correct / len(dataset)

threshold = 150
print("Training accuracy:", accuracy(training, threshold))
print("Test accuracy:", accuracy(test, threshold))

The operation correct / len(...) gives a number between 0 and 1. A value of 1.0 means "got them all right," and 0.5 means "got half right."

Mini practice

Try the steps below on your own computer (with an adult nearby):

1. Paste the three code blocks above, in order, into a single Python file.
2. Run it and note the training and test accuracy.
3. Change the line threshold = 150: try 145, then 135. How does the accuracy change?
4. Find the threshold that gives the best test accuracy and write one sentence explaining why you chose it.

A mistake and its fix

In my first try I set the threshold too low:

Goal: Separate apples and oranges
Problem: With threshold = 100 everything was called "orange"; apples came out wrong
Check: The lightest orange is 155 g and the heaviest apple is 140 g; the border must sit between them
Fix: I set threshold = 150; now both groups are separated correctly

Lesson: The threshold must sit inside the gap that separates the two groups. We chose the rule by looking at the data itself, not by guessing.

An idea to improve it

So far we used a single feature (weight). You could add a second feature, such as colour. A two-condition rule like "orange if the weight is above 150 and the colour is orange" may make fewer mistakes on borderline examples. Before making the rule more complex, measure the accuracy of the single feature and ask whether the extra rule is really needed.

Common mistakes

Mixing up training and test

If you choose the threshold by looking at the test data, high accuracy will mislead you. The test data must stay unseen while you decide.

Trusting accuracy blindly

If the data had 90 oranges and 10 apples, even a rule that says "everything is an orange" would score 90% accuracy. But that rule never recognises a single apple. Accuracy alone is not enough; you also need to see which group the mistakes fall in.

Drawing big conclusions from small data

Twelve examples do not represent the real world. Our threshold works in this basket; it may fail on another orchard's fruit. A rule found from little data does not mean "correct everywhere."

Ignoring bias

If, by chance, all the oranges in our training data are very heavy, the classifier will mistake small oranges for apples. The rule learns whatever the data shows; if the data is biased, the rule becomes biased too.

Safety note

• Do not use personal data. In this experiment we use harmless, made-up numbers like fruit weights. Never put personal information — your own or anyone else's name, photo, address, or phone number — into a dataset or an online AI tool.
• A human is responsible for the result. If this classifier mislabels a fruit, no harm is done. But in real life classifiers can make decisions about people (who qualifies for a loan, whose message gets blocked). In such decisions a human must check the result; the machine's output should not be the final word.
• If the data is biased, the result is biased. Do not blindly accept an important output, especially one about a person; ask where it came from and what data it is based on.
• Use AI tools within their age rules and together with an adult.

Inspect the errors, not only the score

A single accuracy percentage can hide very different failure patterns. Separate true results, false positives and false negatives for each class, then look for shared conditions such as background, lighting, angle or unequal training examples. Change only one part of the data or model at a time and reuse the same held-out test set so that the comparison remains fair.

The activity must avoid personal or sensitive data. The result is a classroom model trained on a limited dataset, not a dependable system for judging people or making important decisions. Its limitations and unsuccessful examples belong in the project evidence.

Lesson summary

• Classification places an example into the correct label (group); we did it with a simple threshold-based rule.
• We did not train a real model; we wrote the rule ourselves, so we know exactly how it works.
• Splitting data into training and test lets us honestly measure whether the rule also works on new examples.
• Accuracy is the number of correct predictions over the total; but on its own it is not enough and can mislead on unbalanced data.
• Biased or small data produces biased results; on important decisions a human must have the final say.

Check questions

1. What is the difference between training data and test data, and why do we keep the test set separate?
2. What does our threshold-based classifier look at to make its decision?
3. If the accuracy comes out as 0.5, what does that mean?
4. On data with 90 oranges and 10 apples, what accuracy does the rule "everything is an orange" get, and why is this misleading?
5. Why do we say that we did not train a real AI model in this experiment?

Answers

1. Training data are the examples we look at while choosing the threshold; test data are examples we have not seen before the choice. We keep the test set separate because we can only tell whether a rule is truly good on examples it has not seen.
2. It looks at a single feature, the fruit's weight; if the weight is above the threshold it says "orange," otherwise "apple."
3. It means the rule got half of the examples right and half wrong. In a two-group problem that is only as good as random guessing, so the rule is not really working.
4. The accuracy is 0.90 (90%) because 90% of the examples are already oranges. It is misleading because the rule never gets a single apple right; the high number hides the real performance.
5. Because we chose the rule (the threshold) ourselves by looking at the pattern in the data; the machine did not learn on its own. It is an honest, instructive experiment, not a truly trained model.

Source and verification note

For “Project: Simple Classification Experiment”, verification focuses on whether the relationship between What is classification? and Two everyday examples remains consistent across examples. Datasets in this module are small and educational; real personal data should not be used. An AI result should be evaluated not only for accuracy but also for data balance, error distribution and explainability.

Next lesson

Project Workshop module: A workshop where you combine the coding, robotics, and data ideas you have learned to design and build your own project from start to finish.