Unit 3: Evaluating Models Class 10 AI Notes: Important Points

Share with others

Unit 3: Evaluating Models Class 10 AI Notes: Important Points

In this Unit 3: Evaluating Models Class 10 AI Notes:, students are introduced to the common metrics used to evaluate AI models. They will know how to derive and calculate the evaluation metrics and would also get an idea on how to improve the accuracy/efficiency of an AI Model. They will be introduced to the concept of Train/ Test Split, Common evaluation metrics such as Accuracy, Confusion Matrix, Precision, Recall, F1 Score. Learners will also be able to identify the use of this metrics in use cases encountered in everyday life.

What is Model evaluation?

Model evaluation is the process of using different evaluation metrics to understand a machine learning model’s performance.

When you build an AI model, you get feedback from metrics, make improvements and continue until you achieve the desirable accuracy.

What is the need of Model evaluation?

It helps you to understand the strengths, weaknesses, and suitability of AI Model. This feedback loop is essential for building trustworthy and reliable AI systems.

What are the different Evaluation techniques?

There can be different Evaluation techniques, depending of the type and purpose of the model.

Train-test split

The train-test split is a technique for evaluating the performance for any supervised learning algorithm. In this procedure, we take the dataset and divide it into two subsets: The training dataset and the testing dataset. This procedure is appropriate when there is a sufficiently large dataset available.

What is the need of Train-test split?

This train-test split is required to estimate the performance of the machine learning model on new data ie the data which is not used to train the model.

When we use the training dataset for evaluating the model then the AI model will always predict the correct label for any point in the training set. This is known as overfitting.

What is Accuracy and Error?

Accuracy is an evaluation metric that allows you to measure the total number of predictions a model getsright.

Error can be described as an action that is inaccurate or wrong.

Let we understand this with the help of the following illustrations.

Error: If the model predicts you don’t have a disease but you actually have a disease, that’s an error. The error quantifies how far off the prediction was from reality.

Accuracy: If the model correctly predicts disease or no disease for a particular period, it has 100% accuracy for that period.

How to calculate the error and the accuracy of House Price prediction AI model?

Read the instructions and fill in the blank cells in the table.
The formula for finding error and accuracy is shown in the table
Accuracy of the AI model is the mean accuracy of all five samples
Percentage accuracy can be seen by multiplying the accuracy with 100

Predicted House Price	Actual House Price	Error Abs (Actual-Predicted)	Error Rate (Error/Actual)	Accuracy (1-Error rate)	Accuracy% (Accuracy*100)%
391k	402k	abs(402- 391) = 11k	11k/402k=0.027	1-0.027=0.973	*0.973100= 97.3%**
453k	488k	abs(488- 453) = 35k	35k/488k = 0.071	1-0.071=0.929	*0.929100= 92.9%**
125k	97k	abs(97- 125) = 28k	28k/97k = 0.288	1-0.288=0.712	*0.712100 = 71.2%**
871k	907k	abs(907- 871) = 36k	36k/907k =0.039	1-0.039=0.961	*0.961100 = 96.1%**
322k	425k	abs(425- 322) = 103k	103k/425k = 0.242	1-0.242=0.758	*0.758 100 = 75.8%**

Evaluating Models Class 10 AI Notes

NOTE: Error refers to the difference between a model’s prediction and the actual outcome. It quantifies how often the model makes mistakes.

NOTE: Abs means the absolute value, which means only the magnitude of the difference without any negative sign (if any)

Evaluation metrics for Classification

What is Classification?

Classification refers to an AI task in which data is arranged into different categories based on their features. for example an AI model classifying the items of the supermarket into two classes:

▪fruits and vegetables
▪grocery

Q1. Which of the following is a classification use case example?

House price prediction
Credit card fraud detection
Salary prediction

Ans. Credit card fraud detection → Classification (classified into : Fraud or Not Fraud)

Classification Metrics

Popular metrics used for classification model

Confusion matrix
Classification accuracy
Precision
Recall

1. Confusion matrix

The confusion matrix is a handy presentation of the accuracy of a model with two or more classes. In this matrix the table presents the actual values on the y-axis and predicted values on the x-axis.

for example: You have designed a classifier that predicts whether a person is infected with a certain disease or not. The output is 1 if the person is infected or 0 if the person is not infected.

Box with number 12 shows: Predictions for 1 that were actually 1 appear in this cell. Means the person infected with the disease is also predicted as being not infected.

Box with number 6 shows: Predictions for 0 that were actually 1 appear in this cell. Means the person infected with the disease is predicted as not being infected.

Box with number 4 shows: Predictions for 1 that were actually 0 appear in this cell. Means the person not infected with the disease is predicted as being infected.

Box with number 21 shows: Predictions for 0 that were actually 0 appear in this cell. Means the person not infected with the disease is also predicted as being not infected.

How to Build the confusion matrix?

Let we assume that an AI model is predicting the presence of a disease. This model will have output as Yes or No where “Yes” would mean they have the disease, and “No” would mean they don’t have the disease. Now we will construct the confusion matrix from the chart given below:

Actual Value	Predicted Value
Yes	Yes
No	No
No	Yes
Yes	No
No	No
Yes	Yes
Yes	No
No	No
No	No
No	No

Evaluating Models Class 10 AI Notes

Now we will fill the matrix based on the table given above

Reflection of above Matrix:

There are 07 correct predictions out of 10 predictions

True Positive:

True Positive (TP) is the outcome of the model correctly predicting the positive class. Any class can be assumed as a positive class, and the rest can be assumed as negative. Let’s say class 1 is assumed as the positive class. Can you tell the TP value from the matrix given below

Value of True Positive value is : 12

Let we take another example of True Positive:

1. You had predicted that INDIA would win the cricket world cup, and it won.

2. In the earlier activity, the cases in which we predicted yes (they have the disease), and they do have the disease.

True Negative:

True Negative (TN) is the outcome of the model correctly predicting the negative class. Since in the previous example, class 1 is assumed the positive class so class 0 should be assumed the negative class.

Value of True Negative value is : 21

Let we take another example of True Negative:

1. You had predicted that Germany would not win, and it lost.

2. In the earlier activity, the cases in which we predicted No (they don’t have the disease), and they don’t have the disease.

False Positive

False Positive (FP) is the outcome of the model wrongly predicting the negative class as positive class. Here, when a class 0 is predicted as class 1, it falls into the FP cell.

Value of False Positive value is : 4

False Negative

False Negative (FN) is the outcome of the model wrongly predicting the positive class as the negative class. Here, when class 1 is predicted as class 0, it falls into the FN cell.

Value of False Negative value is : 6

Summary of above matrix

How to Calculate the Classification accuracy from Confusion matrix?.

Classification accuracy is the ratio of the number of correct predictions made to all of all predictions made.

Classification Accuracy = Correct Prediction/Total Prediction

= (TP + TN)/(TP + TN + FP + FN)

= (12 + 21)/(12 + 21 + 6 + 4)

=0.767

How to calculate Precision from Confusion matrix?

Precision is the ratio of the total number of correctly classified positive examples and the total number of predicted positive examples. When Precision = 0.843 means that when our model predicts a patient has heart disease, it is correct around 84% of the time.

Precision = Correct positive predictions/Total positive predictions

=TP/TP+FP

For example

Suppose there are 50 emails. The AI says 10 emails are spam.

Out of these 10: 9 are actually spam and 1 is actually important emails (False Positive). So,

Precision = 9 / (9 + 1) = 90%

This means whenever the AI says “Spam”, it is correct 90% of the time.

A real-life situation where Precision is the most important

Suppose ISRO wants to launch a satellite.

Let’s assume a day with favorable weather condition is considered Positive class and a day with non-favorable weather condition is considered as Negative class.

Missing out on predicting a good weather day is okay (low recall) but predicting the bad weather day (Negative class) as a good weather day (Positive class) to launch the satellite can be disastrous. So, in this case, the FPs need to be reduced as much as possible.

Actual Weather	AI Prediction	Result
Good	Good	Correct
Good	Bad	False Negative (Launch delayed)
Bad	Good	False Positive (Very Dangerous)
Bad	Bad	Correct

Evaluating Models Class 10 AI Notes

Recall from Confusion matrix

Recall is the measure of our model correctly identifying True Positives. Recall = 0.86 tells us that out of the total patients who have heart disease 86% have been correctly identified.

Recall = Correct positive predictions/Total actual positive values

= TP/(TP+FN)

NOTE: Recall is also called as Sensitivity or True Positive Rate?

The metrics Recall is generally used for unbalanced dataset when dealing with the False Negatives become important and the model needs to reduce the FNs as much as possible.

For Example: Let’s consider detection of a covid-19 affected case as positive class and detection of covid-19 non-affected case as negative class. Imagine if a covid-19 affected person (Positive) is falsely predicted as non-affected of Covid-19 (Negative), the person if rely solely on the AI would not get any treatment and also may end up infecting many other persons.

So, in this case, the FNs needs to be reduced as much as possible.

F1 Score

F1-Score provides a way to combine both precisions and recall into a single measure that captures both properties. In those use cases, where the dataset is unbalanced, and we are unable to decide whether FP is more important or FN, we should use the F1 score as the suitable metric.

F1 Score = 2* Precision x Recall/(Precision + Recall)

How to find the appropriate metric to evaluate the AI model?

Different evaluation metrics are used for evaluation in different scenarios and it is important that we should learn how to choose the correct one.

Scenario: Flagging fraudulent transactions

You have designed a model to detect any fraudulent transactions with credit card. Now you are testing your model with highly unbalanced dataset.

It is okay to classify a legit transaction as fraudulent — it can always be re-verified by passing through additional checks.

But it is definitely not okay to classify a fraudulent transaction as legit (false negative). So here false negatives should be reduced as much as possible. Hence in this case, Recall is more important. Construct the confusion matrix for the data given below and calculate the recall from the confusion matrix.

Actual Value	Predicted Value
Non-Fraudulent	Non-Fraudulent
Non-Fraudulent	Fraudulent
Non-Fraudulent	Non-Fraudulent
Fraudulent	Non-Fraudulent
Fraudulent	Fraudulent
Non-Fraudulent	Non-Fraudulent
Fraudulent	Non-Fraudulent
Non-Fraudulent	Fraudulent
Non-Fraudulent	Non-Fraudulent
Non-Fraudulent	Non-Fraudulent

Evaluating Models Class 10 AI Notes

We will create the confusion matrix from the table given above

Calculate the recall from the confusion matrix based on.

Recall = TP/(TP+FN)

= 1/(1 + 2)

=1/3

= 0.33

Ethical concerns around model evaluation

While evaluating an AI model, the following ethical concerns need to be kept in mind

Bias: Ensure that the evaluation metrics chosen don’t result in any kind of bias

Transparency: Honest explanation how the chosen evaluation metrics work and produce results without keeping any information hidden

Accountability: Take responsibility for your choice of metrics and methodology of evaluation in case any user faces a disadvantage because of your chosen methodology

Disclaimer : I tried to give you the easy handouts of “Unit 3: Evaluating Models Class 10 AI Notes: Important Points“, but if you feel that there is/are mistakes in the handouts of “Unit 3: Evaluating Models Class 10 AI Notes: Important Points“given above, you can directly contact me at csiplearninghub@gmail.com. NCERT Book and Study material available on CBSE official website are used as a reference to create above “Unit 3: Evaluating Models Class 10 AI Notes: Important Points“ All the screenshots used in above article are taken from NCERT Book and Study material available on CBSE official website.

This Unit 3: Evaluating Models Class 10 AI Notes: Important Points covers the following topics

What is Train-test split? Why do we need to do Train-test split? Accuracy and Error. What is Classification?F1 Score, Classification metrics, Confusion matrix, Accuracy from Confusion matrix, Precision from Confusion matrix, Recall from Confusion matrix, Bias, Accountability, Transparency