Evaluation Metrics for Object Detection and Recognition

Introduction

Computer vision applications are now being used widely everywhere, and computer vision-related image editing and detection are the most common and basic tasks that developers perform. Object detection and recognition are used in e-commerce, CCTV surveillance, medical imaging, and autonomous driving. All the tasks mentioned above perform main activities like finding out the objects and localizing within the images and videos.

Evaluation metrics

Evaluation metrics will help to assess the performance of the object detection and recognition models in computer vision and deep learning tasks. These metrics benefit in quantifying the accuracy of the models and give valuable insights into the model's merits and demerits. Many find terms like IOU, AP, MIOU, mAP, etc., while writing and researching papers are the common metrics that are used in object detection. In this article, we will cover all the important metrics that help to identify the strengths and weaknesses of evaluation metrics.

Common Evaluation Metrics

These are the metrics that evaluate every machine learning and computer vision task's accuracy and performance. They are

Accuracy
F1 Score
Dice Coefficient
Intersection over Union (IoU)
Mean Intersection over Union (more)
Average Precision (AP)
Mean Average Precision (mAP)

Now, let's dive deep into every evaluation metric and its performance.

1. Accuracy

Accuracy is the complete basic metric that defines the model's overall performance and correctness in object recognition and detection. Accuracy is mostly utilized in the classification tasks like yes or no. However, accuracy can also be used in object detection and recognition as these tasks can also be classified, and their correctness can be calculated using the Accuracy metric.

The formula for Accuracy calculation is as follows:

Where:

TP (True Positives): It represents correct positive predictions.

TN (True Negatives): It represents correct negative predictions.

FP (False Positives): It represents incorrect positive predictions.

FN (False Negatives): It represents incorrect negative predictions.

Accuracy gives a simple basic calculation of how the model is working completely, and it may not be used in imbalanced datasets as different classes mislead other classes in terms of sample weights.

2. F1 Score

Before learning what the F1 score is, we have to look at two important terms. They are Precision and Recall.

Precision: It measures all the proportions of the true positives from all the positive predictions.

Recall: It measures the proportion of the true positives from all the actual positives.

Now, the F1 score will combine the above two metrics, Precision, and recall, into a single metric value. The f1 score range will be between 0 and 1. If the model's F1 score is 1, then the performance of the model is higher, and anything nearing a 0 F1 score states that the performance is worse.

The formula for F1 Score is:

3. Dice Coefficient

Image segmentation tasks use dice coefficient metric frequently for the analysis of images. This metric states the similar relationship between the two masks of predicted and ground truth. The Dice Coefficient has a range from 0 (no overlap) to 1 (perfect overlap).

The Dice Coefficient is called with the other name Sørensen-Dice index. The formula for it to be calculated is as follows:

Where:

|A ∩ B|: It is the size of the intersection between the predicted mask (A) and the ground truth mask (B).

|A| and |B| are the sizes of the individual masks.

4. Intersection over Union (IoU)

Intersection over Union is the common metric used in Object detection and Image segmentation tasks. It states the overlapping similarity between two boxes called the segmentation box and ground truth mask. The IoU metric has a range from 0 (no overlap) to 1 (perfect overlap).

The following steps can calculate the IoU.

Calculating the Area of Union
Calculating the Area of Intersection(Overlap between two boxes)
Dividing Area of Intersection by Area of Union

The formula is as follows:

5. Mean Intersection over Union(mIOU)

Miou(Mean Intersection over Union) is the primary metric used for the evaluation of the accuracy results of the image segmentation tasks. It will measure how well the segmentation masks relate to the ground masks. mIoU will estimate the level of cross-over and overlapping between the original and predicted segmentations. Miou is the overlapping between two boxes. If the overlapping region is bigger, then the value of Miou is also will be greater.

The steps for calculating MIOU are as follows:

Calculate IoU
Calculate the average(mean) of all IoU values

The formula for calculating mIoU is

here, the n is the total number of classes.

6. Average Precision (AP)

Average Precision will state the ability of the models to rank and retrieve valuable information. The calculation of this metric is done by generating a Precision recall curve(PR Curve) and then calculating the Area under the Curve, respectively.

The steps followed are

Firstly, Compute the Precision and recall at separate confidence thresholds for every class. This will ultimately give a set of precision and recall points. Then, plot the above points on the graph.
Precision will be on the y-axis, and recall on the x-axis. PR curve will be generated.
Now compute the Area under the PR curve. This Area is the average Precision for the specific class.
If there are multiple classes, compute AP for every class, and then calculate the mean (average) of these individual AP values to obtain the final AP.

7. Mean Average Precision (mAP)

In the previous Average Precision, all the individual objects are assessed, whereas in the mean Average Precision, the entire model's Precision is assessed. It is mostly used for multiple classes. The steps for calculating mAP are as follows:

Calculate AP for all the separate classes
Now, Calculate the mean of all the above AP values

Python Implementation

Code

# Import all necessary libraries
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score, average_precision_score, precision_recall_curve

# Simulated ground truth bounding boxes (format: [x1, y1, x2, y2])
ground_truth_boxes = np.array([[50, 50, 200, 200], [300, 300, 450, 450], [100, 100, 250, 250]])

# Simulated predicted bounding boxes (format: [x1, y1, x2, y2])
predicted_boxes = np.array([[60, 60, 190, 190], [310, 310, 440, 440], [90, 90, 240, 240]])

# Function to calculate the metric IoU (Intersection over Union) for a couple of bounding boxes
def calculate_iou(boxA, boxB):
    xA = max(boxA[0], boxB[0])
    yA = max(boxA[1], boxB[1])
    xB = min(boxA[2], boxB[2])
    yB = min(boxA[3], boxB[3])

    intersection_area = max(0, xB - xA + 1) * max(0, yB - yA + 1)
    boxA_area = (boxA[2] - boxA[0] + 1) * (boxA[3] - boxA[1] + 1)
    boxB_area = (boxB[2] - boxB[0] + 1) * (boxB[3] - boxB[1] + 1)

    iou = intersection_area / float(boxA_area + boxB_area - intersection_area)
    return iou

# Evaluating the object detection performance for different classes
iou_scores = []
binary_iou_scores = []

for true_box in ground_truth_boxes:
    iou_scores_for_true_box = [calculate_iou(true_box, pred_box) for pred_box in predicted_boxes]
    iou_scores.append(max(iou_scores_for_true_box))
    binary_iou_scores.append(max(iou_scores_for_true_box) >= 0.5)  # IoU threshold for correct detection

# Calculating other metrics Precision, Recall, and F1 Score
precision = precision_score([1, 1, 1], binary_iou_scores)
recall = recall_score([1, 1, 1], binary_iou_scores)
f1 = f1_score([1, 1, 1], binary_iou_scores)

# Calculating Accuracy
accuracy = accuracy_score([1, 1, 1], binary_iou_scores)

# Calculating Dice Coefficient
dice_coefficient = 2 * np.sum(binary_iou_scores) / (len(binary_iou_scores) + 1e-8)

# Calculating Average Precision (AP) using Precision-Recall Curve
precision_values, recall_values, _ = precision_recall_curve([1, 1, 1], [max(iou_scores_for_true_box) for _ in range(len(predicted_boxes))])
average_precision = average_precision_score([1, 1, 1], [max(iou_scores_for_true_box) for _ in range(len(predicted_boxes))])

# Calculating Mean Average Precision (mAP) by finding the average of AP over many multiple classes
# In this below example, we have a single class so that mAP  will become equal to AP
mAP = average_precision

# Calculating Mean Intersection over Union (mIoU)
mIoU = np.mean(iou_scores)

# Printing the final results of different evaluation metrics
print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 Score: {f1:.2f}")
print(f"Accuracy: {accuracy:.2f}")
print(f"Dice Coefficient: {dice_coefficient:.2f}")
print(f"Average Precision (AP): {average_precision:.2f}")
print(f"Mean Average Precision (mAP): {mAP:.2f}")
print(f"Mean Intersection over Union (mIoU): {mIoU:.2f}")

Output:

Precision: 1.00
Recall: 1.00
F1 Score: 1.00
Accuracy: 1.00
Dice Coefficient: 2.00
Average Precision (AP): 1.00
Mean Average Precision (mAP): 1.00
Mean Intersection over Union (mIoU): 0.76

Next TopicMean Intersection over Union (mIoU) for image segmentation

← prev next →