Point Clouds Ground Truth class Predicted class
Chair Chair
Chair Chair
Vase Vase
Lamp Lamp Visualization of wrong class prediction:
Vase Lamp
Lamp Vase
Interpretation: The Classification model performance is overall good. It is only failing for some objects that seem to be very close to another class. For instance, the vase is having a shape of lamp, and the lamp is having a lot of similaries with a flower vase. Therefore, such images are ambiguous for the model and makes the model to mispredict.
Segmentation Model =============================================================================== Model's best accuracy achieved: 0.8888 Visualization of a few Segmentation results with corresponding ground truth:
Ground Truth Prediction Accuracy
0.9725
0.9515
0.9506
Segmentation of Bad Prediction with lower accuracy:
0.5632
0.6494
Interpretation: The Segmentation model performance is overall good. It is only failing for some objects that are difficult to segment. For instance, there are some bad performaed cases for large overlaps between different parts of the sofa, or between those parts that are not clearly seperated. Therefore, such images are ambiguous for the model and makes it very difficult for the model to segment clearly.
Robustness Analysis =============================================================================== Experiment 1: Rotate the input point clouds -------------------------------------------------------------------------------
Classification Visualization:
Rotation Angle Visualization Ground Truth class Predicted class Accuracy
0 
Chair Chair 0.8289
10 
Chair Chair 0.8216
20 
Chair Lamp 0.6075
30 
Chair Vase 0.3410
40 
Chair Vase 0.2717
90 
Chair Vase 0.2203
180 
Chair Vase 0.5005
Segmentation Visualization:
Rotation Angle Ground Truth Prediction Accuracy
0 
0.7693
10 
0.6578
20 
0.5727
30 
0.5420
40 
0.5360
90 
0.3240
180 
0.2420
Interpretation: From the results, we can tell that the test accuracies decrease for both the Classification and Segmentation models. For Classification model, the model performs poorly when the angle is increased. However, at 180° the model showed a better accuracy, because the chair looks like a chair, it is just inverted. For the Segmentation model, the accuracies fall with the increase in angle.
Experiment 2: Change number of points -------------------------------------------------------------------------------
Classification Visualization:
Number of Points Visualization Ground Truth class Predicted class Accuracy
50 
Chair Lamp 0.3034
100 
Chair Lamp 0.5519
500 
Chair Chair 0.8006
1000 
Chair Chair 0.8289
5000 
Chair Chair 0.8331
10000 
Chair Chair 0.8289
Segmentation Visualization:
Number of Points Ground Truth Prediction Accuracy
50 
0.8845
100 
0.8856
500 
0.8870
1000 
0.8960
5000 
0.9200
10000 
0.9400
Interpretation: From the results, it is evident that with number of points, the accuracy increases for both the tasks - Classificationa and Segmentation. However, it is seen that after certain number of points, the accuracy gets stagnant and number of points is not an effective parameter then. For instance, in the Classification task the accuracy is about 0.82-0.83 for the number of points more than 1000. Also, it is the same case for class prediction. After, 1000 points, the model predicts the correct class.