On a customized dataset, the AP is low, like under 10%, but the mIoU is high, like above 60%.

[hehongjie]

Hello,

Thank you for your great work.

I am currently running the Oneformer with a customized dataset for unified image segmentation. I encountered two issues in fact.

1. When enabling the AMP, the model training has the NaN error. If it is disabled, the model can be trained without any errors reported.
2. On a custome dataset, where only 1 class belongs to "thing" class (building), the AP is low to under 10%, and the mIoU is higher than 60%. Here are some training log information attached below.

I arranged my dataset into COCO format and directrly use COCO dataset mapper and registry function.

[03/31 17:39:39 oneformer.evaluation.coco_evaluator]: Evaluating predictions with unofficial COCO API...
Loading and preparing results...
DONE (t=0.45s)
creating index...
index created!
[03/31 17:39:40 d2.evaluation.fast_eval_api]: Evaluate annotation type segm
[03/31 17:39:41 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 0.80 seconds.
[03/31 17:39:41 d2.evaluation.fast_eval_api]: Accumulating evaluation results...
[03/31 17:39:41 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 0.07 seconds.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.023
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.061
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.011
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.006
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.023
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.032
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.005
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.025
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.042
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.045
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.051
[03/31 17:39:41 oneformer.evaluation.coco_evaluator]: Evaluation results for segm:

[03/31 17:39:42 d2.evaluation.sem_seg_evaluation]: OrderedDict([('sem_seg', {'mIoU': 60.20418008168539, 'fwIoU': 64.45364952087402, 'IoU-building': 79.00359034538269, 'IoU-road': 64.76172804832458, 'IoU-water': 68.51374506950378, 'IoU-barren': 27.051246166229248, 'IoU-forest': 54.615336656570435, 'IoU-agriculture': 67.27946996688843, 'mACC': 75.01688798268637, 'pACC': 77.73436903953552, 'ACC-building': 88.89179229736328, 'ACC-road': 75.64088106155396, 'ACC-water': 91.73657298088074, 'ACC-barren': 39.59067761898041, 'ACC-forest': 79.11878228187561, 'ACC-agriculture': 75.12264251708984})])

[03/31 17:39:42 d2.evaluation.testing]: copypaste: Task: panoptic_seg
[03/31 17:39:42 d2.evaluation.testing]: copypaste: PQ,SQ,RQ,PQ_th,SQ_th,RQ_th,PQ_st,SQ_st,RQ_st
[03/31 17:39:42 d2.evaluation.testing]: copypaste: 1099.6590,2251.9732,52.7991,49.5542,80.5105,61.5500,1309.6799,2686.2657,51.0490

[03/31 17:39:42 d2.evaluation.testing]: copypaste: Task: segm
[03/31 17:39:42 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl
[03/31 17:39:42 d2.evaluation.testing]: copypaste: 2.2837,6.1118,1.1393,0.5980,2.3482,3.1830

[03/31 17:39:42 d2.evaluation.testing]: copypaste: Task: sem_seg
[03/31 17:39:42 d2.evaluation.testing]: copypaste: mIoU,fwIoU,mACC,pACC
[03/31 17:39:42 d2.evaluation.testing]: copypaste: 60.2042,64.4536,75.0169,77.7344