Test Dataset Download Instructions

Download the test dataset from here: IDDAW Test Images

Download Instructions

  • Click to download the Dataset and more details. This will redirect you to the registration page to download the dataset.
  • Register an account at IDD Insaan website.
  • Go to Dataset > Download page in the menu.
  • Download the IDD-AW Dataset by clicking on 'Download' under 'IDD AW Dataset'. This will generate a 24 hour token to download the dataset.
  • Extract the downloaded compressed file into a folder.

    • IDD-AW Dataset page
    Download here

    The dataset, including training data, ground truth, and an evaluation script, will be made publicly available post-competition in accordance with ICPR guidelines.

    Evaluation

    Heirarchical Labeling and Tree Distance

    The essence of SmIoU lies in the introduction of hierarchical penalty, a strategy that takes into account the semantic relationships between classes. Heirarchical Labeling refers to the structured organization of classes into a tree-like hierarchy, capturing semantic relationships and dependencies between them in the context of semantic segmentation. It provides a inherent semantic relationship between classes and also provides structured heirarchy for a comprehensive understanding of classes.

    Misclassifications within critical classes, and non-critical classes classified as critical, are penalized based on their distance in the class hierarchy. The tree distance (td) between a pair of labels is the length of the shortest path in the class hierarchy tree divided by 2

    Calculation of Safe mIoU

    The calculation of SmIoU involves computing the individual safe ious for each class. The safe IoUs are the IoUs with a penalty for misclassification of that class weighted by the tree distance. The final SmIoU score is obtained by taking the mean of these individual safe ious.

    Let C be the set of all classes at the bottom level of the hierarchy, d(c,s) be the tree distance between class c and s, and n be the number of levels in the hierarchy, gtc the set of pixels in the ground truth with label c and preds the set of pixels in the prediction with label s . We define the following quantities:

    Now we define SmIoU as follows:

    $$\label{eq:1} I_c^{\text{safe}} = \begin{cases} \begin{aligned}[t] I_{c,c} & - \sum_{s \in C , s \neq c}\frac{d(c,s)}{n}I_{c,s}^{\text{safe}} & \text{ if } c \in C_{\text{imp}} \\ I_{c,c} & - \sum_{s \in C_{\text{imp}} }\frac{d(c,s)}{n}I_{c,s}^{\text{safe}}& \text{ else. } \end{aligned} \end{cases}$$

    $$\label{eq:2} \text{SmIoU} = \frac{\sum_{c\in C} I_c^{\text{safe}}}{|C|}$$

    Code for calculating SmIoU

  • Download the IDD-AW Dataset
  • First create separate directories for groundtruth and your predictions
  • Clone the code from IDDAW kit into your local dir
  • Install the requirements using pip install -r requirements.txt
  • Just run the command python safe_iou.py --gt_dir gt_dir --pred_dir your_preds_dir