Contest Track - Structural Shape Retrieval

 

Contest Results

The following participants have successfully submitted their structural shape retrieval results to this SHREC ’09 track:

  1. •   Silvia Biasotti, Simone Marini (IMATI):

  2. ‣       ERG1, ERG2

  3. •   Petros Daras, Apostolos Axenopoulos, Athanasios Mademlis (Informatics Telematics Institute):

  4. ‣       Compact Multi View Descriptor 1

  5. ‣       Compound SID CMVD 1, Compound SID CMVD 2, Compound SID CMVD 3

  6. •   XiaoLan Li, Afzal Godil, Helin Dutagaci (NIST):

  7. ‣       BagOfWords 1, ConcentricBagOfWords 2.

  8. •   Thibault Napoleon (TelecomParisTech CNRS LTCI):

  9. ‣       MCC 1, MCC 2, MCC 3, MCC 4

  10. •   Ryutarou Ohbuchi, Takahiko Furuya, Masaki Tezuka (University of Yamanashi):

  11. ‣       BF-SIFT 1

  12. ‣       MR-SPRH-UDR 1


As described in the Evaluation part of this website we use the F-Measure as foremost important performance measure. The F-Measure is a single harmonic value based on Precision and Recall. The average precision and recall values of the ten queries are used as input for the computation of this F-Measure.  Since the values have been calculated for two Tiers this results in two F-Measures per method run.

In the image below both the 1st and 2nd Tier F-Measure values have been plotted in a graph where the top line represents the maximum value. (An absolute representation where the maximum values are stated in the top left corner for respectively 1st and 2nd Tier.)

This means that there are two different ranking lists: One for Tier 1 and one for Tier 2. To determine the final ranked list for the performance of the method runs, bot the F-Measures are rewritten as a percentage with respect to their possible maximum value. The average of both percentages then represents the complete performance of one method run.

Rank    Total %    Methodname                            F-Measure Tier 1    % Tier 1            F-Measure Tier 2    % Tier 2

1           74.67 %   MCC 3                                     0.648                       81.0 %              0.586                       68.3 %

2           73.75 %   Compound_SID_CMVD 3       0.620                       77.5 %              0.600                       70.0 %

3           72.42 %   Compound_SID_CMVD 2       0.612                       76.5 %              0.586                       68.3 %  

4           69.17 %   Compound_SID_CMVD 1       0.592                       74.0 %              0.551                       64.3 %

4           69.17 %   MCC 2                                     0.592                       74.0 %              0.551                       64.3 %

4           69.17 %   BF-SIFT 1                               0.592                       74.0 %              0.551                       64.3 %

7           68.00 %   MR-SPRH-UDR 1                   0.576                       72.0 %              0.549                       64.0 %

8           65.92 %   MCC 4                                     0.572                       71.5 %              0.517                       60.3 %

9           65.67 %   CMVD 1                                  0.552                       69.0 %              0.534                       62.3 %

10         64.58 %   MCC 1                                     0.548                       68.5 %             0.520                        60.7 %

11         57.42 %   ERG 2                                     0.492                       61.5 %              0.457                       53.3 %

12         52.33 %   ERG 1                                     0.448                       56.0 %              0.417                       48.7 %

13         25.83 %   BagOfWords 1                        0.232                       29.0 %              0.194                       22.7 %

14         22.83 %   ConcentricBagOfWords 2      0.200                        25.0 %              0.177                       20.7 %

As both Tiers have different maximum values their correlation is less clear. For this reason there is also a graph variant where a similar maximum is taken. Note that the first image would be completely filled in case of a maximum score. The length of the bars corresponds to the absolute percentage while in the image below the length of the bars represent their F-Measure values.

Precision and recall are shown at the bottom of this page for a more detailed view on the performance. The F-Measure is only high when both Precision and Recall are high. But the percentages are the same for the F-Measure, Precision and Recall since they are all generated by comparing them with their maximum. They are left out in the table at the bottom of the page.

There are also detailed overviews of the performance in every query. Or the total performance of one method run on all queries. (Both shown in the small images below) The different queries with their results can be downloaded here:


Download separate query performance: Appendix A.pdf.

Precision and Recall

Rank    Total %    Methodname                                    Precision T1      Precision T2      Recall T1       Recall T2 

1           74.67 %   MCC 3                                             0.810                 0.510                  0.540              0.683

2           73.75 %   Compound_SID_CMVD 3               0.775                 0.525                  0.517              0.700

3           72.42 %   Compound_SID_CMVD 2               0.765                 0.513                  0.510              0.683

4           69.17 %   Compound_SID_CMVD 1               0.740                 0.483                  0.493              0.643

4           69.17 %   MCC 2                                             0.740                 0.483                  0.493              0.643

4           69.17 %   BF-SIFT 1                                       0.740                  0.483                  0.493              0.643

7           68.00 %   MR-SPRH-UDR 1                           0.720                  0.480                  0.480              0.640

8           65.92 %   MCC 4                                            0.715                   0.453                  0.477              0.603

9           65.67 %   CMVD 1                                          0.690                  0.468                  0.460              0.623

10         64.58 %   MCC 1                                             0.685                  0.455                  0.457              0.607

11         57.42 %   ERG 2                                             0.615                  0.400                  0.410              0.533

12         52.33 %   ERG 1                                             0.560                  0.365                  0.373              0.487

13         25.83 %   BagOfWords 1                                0.290                  0.170                  0.193              0.227

14         22.83 %   ConcentricBagOfWords 2               0.250                  0.155                  0.167              0.207