[FrontPage] [TitleIndex] [WordIndex] 

Documentation

Presentations

Detailed competition results

The volumen of data gathered during the competition is huge (about 35 Gigabytes of space) and therefore no tarball is provided here as in other editions of the IPC. Instead, the data is summarized as follows:

The pdf files represent various scores of all planners in all domains for all tracks in the Deterministic Part. All pdf files end with a summarizing page that shows the overall ranking where planners have been sorted in decreasing order of score. There are up to four different pdf files per track, each one showing scores according to different metrics (one being the official one and another three alternative measures provided for the sake of completeness):

  1. Quality: this is the official score used at the Seventh International Planning Competition. In this case each planner is scored with the ratio Q*/Q where Q* is the best quality found by all planners for a particular problem and Q is the quality of the best plan found by the particular planner being measured for the same problem

  2. Time: much the same like the previous one but in this case each planner is scored the ratio S*/S where S* is the time of the fastest planner for a particular problem and S is the time of the planner being measured for the same problem.

  3. Solutions: in this case each planner who solved a problem is scored 1 and 0 otherwise. Therefore, the overall score of a planner in a given domain is the number of plans (validated by VAL) that that planner solved

  4. Quality-Time: the last criteria computes a tuple (Q, T) per planner and problem in each domain: Q stands for the best quality found by each planner and T stands for the time it took to find it (if various solutions were found, the best one is picked up). Next, each planner is scored 1.0 for every planner whose tuple dominates it (i.e., that Q'>Q and T'>T with (Q',T') being the tuple of the other planner) for the same problem

Below each pdf file there are a colour table which shows the same information with the following conventions: red stands for invalid solutions (according to VAL); yellow stands for no solution found; finally, the gray scale stands for the score found: the darker the higher.

On the other hand, the excel worksheets provide alternative data on various issues:

  1. Problems solved: lists what problems were claimed to be solved by each planner and whether they were valid according to VAL, one by one.

  2. Total solved: summarizes the behaviour of each planner in every domain showing the number of problems it was given; the number of problems it claimed to have solved; the number of valid solutions (according to VAL); the number of failures or, alternatively, the number of times the planner exited (or was killed) before finding a solution and, finally, an attempt to diagnose the failure. First, the number of failures on time are shown; next, the number of failures on memory and finally, the number of failures for unknown reasons (e.g., segmentation fault)

  3. Diagnosis: in order to improve the analysis of data, an additional table has been created which lists, per planner and domain, the particular problems were fails occured either on time, memory or for unknown reasons.

  4. Quality-Time: it shows the time elapsed for finding every single solution computed by each planner in each problem and the time it took for it to find it. If more than one solution was found, all of them are listed.

Bear in mind that the attempt made to diagnose errors in the second and third excel worksheets is purely heuristic and hence prone to errors.

Regarding the second tarball, the snapshots given are readable by the scripts report.py and score.py which are (hopefully!) useful for inspecting all the information gathered.

Finally, all the domains and problems used in the competition along with the planners that entered the competition in all tracks are available in the following svn server:

Public access is granted to everyone. Besides, it also contains the same tarballs detailed at the beginning of this section in the directories docs/ and results/ At last, but not least, the directory scripts/ contains all the software used for running the Seventh International Planning Competition. Dumps of this svn repository are available for private experimentations upon request. For further enquiries contact Carlos Linares López <carlos DOT linares AT uc3m DOT es>

A detailed view of all the results has been placed in a different svn server which is publicly available as well:

As mentioned earlier, the data contained herein takes about 35 Gigabytes of space (without taking into account the .svn folders that result after a checkout). Thus, it is not recommended to checkout its contents but to browse it with your favourite svn GUI. The information contained in this script has been arranged as follows:

2013-10-04 16:47