This page summarizes the rules for the deterministic part of IPC-2011. It is meant to be the central place where participants can find everything they need to know to build a good entry for the competition. In particular, you will find information or links about:
- the different tracks/subtracks of the competition
- the modeling language used to represent competition domains and problems
- the domains and problems used in the competition
- evaluation:
- how the results are collected (including timeout and memory limits)
- how the planners are scored and ranked
- miscellaneous issues:
- availability of source code
- hardware platform
- external memory search
Other useful information such as deadlines, updates, and supporting documents/information can be found by following links from the competition homepage. If you still do not find the answer to your question, please use the mailing list to discuss the issue. Issues discussed on the mailing list which are of general importance will be reflected in the contents of this page.
Tracks
The IPC-2011 deterministic part is divided into three main tracks: sequential, temporal planning and preferences. Each track in turn is divided into two subtracks: satisficing and optimal planning. Besides, a new subtrack has been arranged for this IPC for the first time: a multi-core for the sequential satisficing subtrack. Awards will be given to planners in each subtrack and not aggregated across several subtracks, so each subtrack essentially defines a competition on its own.
The seven subtracks differ in which PDDL features are required of the participating planners, which objective function planners are expected to optimize, and whether suboptimal solutions are considered acceptable.
In each track, there are certain "core features" (e.g. durative actions in the temporal track) that are prerequisites for entering the track, and there are also optional features (e.g. ADL, derived predicates) that may appear in some of the domains. For planners that do not support the optional features, we will offer equivalent domain versions where these features have been compiled away. So it is enough to support the core features to fully participate in a given track. Note, however, that compiled problem formulations are often significantly larger than the original problem formulations, so it may be advantageous to support the optional features directly.
The following links contain a detailed description of each track:
Sequential: satisficing | optimization | multi-core. The objective of this track is to minimize total action cost of a plan.
Temporal: satisficing | optimization. The objective of this track is to minimize plan makespan.
Preferences: satisficing | optimization. The objective of this track is to find a plan with minimum penalization (a combination of total action cost and penalization for not achieving some goals).
Competition Rules
The deterministic part currently considers three tracks: sequential, temporal and preferences. In all cases, two different subtracks shall be arranged: optimal and satisficing; an additional multi-core subtrack shall be arranged for the sequential part as well.
Potential participants in the preferences track are requested to send an email to agolaya@inf.uc3m.es with the registration form completed. For taking part in the sequential and/or temporal track, please send the registration form by e-mail to carlos.linares@uc3m.es
The evaluation process will comprise two phases: Initially, the competitors will be given a set of representative domain/problem instances to test their planners on their own machines. Then, they will submit a final version of the source code of their planners that will be run by the organizers on the actual competition domains/problems (unknown to the competitors till this time). This way no fine-tuning of the planners will be possible.
PDDL in this year's competition
IPC-2011 will be based on PDDL 3.1, i.e. the same language used in IPC-2008 with no extensions at all. A competing team may participate for any track. However, the competitors are strongly encouraged to participate in every track supported by their planners.
See the PDDL resources for further information
IPC-2011 Domains/Problems
Like previous competitions, we plan to introduce new domains and also re-use several domains from previous competitions. We are actively seeking interesting and challenging new domains for this competition. Therefore, if you know any problem that can potentially be a good competition domain, please send the suggestion by email to one of the organizers (Ángel García Olaya <agolaya AT inf DOT uc3m DOT es>, Carlos Linares López <carlos DOT linares AT uc3m DOT es>). We prefer domains that are inspired by real-world applications, and also can be naturally represented (or closely approximated) by the PDDL fragment of any of the three main tracks of this competition.
Depending on the properties of each domain, a given domain may not be used in all three tracks. The problem set, which is likely to be randomly generated for each domain, can also be different between the two subtracks (satisficing and optimization). In particular, given that it is much harder to find proven optimal solutions than finding a satisficing solution, the problems used in the optimization subtracks are likely to be less complex than the ones used in the satisficing subtracks.
Evaluation schema
As in the IPC 2008, this competition is going to be arranged as a blind evaluation. Thus, instead of having the competitors run their planners on problems provided (incrementally) by the organizers, we as the organizers will run all planners according to our schedule and only report the final results when the competition ends. The actual problem sets used in the competition will not be known to the competitors and will only be made public at the end of the IPC.
We will provide example problems for all tracks so that the participants can test their planners before submitting the final executable. We hope that the examples will cover all possible constraints that appear in the actual testing problems. However, if there is a problem running a participating planner on a given testing problem, we will try our best to work with the participants to figure out the issue. Please check the IPC-2011 homepage for information on when the example problems for all tracks will be available and when we will start running the submitted planners on testing problems.
Evaluation Criteria: We do preserve the same evaluation criteria used in the last IPC. Thus, we emphasize good plan quality and put less emphasis on solving time. If two plans are found within the time and memory limits, then a plan with better quality, as defined by the objective function of each track, will have a better score. Note that this is only applicable for the satisficing subtracks because plans in the optimization track will have equal quality. In the optimization subtracks, only the number of solutions found is relevant for the final planner score. The details on the evaluation criteria are summarized below.
The following rules apply for optimization tracks:
- Each planner is given 30 minutes to solve each problem. The time needed to solve each problem is not taken into account; what matters is only whether the problem was solved or not within 30 mins.
- The quality of the resulting plans is an on/off criterion, since plans have to be optimal. The organizers will try to obtain optimal solutions for all problems, mainly through the development of specialized domain-specific solvers. If a planner produces at least one non-optimal plan in a given domain, all results of that planner in that domain are ignored, i.e., it is assigned a score of 0 for this domain. However, the organizers will try to give the participants a chance to repair the bug that caused the problem and rerun the planner on this domain, in so far as this is possible within the time constraints and the blind evaluation mode of the competition.
- So, the only thing that matters is the number of solved problems. All problems (either hard or easier) are considered equivalent. The winner of each category (sequential, temporal, net benefit) is simply the planner that solves most of the problems.
The following rules apply for satisficing tracks (including multi-core sequential satisficing):
- Each planner is given 30 minutes to solve each problem. The time needed to solve each problem is not taken into account; what matters is only whether the problem was solved or not within 30 mins.
The quality of the resulting plans is important! Suppose a problem with an optimal solution having quality Q* and a planner finding a plan with quality Q<Q* (assume higher qualities are considered best). This planner would aggregate the quality ratio Q/Q* to its total score (use Q*/Q in case lower qualities are considered best). The organizers will try to find optimal solutions for all problems, through the development of specialized domain-specific solvers. In cases where this is not feasible, Q* will be considered the best plan quality found by any competing planner or by the organizers.
- The planner with the highest sum of quality ratios is declared the winner in each category.
Some additional notes:
- Problem instances for the satisficing tracks will be drawn from the same domains as for the optimization tracks. However, it is expected that problem instances for the satisficing tracks will be larger than those for the optimization tracks.
Supporting Tools: We expect to be able to provide a limited set of supporting tools such as updated planning validation software for PDDL3.1 (VAL) from the Strathclyde Planning Group, PDDL 3.1 syntax checkers and standalone parsers (in a limited set of programming languages).
Miscellaneous Issues
Submission of source code: We require all competitors to make public, through the IPC-2011 website, the source code of the version competing in the competition. This will encourage information sharing and allow independent evaluation and double checking of the competition results by the planning community.
Hardware Platform: The competition will be run in a cluster of 11 nodes. Each node is an INTEL XEON 2.93 GHZ QUAD CORE processor (64 bits) using Linux. Up to 6 GB of RAM memory and 750 GB of hard disk will be available for each planner. Each planner will be run in a single node and no planner will be allowed to run in more than one simultaneously (although in the multicore track it will be allowed to use all the 4 cores of a node). No GPU is available. Memory and time will be externally limited.
Using external memory: External memory search is another trend in the search community that can have a big impact on planner performance, when the amount of main memory is limited. We do not disallow planners that use external memory search in the competition. However, the I/O time needs to be counted against the total allocated time for each run (expected to be 30 minutes at the moment).