On the Difficulty of Benchmarking Inductive Program Synthesis Methods

Created by W.Langdon from gp-bibliography.bib Revision:1.4524

  author =       "Edward Pantridge and Thomas Helmuth and 
                 Nicholas Freitag McPhee and Lee Spector",
  title =        "On the Difficulty of Benchmarking Inductive Program
                 Synthesis Methods",
  booktitle =    "Proceedings of the Genetic and Evolutionary
                 Computation Conference Companion",
  series =       "GECCO '17",
  year =         "2017",
  isbn13 =       "978-1-4503-4939-0",
  address =      "Berlin, Germany",
  pages =        "1589--1596",
  size =         "8 pages",
  URL =          "http://doi.acm.org/10.1145/3067695.3082533",
  DOI =          "doi:10.1145/3067695.3082533",
  acmid =        "3082533",
  publisher =    "ACM",
  publisher_address = "New York, NY, USA",
  keywords =     "genetic algorithms, genetic programming, benchmarking,
                 inductive program synthesis, machine learning",
  month =        "15-19 " # jul,
  abstract =     "A variety of inductive program synthesis (IPS)
                 techniques have recently been developed, emerging from
                 different areas of computer science. However, these
                 techniques have not been adequately compared on general
                 program synthesis problems. In this paper we compare
                 several methods on problems requiring solution programs
                 to handle various data types, control structures, and
                 numbers of outputs. The problem set also spans levels
                 of abstraction; some would ordinarily be approached
                 using machine code or assembly language, while others
                 would ordinarily be approached using high-level
                 languages. The presented comparisons are focused on the
                 possibility of success; that is, on whether the system
                 can produce a program that passes all tests, for all
                 training and unseen testing inputs. The compared
                 systems are Flash Fill, MagicHaskeller, TerpreT, and
                 two forms of genetic programming. The two genetic
                 programming methods chosen were PushGP and Grammar
                 Guided Genetic Programming. The results suggest that
                 PushGP and, to an extent, TerpreT and Grammar Guided
                 Genetic Programming are more capable of finding
                 solutions than the others, albeit at a higher
                 computational cost. A more salient observation is the
                 difficulty of comparing these methods due to
                 drastically different intended applications, despite
                 the common goal of program synthesis.",
  notes =        "Also known as
                 \cite{Pantridge:2017:DBI:3067695.3082533} GECCO-2017 A
                 Recombination of the 26th International Conference on
                 Genetic Algorithms (ICGA-2017) and the 22nd Annual
                 Genetic Programming Conference (GP-2017)",

Genetic Programming entries for Edward R Pantridge Thomas Helmuth Nicholas Freitag McPhee Lee Spector