Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Evaluator #47

Draft
wants to merge 102 commits into
base: master
Choose a base branch
from
Draft

Evaluator #47

wants to merge 102 commits into from

Commits on Jun 14, 2023

  1. timing benchmarks

    @matteo-meluzzi
    matteo-meluzzi committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    1865d0d View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. enumeration benchmarks added

    @matteo-meluzzi
    matteo-meluzzi committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    530c278 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2023

  1. compatibility with 'old' herb

    @matteo-meluzzi
    matteo-meluzzi committed Jun 17, 2023
    Configuration menu
    Copy the full SHA
    91fd389 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. enumeration tests with constraints

    judomat committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    7d092ee View commit details
    Browse the repository at this point in the history

  2. Fix benchmarks without constraints

    @RobbinBaauw
    RobbinBaauw committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    c6cb6f5 View commit details
    Browse the repository at this point in the history

  3. Improve constraints for benchmark

    @RobbinBaauw
    RobbinBaauw committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    c74af97 View commit details
    Browse the repository at this point in the history

  4. Add extra constraint benchmarks

    @RobbinBaauw
    RobbinBaauw committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    66c9a10 View commit details
    Browse the repository at this point in the history

  5. Rename benchmarks to proper name

    @RobbinBaauw
    RobbinBaauw committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    70a6534 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2023

  1. print results in latex

    @matteo-meluzzi
    matteo-meluzzi committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    63934b0 View commit details
    Browse the repository at this point in the history

  2. added a grammar with less terminals

    @matteo-meluzzi
    matteo-meluzzi committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    7cc923a View commit details
    Browse the repository at this point in the history

  3. grammars with constraints have correct ioexamples

    @matteo-meluzzi
    matteo-meluzzi committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    7ffb1ac View commit details
    Browse the repository at this point in the history

  4. generate benchmarks had typo

    @matteo-meluzzi
    matteo-meluzzi committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    603c506 View commit details
    Browse the repository at this point in the history

  5. added propagation count to result tables

    @matteo-meluzzi
    matteo-meluzzi committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    481d30c View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. Prototype Benchmark dataset

    @THinnerichs
    THinnerichs committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    360c9bb View commit details
    Browse the repository at this point in the history

  2. Update prototype string benchmark

    @THinnerichs
    THinnerichs committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    aae71c5 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. Move data to src

    @THinnerichs
    THinnerichs committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    deda1a2 View commit details
    Browse the repository at this point in the history

  2. remove HerbInterpret from dependencies

    @THinnerichs
    THinnerichs committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    316d846 View commit details
    Browse the repository at this point in the history

  3. Add automatic include of sub-modules

    @THinnerichs
    THinnerichs committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    c86af45 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. Add attempt at String transformation grammar

    @ReubenJ
    ReubenJ committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    1956c86 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. Add automatic include of sub-modules

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    565a0a4 View commit details
    Browse the repository at this point in the history

  2. Reformat Strings transformations

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    be37f30 View commit details
    Browse the repository at this point in the history

  3. Reformat Robots

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    93aecf2 View commit details
    Browse the repository at this point in the history

  4. update utilities and dependencies

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    1440a53 View commit details
    Browse the repository at this point in the history

  5. Reformat grammars

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    60d7ef1 View commit details
    Browse the repository at this point in the history

  6. Add bitoperations SyGuS benchmark

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    2898fe3 View commit details
    Browse the repository at this point in the history

  7. Update io functionality

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    f50307a View commit details
    Browse the repository at this point in the history

  8. Update fileparsing

    @THinnerichs
    THinnerichs committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    abf7774 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2023

  1. Add REadmes and updated IO handling

    @THinnerichs
    THinnerichs committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    36dafde View commit details
    Browse the repository at this point in the history

  2. Added documentation for SyGus's main module

    @THinnerichs
    THinnerichs committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    d88f113 View commit details
    Browse the repository at this point in the history

  3. Add ARC benchmark data, grammar and interpret still missing

    @THinnerichs
    THinnerichs committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    a785a14 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. add ARC grammar and functions

    @THinnerichs
    THinnerichs committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    1ee9140 View commit details
    Browse the repository at this point in the history

  2. Include primitives

    @THinnerichs
    THinnerichs committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    3c4b1c7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'ARC_2019' into dev

    @THinnerichs
    THinnerichs committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    1bc73da View commit details
    Browse the repository at this point in the history

  4. Update script for parsing ARC data

    @THinnerichs
    THinnerichs committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    d52a4b4 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. Add working grammar for String transformations

    @THinnerichs
    THinnerichs committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    6f1c3c5 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. Commit merge

    @THinnerichs
    THinnerichs committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    99d74e6 View commit details
    Browse the repository at this point in the history

  2. Finalize Robots dataset

    @THinnerichs
    THinnerichs committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    97a9b4c View commit details
    Browse the repository at this point in the history

  3. Fixed String benchmark

    @THinnerichs
    THinnerichs committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    12ee3a1 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. Fix Robots benchmark

    @THinnerichs
    THinnerichs committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    b079500 View commit details
    Browse the repository at this point in the history

  2. Upload Pixels dataset

    @THinnerichs
    THinnerichs committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    028a737 View commit details
    Browse the repository at this point in the history

  3. remove prolog files from Brute

    @THinnerichs
    THinnerichs committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    dfd74b9 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. Add all examples to working examples

    @THinnerichs
    THinnerichs committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    c0c6a80 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2023

  1. Fixed @make_public warnings, fixed ARC benchmark

    @THinnerichs
    THinnerichs committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    eaf791d View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. Fix ARC benchmark typo

    @THinnerichs
    THinnerichs committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    c8435f5 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2023

  1. Clean up main dir, add convenience functions

    @THinnerichs
    THinnerichs committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    c260835 View commit details
    Browse the repository at this point in the history

  2. Add all_problems functionality and mapping problems to their grammars

    @THinnerichs
    THinnerichs committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    ad095ec View commit details
    Browse the repository at this point in the history

  3. Remove all_problems from each module, and substitute it with more gen… 

    …eral functions
    @THinnerichs
    THinnerichs committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    af8130c View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2023

  1. Make state in Robots mutable

    @THinnerichs
    THinnerichs committed Dec 19, 2023
    Configuration menu
    Copy the full SHA
    30fe68c View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2024

  1. Fix robot grammar to include multi-operation programs

    @Whebon
    Whebon committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    b6d0e68 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' of github.com:Herb-AI/HerbBenchmarks.jl into dev

    @Whebon
    Whebon committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    488d4a7 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. Make all values actual values

    @THinnerichs
    THinnerichs committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    6c2ca4d View commit details
    Browse the repository at this point in the history

  2. Fix bit function arity and unknown functions

    @THinnerichs
    THinnerichs committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    8376e09 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Merge pull request #15 from Herb-AI/fix_sygus_symbols 

    Fix SyGuS benchmarks
    @THinnerichs
    THinnerichs authored Feb 26, 2024
    Configuration menu
    Copy the full SHA
    14f4b5b View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. HerbData.jl -> HerbSpecification.jl

    @ReubenJ
    ReubenJ committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    7206f8c View commit details
    Browse the repository at this point in the history

  2. Grammar -> AbstractGrammar according to HerbCore 0.2

    @THinnerichs @ReubenJ
    THinnerichs authored and ReubenJ committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    08d6c1b View commit details
    Browse the repository at this point in the history

  3. Update tests to use HerbSpecification.jl

    @ReubenJ
    ReubenJ committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    62df519 View commit details
    Browse the repository at this point in the history

  4. Migrate individual benchmarks to HerbSpecification.jl

    @ReubenJ
    ReubenJ committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d90a88b View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Add re-add formatting of 1 IOExample per line

    @THinnerichs
    THinnerichs committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    c2b3060 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #22 from Herb-AI/migrate-to-herbspec 

    Migrate to `HerbSpecification.jl`
    @THinnerichs
    THinnerichs authored Mar 7, 2024
    Configuration menu
    Copy the full SHA
    93f6d26 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Update HerbCore to v0.3.0

    @ReubenJ
    ReubenJ committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    3c08d62 View commit details
    Browse the repository at this point in the history

  2. Move legacy files 

    I'll keep them there until all their functionality is the new structure
    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    9f2ea49 View commit details
    Browse the repository at this point in the history

  3. Extract and extent problem fetching 

    I extracted the functionality that extracts problems from modules. I also added functionality to fetch a single problem-grammar-pair
    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    dd47ef0 View commit details
    Browse the repository at this point in the history

  4. Move benchmark generator

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    f60f174 View commit details
    Browse the repository at this point in the history

  5. Rename utils to export module 

    The rest of utils is extracted, only a method for exporting a module exists
    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    77ca727 View commit details
    Browse the repository at this point in the history

  6. Added unit tests for problem fetcher

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    c1d1c47 View commit details
    Browse the repository at this point in the history

  7. Add Benchmark struct

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    22d6bba View commit details
    Browse the repository at this point in the history

  8. Move file to utils folder

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    39be995 View commit details
    Browse the repository at this point in the history

  9. Move module exporter to src folder

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    1ff3fd5 View commit details
    Browse the repository at this point in the history

  10. Changed benchmark name type from String to Module

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    c52d95c View commit details
    Browse the repository at this point in the history

  11. Included benchmark type in fetching and added types

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    1c4f28b View commit details
    Browse the repository at this point in the history

  12. Corrected module exports

    @stefstef00
    stefstef00 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    27e9281 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. Fix SyGuS benchmarks

    @THinnerichs
    THinnerichs committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    8a8d6bf View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Merge pull request #25 from Herb-AI/fix_sygus_grammars 

    Fix SyGuS benchmarks
    @ReubenJ
    ReubenJ authored Apr 26, 2024
    Configuration menu
    Copy the full SHA
    5b4462b View commit details
    Browse the repository at this point in the history

  2. Fix SyGuS string primitive; Fix @all_problem_pairs macro

    @THinnerichs
    THinnerichs committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    3729b0c View commit details
    Browse the repository at this point in the history

  3. Merge pull request #26 from Herb-AI/fix_string_function_and_problem_m… 

    …acro
    @ReubenJ
    ReubenJ authored Apr 26, 2024
    Configuration menu
    Copy the full SHA
    77a3b4a View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. Fix _arg_x indices

    @THinnerichs
    THinnerichs committed May 2, 2024
    Configuration menu
    Copy the full SHA
    224b8ae View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. Moved benchmark datatype

    @stefstef00
    stefstef00 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    ab7c99b View commit details
    Browse the repository at this point in the history

  2. Added problem grammar pair datatype

    @stefstef00
    stefstef00 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    2a15fc3 View commit details
    Browse the repository at this point in the history

  3. Added program grammar pair datatype

    @stefstef00
    stefstef00 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    9c9769f View commit details
    Browse the repository at this point in the history

  4. Moved benchmark runner

    @stefstef00
    stefstef00 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    d08b861 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Update HerbGrammar -> ^0.3.0

    @ReubenJ
    ReubenJ authored May 14, 2024
    Configuration menu
    Copy the full SHA
    c20ec30 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Fix SyGuS input arguments issue

    @THinnerichs
    THinnerichs committed May 24, 2024
    Configuration menu
    Copy the full SHA
    50be6f2 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Merge branch 'dev' into restructure

    @stefstef00
    stefstef00 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    3db288e View commit details
    Browse the repository at this point in the history

  2. Added method to fetch all benchmarks

    @stefstef00
    stefstef00 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    d1aed0d View commit details
    Browse the repository at this point in the history

  3. Moved legacy files to other folder

    @stefstef00
    stefstef00 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    01d461b View commit details
    Browse the repository at this point in the history

  4. Removed legacy files

    @stefstef00
    stefstef00 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    cd54863 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Added a debug benchmark 

    Containing some simple problems for easy and quick debugging
    @stefstef00
    stefstef00 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    d3fe79e View commit details
    Browse the repository at this point in the history

  2. Added docs

    @stefstef00
    stefstef00 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    86cff51 View commit details
    Browse the repository at this point in the history

  3. Added evaluation result structures 

    There are three types of results:
1. ProblemResult: for a single problem evaluation.
2. BenchmarkResult: for a single benchmark evaluation.
3. EvaluationResult: for multiple benchmark evaluations.

The ProblemResult contains metrics about the search (e.g. execution time, memory usage, ...). The BenchmarkResult contains aggregated statistics (e.g. average execution time, termination cause totals).
    @stefstef00
    stefstef00 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    fc78e61 View commit details
    Browse the repository at this point in the history

  4. Added docs

    @stefstef00
    stefstef00 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    71930c0 View commit details
    Browse the repository at this point in the history

  5. Synth function for benchmarks 

    Created a custom synth function keeps track of more metrics (enumeration count, termination cause, memory usage, etc.)
    @stefstef00
    stefstef00 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    4e33c7e View commit details
    Browse the repository at this point in the history

  6. Add evaluation functions 

    This contains two evaluation functions one for a single benchmark and one for multiple. It calls the synth function on each problem within the benchmark(s) and returns the results including metrics and statistics in the corresponding structures.
    @stefstef00
    stefstef00 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    b3c49ff View commit details
    Browse the repository at this point in the history

  7. Include everything in the module

    @stefstef00
    stefstef00 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    42530a9 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Added support for selecting specific problem ids

    @stefstef00
    stefstef00 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    badb074 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. Added an empty constructor for BenchmarkResult to which results can b… 

    …e appended
    @stefstef00
    stefstef00 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    ac56b84 View commit details
    Browse the repository at this point in the history

  2. Added support for storing evaluation results

    @stefstef00
    stefstef00 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    eda913b View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. Rename "evaluate" to "evaluate_iterator"

    @stefstef00
    stefstef00 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    3fcc4da View commit details
    Browse the repository at this point in the history

  2. Added support for regex problem filtering

    @stefstef00
    stefstef00 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    2577ac5 View commit details
    Browse the repository at this point in the history

  3. ProblemResult metrics now include SolverStatistics metrics

    @stefstef00
    stefstef00 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    7412653 View commit details
    Browse the repository at this point in the history

  4. Added a decomposed synth for easy overloading 

    This decomposed synth calls a lot of submethods that each can be overloaded. When a benchmark needs a custom synth function, it can just overload the methods that are different and prevent code duplication.
    @stefstef00
    stefstef00 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    dcaa0bf View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. A tour of the benchmark evaluator

    @stefstef00
    stefstef00 committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    a87325f View commit details
    Browse the repository at this point in the history

  2. Rename

    @stefstef00
    stefstef00 committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    e3c3206 View commit details
    Browse the repository at this point in the history