Framsticks

Scope

The competition concerns the development of an efficient algorithm to optimize active 3D designs (i.e., simulated agents or robots). The simulation environment is Framsticks, and participants have a Python binding available to the native simulator library, so algorithms should be implemented entirely in Python.

The goal of the competition is to propose an algorithm that will discover agents whose center of gravity moves in the desired way in different environments used during optimization. The properties of the desired movement are defined by the fitness function (unknown to participants); examples of such movements are: following a specific path in 3D, swinging or jumping. The set of parameters that define each environment (such as gravity, water level, terrain, and initial agent rotation) is published, but their values will be set during the evaluation phase. Each submitted algorithm will be tested to optimize agents in 10 different settings (environments and desired movements). These settings will be the same for all participants.

Each submission must contain a short description of the algorithm and a standalone Python source code. The source code can use any freely and publicly available libraries, but participants should take care to describe the way dependencies are supposed to be installed to allow the organizers to run their algorithm. The algorithm will not have access to the Internet.

Judging

Algorithms submitted by the participants will be run in 10 different settings (world configurations and desired movements), with each run being repeated 20 times. During each run, the best fitness will be saved and returned as the outcome of the algorithm. For each setting, this fitness will be averaged over 20 runs, and then normalized taking into account the results of all submissions in this setting. The average of normalized values obtained for each algorithm will constitute the final score of the algorithm. The organizers will make every effort to obtain fair fitness values by repeating optimization runs in case of potential problems. Additionally, as a basic reference level, a baseline algorithm (a standard EA with niching) will be run using the same conditions and constraints as the submitted entries.

In summary:

• 10 settings (world configurations and fitness functions)
  • 20 repeated runs per setting, each run returns best fitness
  • These 20 best fitness values are averaged
  • The resulting average is normalized taking into account other submissions
• Average of 10 normalized values constitutes the final score of the algorithm

The submitted algorithm with the highest final score that exceeded the performance of the baseline will be considered the winner. In case the best submitted algorithm performs worse than the baseline, this algorithm will be considered the best submission, but no winner will be announced. Summarized results of all submissions and the baseline will be published in numerical and graphical form.

Algorithm evaluation

The simulation parameters that may be modified in each of the 10 world configurations are: lifespan of a creature ("starting energy" and "idle metabolism"), world size and heightfield (map), water level, gravity, and creature body constraints (minimal and maximal joint length).

The maximized fitness functions used in each of the 10 settings will evaluate the similarity of the path of the center of gravity (COG) to some desired path. This may be, for example, 1.0/RMSE of individual points in 3D, i.e., COG coordinates sampled in each simulation step, compared to the desired straight line. Another example would be only considering the vertical axis (z) of the COG trajectory and estimating how similar it is to the desired 0.5+0.1*sin(t/100.0) function by calculating 1.0/(sum of absolute differences).

The algorithm will be stopped and will return the best fitness found so far after (1) the number of function evaluation calls exceeds 100k, or (2) the running time of one hour (excluding solution evaluation time) is exceeded, or (3) the algorithm itself calls the end() function, or (4) a Python exception stops the execution. The memory limit is 2 GB. The submitted algorithm should be single-process, single-threaded. No more than one submission per person is allowed. Any genetic encoding can be used, but f0 or f1 are recommended (or any encoding that is based on them).

How to start – installation

Let's prepare the Framsticks environment and the Python evolution interface to work.

• Download the Framsticks simulator – the most recent zip from here and uncompress it in some directory. Note the directory where the frams-objects.dll/.so library is – the Python scripts will need it.
  
• Download "framspy" (Python interface to Framsticks) which is this entire directory. It is best to download it using the svn client (svn checkout) or the git client (git svn clone); the https certificate only encrypts but does not confirm the website's identity, so you may encounter warnings.
  
• Verify the cooperation of the Python code and the simulator – make sure that this works without errors: python frams-test.py <path-to-frams-objects-library>.
• Install the DEAP framework in Python.
• Copy all *.sim files from the downloaded "framspy" repository to the data/ directory in the uncompressed Framsticks distribution.
  
• Edit run-deap-examples.cmd and update the DIR_WITH_FRAMS_LIBRARY path, run this script and ensure that evolution works. Converting this script for Linux is trivial.
  

The Python evolution interface (FramsticksEvolution.py) is just an example of an optimization algorithm. Participants should submit their original search algorithms.

To test your algorithm in a setting very similar to the competition, use FramsticksLibCompetition.py.

If you are new to the Framsticks simulator, a good introduction is parts 1–5 and 7 of the tutorial.

Contact support (at) framsticks.com in case of any problems or if you require help.

Submission

The sources of the algorithm should be sent zipped along with the user/team ID, a short description of the algorithm, and the specification on how to install and run it to maciej.komosinski (at) cs.put.poznan.pl by the deadline of 19 May 2023.

Participants also have an option to describe their submitted algorithm in a 2-page poster paper. The paper will be reviewed and must be submitted by 14 April through the official linklings site.