Enhance-Your-Research Alliance (EYRA) benchmark platform

The Netherlands eScience Center and SURF formed an alliance with the mission to enhance research by providing an online platform to benchmark the performance of research software. The platform is available for researchers from all scientific disciplines, from physics and astronomy to humanities, social sciences and life sciences.

This project is led by Adriƫnne Mendrik (Netherlands eScience Center) and Annette Langedijk (SURF). Contact us if you would like to collaborate or set-up a benchmark: info@eyrabenchmark.net.

  

Why Benchmarking?

One of the currently most well-known benchmarks for algorithm performance is ImageNet. In 2012, the ImageNet challenge [1,2] boosted the field of Artificial Intelligence by showing a drastic improvement in performance after the submission of a deep convolutional neural network (CNN) called AlexNet [3]. In various scientific disciplines there is a growing interest to benchmark algorithm performance on research data using the same data, metrics and ground truth to compare their performance for a specific task. Translating research questions into a benchmark design has the potential to lead to scientific breakthroughs. The point of benchmarking is to test the current status of algorithms, identify opportunities for improvement, and learn from eachother in order to enhance research.

[1] https://en.wikipedia.org/wiki/ImageNet
[2] "From not working to neural networking". The Economist. 25 June 2016. Retrieved 19 April 2019.
[3] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems, 2012.

  

Why EYRA?

We aim to offer a platform to easily set-up and participate in benchmarks for research. We aim to provide:

  • Instructions on how to set-up a benchmark
  • Algorithm submission in docker containers, such that algorithms can be run in the cloud on the testing data.
  • Automatically generated leaderboard and analysis beyond the leaderboard to identify strengths and weaknesses.
  • FAIR (Findable, Accessible, Interoperable, and Reusable) benchmarks, data, algorithms, and evaluation metrics.
  • SURF infrastructure.
  • Connect people from different scientific disciplines and promote cross-fertilization.

We are not there yet. The platform is still actively under development. Click the Subscribe button at the bottom of the page to stay up to date.


Source code

EYRA repository: https://github.com/EYRA-Benchmark

The EYRA benchmark platform is an open source platform based on the open source COMIC platform, which is associated to grand-challenge.org.

 

Acknowledgements

We would like to acknowledge the people below for their contribution to the EYRA benchmark platform.

Netherlands eScience Center

Tom Klaver, Pushpanjali Pawar, Maarten van Meersbergen, Roel Zinkstok, Carlos Martinez-Ortiz, Alessio Sclocco, Rena Bakhshi, Romulo Goncalves, Evelien Schat.

SURF

Maurice Bouwhuis, Mary Hester, Jan Bot, Haukur Pall Jonsson, Giuseppe Gianquitto, Ymke van den Berg, Martin Brandt.

COMIC Platform

James Meakin (Radboud UMC), Bram van Ginneken (Radboud UMC).

Platform Requirements

Maria Eskevich (CLARIN ERIC), Melvin Wevers (Digital Humanities Lab, KNAW Humanities Cluster), Mike Lees (UvA), Marius Staring (LUMC), Kasper Marstal (Erasmus MC), Joeri van Leeuwen (ASTRON), and Liam Connor (UvA).

Design Logo

Thessa Kockelkorn (Double Standard Design)