During the qualifier phase, participating teams will develop algorithms guided by manually reconstructed data sets. At the end of the qualifier phase, the teams will submit their automated reconstructions of test data sets together with a single executable program capable of recreating the submitted reconstructions. The submissions will be compared against manual reconstructions using a well-defined metric that measures the quality of the reconstructed data set as well as the efficiency of the algorithm.
Based on this evaluation, up to five teams will be chosen to participate in the Final Round, which will be held in conjunction with a scientific conference at Janelia Farm Research Campus. Here the algorithms will be evaluated within a real world setting: scientists who have provided the data will use the tools to reconstruct additional data sets. Independent judges, including the data providers, will score the solutions fielded by each team.
Official Rules for the DIADEM Software Competition
Certification Form for Finalists
The Qualifier Phase
The Qualifier Phase of the DIADEM competition takes place remotely, that is, every participant will be expected to download the data locally, develop and test algorithms, and submit entries by the deadline. The data were chosen to be broadly representative of the open problem in modern neuroscience with regard to the scientific question, experimental preparation, and imaging technique.
The "rules of the game" are clear: algorithms have to read in image stacks of neuronal arbors and output digital reconstructions of the trees, which then can be scored against the corresponding manual gold standard. All submissions will be scored and ranked, and the top performers will be invited to compete in the tournament and participate in the scientific conference. Finalists will be expected to optimize their algorithms and parameters — as well as prepare a research report — between the time they are selected and the opening of the Final Round of the DIADEM Challenge.
The Final Round
The Final Round of the DIADEM Challenge will take place at the Janelia Farm Research Campus of the Howard Hughes Medical Institute in conjunction with an independent scientific conference that is being organized with support from the National Institutes of Health. The conference will include a small number of scientific sessions and ample time for personal and technical interactions between the finalists. Articles describing the algorithms, the data, and the contest results will be collected and submitted for publication in a special issue of the journal Neuroinformatics.
Five teams will be invited to participate in the final round of the DIADEM Challenge. Teams will rotate the data set (and thus the data owner working with them) at every session. As a consequence, each of the five teams will try all data sets, and each data owner will work with all teams. At the end, team spirit will be sealed by providing the data owners and scientific organizers with copies of the working executables that performed at the final round.
The scientific conference will provide an opportunity for presentation of research reports and identification of new challenges in the development of effective tools for neuronal imaging. Research reports will be subject to interdisciplinary peer review and should be submitted before the start of the conference. These manuscripts should describe the algorithms as well as characterize their performance on the training data. The data providers will also submit a joint mini-review describing the source of neuronal morphologies and rationale for their selection. Poster versions of all of these reports will be also displayed at the conference.
The organizers expect to prepare an editorial describing the event and will communicate the results to the scientific community through a variety of methods. All digital reconstructions will be posted online at NeuroMorpho.Org, along with the winning automated ones, for further public dissemination.The goal is to ensure that the results of the DIADEM Challenge are rapidly and extensively disseminated. This includes the scientific details of the winning algorithms, the advances and residual limits of the solutions formulated, and the practical availability and applicability of the tools for neuroscience labs.
While researchers will work independently during the qualifier and preparation phases, the final round and the independent scientific conference will gather leading stakeholders from both communities of data owners — cellular neuroanatomists and algorithm providers (software engineers) — to encourage intense and productive interaction. This joint expertise of these overlapping groups of scientists will enable a unique opportunity for both instantaneous technical troubleshooting and long-term scientific advancement. The final round will mimic real-lab dynamics, with algorithm providers and data owners working together toward the common goal of achieving the most effective and satisfactory solution.