Frequently Asked Questions

Q. What is SWC format?
A. A file in SWC format contains information representing a digitally reconstructed neuron. SWC is non-proprietary and stores the minimum amount of parameters required to represent a vector-based three-dimensional reconstruction. Files may begin with headers above the data values, each beginning with #. Parameters are organized into 7 columns, where each row within the column represents one trace point. From left to right these columns are: unique identity value for trace point, structure type, x coordinate, y coordinate, z coordinate, radius, identity value for parent (i.e. trace point that comes before and connects to the current trace point). The first 10 points of an example SWC file are provided below:

#Example header text here

1 2 4882 1797 19 9 -1

2 2 4882 1797 19 9 1

3 2 4875 1821 19 9 2

4 2 4852 1849 19 21 3

5 2 4842 1827 18 12 4

6 2 4835 1816 18 7 5

7 2 4827 1807 18 7 6

8 2 4814 1797 18 4 3

9 2 4803 1785 18 4 8

10 2 4785 1763 18 4 9

The bolded row represents one trace point which has been given an identity = 3, type = 2 (i.e. axon), X = 4875, Y = 1821, Z = 19, radius = 9, and trace point 2 is its parent (i.e. the trace point represented in the row directly above it).

Columns 1,2, and 7 are always integers. Columns 3,4,5, and 6 represent whatever units were used in the reconstructions process (e.g. pixels, micometers, etc.) and can have decimal points.

Column 1 (Identity #) must always increase in value by 1 whereas the column 7 (Parent Identity #) values have no such restriction but must be less than column 1 in the same row. Note that both rows 4 and 8 in the above example connect directly to row 3, meaning that row 3 must be a bifurcation point.

Row 1 has a parent = -1, which means that this row does not have a parent and is thus the root of the reconstruction.The commonly accepted values for Column 2 that are pertinent to the DIADEM datasets are: 1 = cell body; 2 = axon; and 3 = dendrite.

 

Q. Is there a difference in metric for different data sets other than dimensionality?
A. There are a number of differences in the metric between datasets which are detailed on the individual dataset readme pages (except for the threshold differences). The thresholds, both distance and path length error thresholds, are different to account for the resolution in both XY and Z directions. The thresholds for spur (a small terminal branch) removal vary by dataset as well (some don't remover spurs). Also, the Neuromuscular projection dataset handles terminations in a different manner because of the rosette structures. The manual reconstructions end at the beginning of those structures, but the metric will not punish automated reconstructions that trace into the rosette structures.

 

Q. The DIADEM metric seems different from previous formulations circulated and discussed in academic settings by some of the organizers. Are these related?
A. The current metric is substantially different from any version disclosed ahead of the DIADEM launch. The current metric description on the DIADEM challenge website is the only reference document explaining how the metric works.

 

Q. How can we set the matching threshold?
A. The publicly released version of the metric does not allow adjustment of the thresholds other than to the predetermined values.

 

Q. In first data set, "Cerebellar Climbing Fibers", the individual planes are merges of a panel of capture stacks. Distortions in neuron shape are visible at some of the boundaries. Is it possible to either fix it, or in addition, release the data as individual stacks that need to be merged?
A. Merging was not performed by hand, but with the leading software controlling the motorized stage of the microscope. Mechanical error limits in stage movement are evident at the small scale of climbing fibers though they are not large enough to impair manual tracing. The released data set corresponds to the original acquired images and was not tiled in a post-processing step. Thus, individual panel stacks are not available. Although the results may not be optimal, they are representative of the typical experimental configuration in a modern neuroanatomy lab. Contestants are of course free to crop images in individual tiles, and re-merge them as part of their algorithmic solution. Datasets in the final tournament can be expected to be of similar aspect as those released in the qualifier phase.

 

Q. Do different datasets have drastically different thresholds?
A. Yes.

 

Q. Is the source code for the DIADEM metric (scoring function) available?
A. Yes, it can be downloaded here. Note that this overrides the answer to the FAQ 'How can we set the matching threshold?' above.

The following are the MD5sums for the DIADEM metric source code:
If downloaded on/after February 19, 2010:
  a7c9daa3564e947e22f0b707a0bb3a95
If downloaded between January 26, 2010 and February 19, 2010:
  82368ef91ede897b87559d24909e80f6
If downloaded between November 25, 2009 and January 26, 2010:
  14d147ba30e84de13d343fb349c70ca2
If downloaded before November 25, 2009:
  4b1b8cb075e53b7f45fb9c3e65c79ce1

 

Q. Are companies allowed to compete in the DIADEM Challenge?
A. The goal of the competition is to encourage development of new software, by companies as well as non-profits. If companies write new software for the competition, as we hope they will, the confidentiality and IP issues should be much more manageable than they would be for existing commercial software packages. At the same time, in our experience the currently-available commercial software does not perform as well as the research community would like. Hence we do not expect existing commercial software packages entering the competition without additional development.

As long as a finalist makes the software publicly available within the six-month period after the competition, the rights of sponsors, judges, and data providers to use the software are limited to their own research use and aren't transferable or sub-licensable. This was precisely intended to preserve the market for commercializing successful software. Similarly, the restriction on HHMI's own research use of the software, to Janelia only, was intended to preserve the market for commercializing successful software.

To accommodate potential commercial concerns, we also specified that only the executable version of finalist software had to be made available as long as the algorithms (not the code) are published. We believe it is reasonable to expect publication of the algorithms in all cases. Also to accommodate potential commercial concerns, we specified that the rules apply only to the software that is actually submitted by finalists. If a company makes enhancements or improvements to finalist software after the competition, there is no attempt to reach through to those.

We did not want to specify what terms for making the finalist software publicly available would be "reasonable", in recognition that this will depend on factors no one knows in advance, such as how effective and complete the software package is. Our intent was not to try to dictate pricing as long as it is reasonable from a market standpoint.

The whole point of this competition is to get improved software out and available for use by the research community. We would strongly prefer that the finalists do this themselves, but we felt it important to reserve the right of the sponsors to do this if necessary. Our expectation is that if a company submits winning software, it will be able to get a package on the market within the six-month period, so that the sponsors will not need to exercise the license to distribute.

 

Q. Why is the DIADEM metric so complex? Was the choice somewhat arbitrary?
A. The metric implements our best attempt to quantify the human judgement of what differentiates a good reconstruction from a bad one. Since we agreed on a "manual" gold standard, there is some inherent arbitrariness. The basic idea is simple: the nodes of the trees should be in the right position, their topological interconnectivity should be accurate, and the path distance in reasonable range. However, there are many different cases of possible "errors" or "variations", and these are judged differently depending on the impact they have on the overall structure. Moreover, the various datasets have different characteristics (representative of experimental diversity encountered in real-lab scenarios) which are reflected in additional requirements. These qualifications account for most of the metric complexity.

 

Q. The metric compares trees but several of the datasets are actually axons in passing and don't contain extensive branching structures. Should all of the axons in those datasets be traced or just one of them?
A. They should all be traced. Each dataset has the same order of magnitude of total number of bifurcations.

 

Q. What will prevent people from cheating? For example, can I just manually trace the axons and submit these results for the first round?
A. The executable and detailed explanations in an accompanying "readme" file should be submitted along with the reconstructions.

 

Q. What program can be used to open the .rar data set files? Is there a free, downloadable program for this?
A. Please carefully read the Data Set General Readme on the website. It indicates PeaZip as one example. A possible alternative is Zipgenius. These were both free last time we checked.

A google search for ".rar" will find many other hits. As usual with freeware, read carefully during installation to make sure you uncheck any add-on programs you don't want. PeaZip didn't have any last time we checked, but just in case...

 

Q. How completely automatic must submitted algorithms be? Is it acceptable to submit a program where the users enter a few parameters before executing? Is it acceptable to enter a program which the user can interact with during or after the algorithm has been run?
A. Parameter setting is allowed before algorithm execution. For example, one parameter might correspond to each of the seed (starting) points that are given for every trace. More generally, algorithm submission should be accompanied by a sufficiently detailed and clear instruction file to allow the judges to reproduce precisely the submitted digital reconstructions. In the "Tournament" phase of the DIADEM Challenge, contestants will run their algorithms "live" together with the judges, and an even greater level of interaction will be allowed. Part of judging will be based on the amount of necessary "post-processing" edits that are needed to obtain a reconstruction similar enough to the manual gold standard. Any user interaction during the execution of the algorithm will be "counted" towards these correction steps/time.

 

Q. The diadem challenge seems to require registration of image stacks, i.e., aligning the images properly to create a 3D volume. Is this something to be addressed by the automated method, or is the input to the developed algorithm the 3D data set?
A. It is not mandatory for the input of the algorithm to be the 3D data set -- a successful algorithm could reconstruct portions of the arbors from partial stacks or even individual images or tiles, and the stitch them together. The starting point of the DIADEM challenge is the collection of files provided for download. The way image stacks should be put together is detailed in the instructions for each data set. In the Qualifier Phase, image stack registration can be performed as a preprocessing step, either manually or automatically. In the Final Round any manual operation will be counted as editing and treated as a "correction" to the algorithm.

 

Q. Is the output of the algorithm supposed to be the "segmented" tree, i.e. a binary file in which the voxels representing the neuron have one value and everything else have a different value? Or is the "digital reconstruction" simply the 1-voxel thick centerline that can be extracted from this segmentation, which then allows for determining interbranch length, bifurcation and termination nodes etc.?
A. A digital reconstruction, the output of the algorithm, consists of series of interconnected vectors, not voxels. Although in principle each of these vectors is associated with a thickness, the DIADEM metric only considers the branching topology, path distance, and position of the nodes, thus diameter does not affect the computation of the score.

 

Q. Are edge-detection, image thresholding, and tree enhancing filters part of the purpose of the challenge?
A. Any methods that can help automating the production of digital reconstructions from sets of images may be relevant to the DIADEM challenge.

 

Q. In the manual reconstructions provided for the Olfactory Projection training data set, some of the branch points appear slightly misaligned with the underlying labeled structure. Will this affect the scoring?
A. An example of a branch point that appears slightly misaligned with the underlying structure (from the OP_2 Training Round data set) is shown in the figure below (red arrow).

branch point example

Manual reconstructions have been tested to see if these points affect scoring. Specifically, a correctly re-aligned reconstruction was compared to the original file included in one of the data sets. None of the non-terminating nodes were missed (see next FAQ for further observations on terminating nodes). We have therefore left the reconstructions as they were originally traced. While it is improbable that some algorithms would be scored incorrectly, judges will take a closer look at potential false negatives in this data set for contestant scores that are borderline for qualification to the DIADEM Challenge Final Round.

 

Q. In the manual reconstructions provided for the Olfactory Projection training data set, some of the termination points appear to vary in terms of distance from the underlying labeled structures. Will this affect the scoring?
A. An example of two terminations points that end at varying distances compared to the underlying structures (from the OP_2 Training Round data set) is shown in the figure below (red arrows).

termination point example

Manual reconstructions have been tested to see if variation in the positions of termination trace points affects scoring. Specifically, a correctly re-aligned reconstruction was compared to the original file included in one of the data sets. Two terminal nodes were missed, resulting in a final score of 0.989. This score is nearly perfect and well within the typical range observed between two manual reconstructions by independent experts from the same underlying image stack. Because such minor differences are unlikely to affect algorithm rankings, we left the traces as they were originally traced. However, judges will take a closer look at potential false negatives in this data set for contestant scores that are borderline for qualification to the DIADEM Challenge Final Round.

 

Q. Does the metric account for possible floating point error in determining whether a node is within threshold distance in the Z-direction?
A. Based on extensive testing, we believe that the metric should not produce floating point errors. However, this possibility cannot be excluded a priori. Therefore, an updated version of the metric is available for download (in substitution of the previous program) from the same original link on this site, as of the posting date of this FAQ. This new version of the metric provides a small additional margin to the Z component of the distance threshold in order to ensure that no floating point error can affect scoring. Although we invite all contestants to use their judgment, we do not recommend switching to the new metric in light of the lack of any impact on scoring found during testing of the metric. If an incorrect score due to floating point error is suspected, please report the incident with relevant details.

 

Q. What are the "MD5sums" for each DIADEM dataset file (both in part 1 and part 2)?
A. The MD5sum is a hash function that changes with *any* alteration in a file, even very small changes that do not affect the file size. It's a useful way to determine whether the file you are working with is the same as what you are supposed to, for instance to ensure it was downloaded without errors or that it is the most recent version.

The DIADEM datasets consist of a total of 15 files. Their MD5sums are the following:

Neocortical Layer 6 Axons v2.rar 0438722365ab6b624b39aa1bf05540a1
Olfactory Projection Fibers v2.rar 2d6efa7fd48c17e492879edb771a5f70
Cerebellar Climbing Fibers.rar a6d4c274eb909824c858a67bab249315
Neuromuscular Projection Fibers Part 1.rar 919fd73223b1f964d938fc75bc6d105f
Neuromuscular Projection Fibers Part 2.rar 8d469b2befeb5f1c805e1907a6e2213f
Neuromuscular Projection Fibers Part 3.rar 57c73828a91a95c047227b3f8e31d0fd
Neuromuscular Projection Fibers Part 4.rar 7746f091cedff6d6de0c957a2d9efc5e
Neuromuscular Projection Fibers Part 5.rar 1d450bebc1bb5bdb16cbba6a10725839
Neuromuscular Projection Fibers Part 6.rar d9c3ee65643c92672c2ecf6c786b4dba
Neuromuscular Projection Fibers Part 7.rar dce5055d3d0ce9ec4b08a5561b3d1d9f
Hippocampal CA3 Interneuron Part 1.rar 95082d78f6930997490a07d1fd985331
Hippocampal CA3 Interneuron Part 2.rar 2c2b110985e5f73bbda115ad15a765a1
Hippocampal CA3 Interneuron Part 3.rar 9a773f9edb5c9b43ffb8fdcb56baa0df
Hippocampal CA3 Interneuron Part 4.rar 4a5db38fd174273a2d9b385ff020d762
Hippocampal CA3 Interneuron Part 5.rar 6b420fcaf46b177c5ddb8415e4dcb738

In addition, the following are the MD5sums for the zip and tar files of the DIADEM metric:

If downloaded on/after February 19, 2010:
  DiademMetric.zip df54d392f8219b931917bc2c4c27506f
  DiademMetric.tar b07078a5fb7603f7779ff809dacd9c3c

If downloaded between January 26, 2010 and February 19, 2010:
  DiademMetric.zip 7f2f375fc832697f931f8a60681ad660
  DiademMetric.tar ea612584fd0f7281954f3057edcf9f5c

If downloaded between November 25, 2009 and January 26, 2010:
  DiademMetric.zip 135750948fc323e047f96c60688f98e8
  DiademMetric.tar bba7afaeb0121ed8a2fb620e0dc980cf

If downloaded between October 22 and November 25, 2009:
  DiademMetric.zip 3d36bc53704a015f0e82d60376ceb841
  DiademMetric.tar 3fae066ffa47b9e34b3408a2b577e7ad

If downloaded between September 2 and October 22, 2009:
  DiademMetric.zip 6ac3257db619ac75de97d0c8ade534a4
  DiademMetric.tar abf49193a5dc54c2d5413b3fec737911

If downloaded prior to September 2, 2009:
  DiademMetric.zip 3b65d5e8c5a0f9ec9c16e2eb1d51b7d8
  DiademMetric.tar 39a4f672db5bd739b8e734679e803a92

 

Q. In Neocortical Layer 6 Axons, there appear to be some inconsistencies in at least one of the manual reconstruction files: what are the correct coordinates of the origin in NC_10.swc? Are there other files that may be inconsistent?
A. We found and corrected one inconsistency in each of two manual reconstruction files in the Neocortical Layer 6 Axons dataset, namely files NC_07.swc and NC_10.swc. If you downloaded the file "Neocortical Layer 6 Axons v2.rar" on/after Friday, Sept 11, 2009, these inconsistencies have already been corrected and no further action is required. The right coordinates of the origin (first data line) in the manual reconstruction file NC_10.swc included in this archive are (X,Y,Z:109,609,-5). The MD5sum for the correct rar files is:

Neocortical Layer 6 Axons v2.rar 0438722365ab6b624b39aa1bf05540a1

However, if you downloaded the file "Neocortical Layer 6 Axons.rar" prior to Friday, Sept 11, 2009 (MD5sum 5ff7a24b809ede4512038fafbd5db29e), its file NC_10.swc (MD5sum d916e483537e3da158cf267aa07e0ebb) contains the incorrect origin coordinates (X,Y,Z:109,609,-19). Although the rest of the file lines up fine, this mistake on the origin could seriously affect scores. Please download the corrected NC_10.swc file.

Moreover, we found and corrected an inconsistency in the old file NC_07.swc (MD5sum b78d646318e3a87b766d148e8c557d06). Please download the corrected NC_07.swc file.

The MD5sum values for the corrected manual reconstruction files are:

NC_07.swc 5e6238f5abee8195687007630da8c81f
NC_10.swc 75cbbf4417ccc682f3cdf5f0964cc618

 

Q. In the swc file, does column 2 (type or tag of the tracing point) influence the scores, that is, should the program correctly determine whether it is an axon or a dendrite?
A. No

 

Q. The same branch between two bifurcations can be divided by intermediate points differently. Does this choice affect the score?
A. The metric is based on the location of the nodes (bifurcations and terminations), but the distance along the path does affect the computation of the score (as explained in the Rules of the competition). Therefore, the intermediate points should follow the image path as accurately as it is necessary to ensure that the branch path length is accurately reproduced.

 

Q. Why is the metric executable not working with the cerebellar files on my computer? Comparing the manual reconstruction file with itself
(java -jar ./DiademMetric.jar -G CF_1.swc -T CF_1.swc -D 1 -m true) gives the result:

java.lang.NullPointerException
    at org.krasnow.cng.diadem.DiademMetric.scoreTrees(DiademMetric.java:580)
    at org.krasnow.cng.diadem.DiademMetric.scoreReconstruction(DiademMetric.java:428)
    at org.krasnow.cng.diadem.DiademMetric.main(DiademMetric.java:1881)

It should give a perfect score of 1.00, but instead it crashes. What I am doing wrong?
A. The problem is due to a coding error which resulted in a crash when running the Climbing Fiber or Neuromuscular Projection data sets (sets 1 and 4). Specifically, the error produced a NullPointerException at line 580 of the code in the method "scoreTrees". Any successful runs using the prior version are unaffected by the error and are correct. If you experience this problem, you need to download the current version of the DIADEM metric. The MD5sum for these new correct files are:

DiademMetric.zip: 3d36bc53704a015f0e82d60376ceb841
DiademMetric.tar: 3fae066ffa47b9e34b3408a2b577e7ad

If you experience some other problem, please report the issue to diadem@janelia.hhmi.org.

 

Q. The DIADEM metric provides a dramatically incorrect score and/or ignores a large portion of my SWC file. What am I doing wrong?
A. Most likely you have an older release of the metric (version prior to 11/25/2009) and need to download the more recent version. An error in the previous release of the metric occurred if any line of data did not contain the precise formatting expected (e.g. tabs between data, any character other than a normal space at the end of a line). The line for node 4 in the climbing fiber CF_1.swc contained a tab at the end. This caused the line to be ignored and thus all descendant nodes could not be attached to the tree. Ultimately the metric would conclude without a clear error, but would likely return very poor scores for automated traces run against the gold standard CF_1.swc. The updated metric ignores any whitespace at the end of a data line, though any non-whitespace characters (or any incorrect formatting) causes the program to terminate with an error message detailing the file and line number of the improper data format. Tabs and spaces are now treated equally to provide greater flexibility, though other programs may have more demanding format constraints. As before, lines beginning with the "#" symbol are seen as comments and are ignored.

 

Q. What changed in the DIADEM metric released on 11/25/2009? Why does it score my reconstructions differently from the previous versions?
A. In addition to the SWC parsing revision discussed in the previous answer, several modifications were made in the metric to improve its accuracy. Changes include Euclidean distance Z thresholds, path length error thresholds, the method of path length error calculation, and continuation determination.

Euclidean distance Z thresholds were increased for the Cerebellar Climbing Fiber, Neocortical Layer 6, CA3 Interneuron, and Olfactory Projection data sets. The Euclidean distance XY threshold for the CA3 Interneuron data set was also increased. The increased distance thresholds required an increase in path length error thresholds in Z. XY path length error thresholds were also moderately increased for most data sets to decrease the probability of false negatives.

Rather than simply subtracting the distance between matched nodes from path length difference, test path lengths are now adjusted based on the local trajectory of the gold standard path endpoints in order to make path length discrepancies based on endpoint position negligible. Specifically, the path of the gold standard is now followed until it leaves the threshold region of the endpoint. The crossing is referred to as the "trajectory point", as it approximately tracks the local trajectory of the gold standard path. The distances from the trajectory point to the gold standard endpoint and from the trajectory point to the test endpoint are compared. The difference between those distances is added to or subtracted from the test path length depending on whether the test distance is shorter or longer, respectively. This change reduces false positives by adjusting the resultant path length difference (and therefore the path length error) more appropriately.

In order to reduce the risk of false positive continuations by sampling an excessive number of descendants, continuations are now determined after all nodes have been checked for direct matches. As has always been the case, the gold standard tree is traversed toward the root and toward terminations to find ancestors and descendants to use to find a matching path in the test reconstruction. Ancestors were only traversed until a known match was found in order to keep the path check as local as possible and avoid ameliorating possible path length differences by including regions of known similarity. This same concept is now applied to descendants, whereby once a descendant node is found with a known match, its descendants will not be tested.

 

Q. I notice a lot of Z step jumps in OP_1.swc from the Olfactory Projection data set. Is this correct?
A. No, this effect is due to a rounding error that has been fixed. If you downloaded the file "Olfactory Projection Fibers v2.rar" on/after Wednesday, Nov 25, 2009, these edits have already been made to OP_1.swc and no further action is required.

The MD5sum value for the updated "Olfactory Projection Fibers v2.rar" is: 2d6efa7fd48c17e492879edb771a5f70

However, if you downloaded the file "Olfactory Projection Fibers.rar" prior to Wednesday, Nov 25, 2009 (MD5sum 627e3e3a656c491514b1db1207d793d7), these edits have not been made, and scores may be affected. Please download the updated "Olfactory Projection Fibers v2.rar" or download the corrected OP_1.swc file (MD5sum d87c1bad139d675404136f7d21c06029).

Note that both OP_2.swc and OP_3.swc have Z values in integers. This is not an error for these reconstructions, unlike for OP_1.swc. Thus, the OP_2 and OP_3 reconstructions included in “Olfactory Projection Fibers v2.rar” are the same as in the previous "Olfactory Projection Fibers.rar".

 

Q. In some data sets, there are several branches that I would have manually traced differently than the training reconstruction. How can these be considered objective gold standards to evaluate automated tracings?
A. Experimental data is sometime ambiguous, and arbitrary choices are occasionally unavoidable. Lab providers have confirmed that there is subjectivity in the more complex data sets. The scoring thresholds should account for much of the subjectivity. If you feel certain that a point should have been traced differently, it is strongly suggested that you trace it how you feel it should be traced. Getting hung up trying to develop an algorithm that works around such problems is counter-productive to the purpose of DIADEM. The number of potentially contentious trace points is small enough that good algorithms will still get good scores. For those scores where subjective or contentious points are a major issue, the judges will go point by point through the misses to determine how the trace should be scored.

 

Q. Is there any restriction on programming language?
A. There is no restriction as long as all necessary tools to run the program are publicly available.

 

Q. What are the MD5sums for all of the individual files comprising the data sets?
A. A zip file containing five lists of md5sums (one for each data set) for each individual data set file can be downloaded here. Please contact the DIADEM organizers if you are concerned about any value inconsistencies. Note that the md5sums for the data set rar files are provided in FAQ points above.

 

Q. What changed in the DIADEM metric released on 1/26/2010?
A. An error was found in the metric resulting in a rare NullPointerException thrown on line 1374 of DiademMetric.java. This error has no impact on scores. If you experience any other error in the metric, please contact the DIADEM organizers.

 

Q. What changed in the DIADEM metric released on 2/19/2010? Why are some of nodes that should be scored as continuations being scored as misses in the prior release?
A. Four errors exist in the Diadem Metric code introduced in the release (11/25/09) that contained new functionality for improving the accuracy of the metric. The first error causes a NullPointerException if either the gold standard or test reconstruction has no bifurcations (note: all of the provided gold standards have at least one bifurcation). The second error causes a NullPointerException if either reconstruction has a bifurcation at their second compartment (i.e. the second point in the SWC file), such as in NC_06.swc. The third error causes only one of the root's subtrees to be considered if the root is a bifurcation (or multifurcation). None of the gold standard files have bifurcations at the root node, so this would only be an issue for test reconstructions.

The fourth error occurs when more than one bifurcation is missed in a row. While checking the descendants and ancestors of the gold standard nodes for matching nodes in the test tree, the error causes only the immediate children of the target gold standard node to be checked. The correct functionality is for all descendants to be searched down until a termination or a known match is discovered along a particular descendant path. Ultimately this means that any time more than one bifurcation is missed in a row (i.e. there are multiple continuations in a row), all but the last node will fail to be marked as a continuation and thus the score will be lower than it should be. The current metric release fixes these problems.

 

Q. Do I have a chance to win if my algorithms are only working on a subset of the data?
A. We anticipate that most programs will perform better on some data sets than others. The winning entries are expected to do well (~ 20-fold faster than manual, with manual corrections) on the majority (3/5) of data sets.

 

Q. Will you release the list of participants?
A. No, only the names of the finalists will be revealed.

 

Q. My program is awesome, yet I only get a 90% score with the DIADEM metric. How can I trust a fair judgement?
A. The metric will only be used as a rough guide to judge the performance of a reconstruction algorithm. Each entry will be carefully inspected and scored by the data owners and other independent expert judges. In particular, the score will be based on three general (and appropriately weighted) components:

  1. the quantitative metric posted on the site, which provides a value between zero (completely wrong result) and one (perfect result);
  2. a semi-quantitative assessment by the data owners expressed in terms of "fold-increase" in reconstruction speed relative to manual tracing. Specifically, they will be asked to provide such an evaluation after inspecting the results (qualifier submissions) and/or estimating the amount of editing necessary to "fix" a sample of the reconstruction (in the tournament) in a standard GUI environment (e.g. Neuromantic);
  3. a qualitative assessment by "other" expert judges expressed in terms of their own tendency to adopt the given automated system compared to the existing computer-assisted manual set-ups (0%: no way, I'd much rather stay with what I have - 100% yes I'm gonna switch to this new gizmo the minute I step back in the lab). This component will also account for ergonomics, user-friendliness, general applicability, etc.

 

Q. Do the three components of the scoring judgment (metric, speed, and qualitative assessment) apply just to the final tournament phase, or to the qualifiers too? How will they be weighted? Will you provide contestants with the manual tracing speeds for the various data sets?
A. The three components, as described, will apply to the final tournament judgment. Qualifier entries will not be evaluated on the basis of speed, but only based on metric and qualitative assessment of the algorithm. The exact weight of these two factors cannot be determined in advance, but will indicatively be 50-50, with the goal of just selecting the best algorithms for the final phase. Since speed will not play a role in judging qualifier entries, manual tracing speeds will only be provided to the contestants who qualify for the final tournament (although anyone can estimate approximate values by manually reconstructing portions of the datasets).