Neuronland

NLMorphologyConverter -  Testing Regime
Two features of the converter allow us to implement quite easily a very effective testing regime, ensuring a high confidence in the quality of the algorithms,
  1. Bi-directionality - incorporation of both import and export functionality into the application (in fact, for many formats there will be little to no interest in the export capability - the main advantage of implementing the exporter derives from its usage in tests).
  2. Symmetry of import and export algorithm implementation - i.e. for original and exported files, the same internal representation (data structure layout in computer memory) of a neuron is generated. This is achieved simply by taking care with the ordering of soma, tree, spine and marker data in the import and export algorithms.
To check the functionality of NLMorphologyConverter, an automated testing regime has been set up, using a large database of publicly available morphology data.
Per-file automated testing which has been implemented -
  • Auto-validation - Checks that validation algorithms correctly distinguish all supported file formats.
  • Symmetric Import - Export - Re-import .Checks that the import and export functionality for a specific file format is consistent. To pass the test, original and generated files must produce the same internal representation of a neuron, and usually original and exported files are very similar if not identical. Success in this test does not guarantee that the files are imported correctly, just that the import/export algorithms produce equivalent data. If the initial import algorithm is not correct in some way, but generates valid neuron data, then all subsequent tests still can be successful. This test is now extented to cover all different supported styles and versions of a specific file format.
  • Asymmetric Import - Export - Re-import.  Checks that the import (native format) and export/import functionality for all other file formats is consistent. To pass the test, original and generated files must produce the same internal representation of a neuron. This is a very intensive test, and checks the consistency between the import and export algorithms for all file formats, providing a higher level of confidence in their correctness. However, this test still does not guarantee the correctness of the original import algorithm.This test is now extented to cover all different supported styles and/or versions of a specific file format.
  • Import - Statistics Comparison.  Check that imported tree has expected properties, which will be contained in a manually constructed file for each neuron. This test fills a gap that the current automated tests cannot fill, by validating the initial import against known neuron properties and statistics. This takes a lot of time to setup, and only a limited number of datafiles have been prepared, across most formats.
Single data, multi-format testing which has been implemented -
  • Multi-format import consistency. Certain neuron data are already available in more than one format (e.g. CNIC archive which contains the same data in SWC, NeuroZoom, and NeuronGeneric formats) - this test validates that N different files (of different formats, but each supposedly with the same neuron data) all result in the same internal memory data representation. Data generated by the NLMorphologyConverter are not included in this test (it would be pointless to do so, since the current testing regime has already assured that the same internal representation will be generated). To function well this test obviously relies on the original method of converting between file formats being correct.
Per-file automated testing which could be implemented in the future -
  • Import - Export - Re-import - Export - File Comparison. An extension of the asymmetric test, which compares the output file generated.  This test really only checks for rare cases of redundant data accumulating in the generated files. (Note - a text file comparison can be performed directly between original and exported data for some formats, but in general files cannot be reproduced identically.)
  • Import - External Tool Conversion - Import.  Incorporate other format converters into the testing regime. 

What actually gets compared in comparison tests (where possible):
  • Trees  (Number, connectivity, and all sample points - position + diameter)
  • Somas (Number, ordering, and all sample points - position + diameter)
  • Spines (Global, Axon, Dendrite spine points: number, ordering, position, and associated sample points.)
  • Markers (Global, Soma, Axon, Dendrite, and their ordering, position, and names.)
Limitations of the tests -
  • Many formats support a data representation that is not as well-ordered as that generated by the export algorithms, and original data may not be available for testing certain special cases. (This problem has been reduced somewhat by including support for different format layout styles.)
  • Asymmetric Tests:  if some dendrites/axons are non-binary trees, then all dendrites/axons are not compared when the test format only supports binary trees (this will be altered in the future, so that at least the first 2 children of an N-ary branch point will be tested.)
  • In general, if the target test format does not support a specific type of structure found in the source format, that structure will not be tested.  Any other compatible data will be tested.
  • Supported formats which are NOT currently included in the asymmetric test - NeuronJ, ImarisHDF (no export algorithm), and Glasgow (due to known limitations of the implicit connectivity of this format).
  • Tests are currently only run on the windows OS. There is the possibility that platform-specific bugs in the Mac OSX version have not been found yet.


Test Results -


Neuron Morphology Database
(Formats Available)
 Number of files
 		Test Result Summary (NLMorphologyConverter Version 0.9.0)
  Validation 
Manual Corrections required for successful test.  + Other notes..
  Symmetric
 Asymmetric
NeuroMorpho database version 5.6
20011  (comprising original files + standardized SWC files)
 		All files validated successfuly
 Manually Corrected Files ->

markram : Source Version
  • C050398B-I4.asc (remove leading ")
  • C140300B-I2.asc (remove leading ")
smith : CNG Version
  • 38-4-4-HCB.CNG.swc (missing parent tag -1,  1->-1)
smith: Source Version
  • S4-V-C4.swc (missing parent tag -1, 223 -> -1)
stevens: Source Version
  • 20060829g850r2c1.ASC  (missing COMMENT + IMAGECOORDS TAG..)
yuste: Source Version
  • Various files with lines concatenated (and therefore ignored since they become effectively commented-out.)  Causes failures when it affects essential structure.
 All files PASS


 All files/formats PASS.
Duke/Southampton Archive
(NeurolucidaClassic, SWC)
 188
83   - NeurolucidaClassic
105 - SWC 		All files validated successfully n262.asc was manually corrected for 2 bad data points All files PASS.

 All files/formats PASS.
.
Virtual NeuroMorphology Electronic Database
(SWC, Eutectics, ArborVitae)
 561
6     - ArborVitae
22   - Eutectics
533 - SWC 		All files validated successfully
All files PASS.
 All files/formats PASS.
Gulyas Archive
(GulyasTree, SWC.)
 204
103 - GulyasTree
101 - SWC 		All files validated successfully
All files PASS. All files/formats PASS.
CNIC Archive
(NeuroZoom, SWC, NeuronSWC2HOC)
 240
80  - NeuroZoom
80  - SWC
80  - NeuronSWC2HOC
 All files validated successfully
All files PASS.   All files/formats PASS.
Yuste Archive
(NeurolucidaDAT)
 804
Note - NeurolucidaASC and SWC versions can be found in the NeuroMorpho database.



 All files validated successfully 5 files from this archive were ignored since they are corrupted (terminate prematurely):
JM083105-20.dat
KP052207.dat
MD070426-4.dat
MD070430-1.dat
NL070508-1.dat		 All files PASS. 


All files/formats PASS.
Private Archive.
 todo