Results
3DOMcity Task 1: Results
The evaluation procedure consists of three parts:
test
1. PRECISION ASSESSMENT
Precision in image space is evaluated by computing the re-projection errors of targets centroids and corresponding statistics: min, mean, median, max, RMS(x), RMS(y), RMS.
2. ACCURACY ASSESSMENT
Accuracy in object space is evaluated by computing the errors on 3 check scale bars.
3. RELATIVE ACCURACY ASSESSMENT
Since no independent nominal values are available for image external orientation parameters, experimental values are estimated by performing a self-calibrating BBA of the image network. A rigid similarity transformation (6DOF) is estimated to map the submitted network to the reference one (3DOM). The relative accuracy of the submitted EOP parameters is evaluated by computing their deviations from reference EOP and corresponsing statistics: min, mean, median, max, RMSE.
In the following tables a short version of the evaluation results is reported.
The full overview is available here.
3DOMcity Task 2: Results
A multi-stripe triangulation-based laser scanner, NextEngine 3D Scanner, is used to acquire the reference point clouds of two buildings of interest. In order to ensure a consistent comparison between the submitted and reference point clouds, a decimation process is performed: the mean spatial resolutions of the two point sets are computed and the largest one is adopted as sub-sampling threshold for the denser cloud.
The evaluation procedure consists of two parts:
1. ACCURACY ASSESSMENT
Accuracy is estimated by measuring the signed distances from the submitted DIM result to the reference one. The evaluation is performed on well-distributed patches, selected such as to (i) exclude areas of missing or noisy data in the reference clouds, and (ii) include areas that differ in texture and geometric complexity. The following statistics are reported: mean, median, sigma and sigmaMAD.
2. COMPLETENESS ASSESSMENT
Completeness is estimated by measuring the absolute distances from the reference point cloud to the submitted DIM one. The whole population of reference points is included in the evaluation, and completeness is defined as the fraction of reference points that are within an allowable distance from their neighbors in the DIM point cloud. The maximum acceptable distance is set to 3 times the mean spatial resolution adopted during the initial decimation process.
In the following table a short version of the evaluation results is reported. Mean, median, sigma and sigmaMAD (accuracy assessment) are in mm; completeness is reported in percentage.
The full overview is available here.
3DOMcity Task 3: Results
ACCURACY ASSESSMENT
The classification results are analysed for accuracy assessment by computing: confusion matrix, precision, recall, F1 score, true negative rate (TNR) and balanced accuracy.
Coming soon...