The evaluation tests are performed using the following reference data: point clouds acquired with a triangulation-based laser scanner, a set of circular/coded targets, two calibrated scale bars, two printed scale bars, one calliper and one rigid measuring tape.
Task 1: Image Orientation
The photogrammetric network adjustment is the first task of the contest. Starting from the set of images and reference measurements, the participants are asked to perform the block triangulation and submit the adjusted orientation parameters. Precision in image space, accuracy in object space and relative EOP accuracy will be estimated as evaluation metrics.
Task 2: Dense Image Matching (DIM)
The generation of a 3D point cloud is the second task of the contest. It can be either performed independently from Task 1, by donwloading the undistorted images and reference EOP, or combined with Task 1, by first assessing the accuracy of the image orientations, and then starting from these to compute and evaluate the subsequent DIM result. Evaluation metrics will include accuracy (how close the submitted result is to the reference one) and completeness (how much of the reference result is reconstructed in the submitted one).
Task 3: Point Cloud Classification
The classification of the 3D point cloud (ca 28 mil points, ca 0.16 mm average distance betwen points ie ca 1700 points/m2) is the third task of the 3DOMcity photogrammetric contest. A reference 3D point cloud, a training set and an evaluation set are provided as input data. Participants are asked to classify the point cloud into the following classes of interests: ground, grass, shrub, tree, facade, and roof. The classification results will be analysed for accuracy assessment by computing: confusion matrix, precision, recall, F1 score, true negative rate and balanced accuracy
Further details on the benchmark datasets, tasks and evaluation procedures can be downloaded here (Last updated on Oct 9).
To access the benchmark data, please fill in the form available here and send it to email@example.com .
When using the 3DOMcity dataset in your research and publications, we will be happy if you cite us:
E. Özdemir, I. Toschi, and F. Remondino, 2019: A MULTI-PURPOSE BENCHMARK FOR PHOTOGRAMMETRIC URBAN 3D RECONSTRUCTION IN A CONTROLLED ENVIRONMENT. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., Vol. XLII-1/W2, pp. 53–60.