Evaluation Instruction
The Format of Submission Result
The estimated 6 DoF camera poses (from camera coordinate to the world coordinate) are required to evaluate the performance. Considering that there is a certain randomness of estimation, each sequence is required to be run for 5 times, resulting in 5 pose files and 5 running time files. We will select the median result from all five results for evaluation. It should be noted that the format for each pose file is described as follows:
timestamp[i] p_x p_y p_z q_x q_y q_z q_w
where (p_x p_y p_z)
is the camera position,
and the unit quaternion (q_x, q_y, q_z, q_w)
is the camera orientation.
You should output the realtime poses after each frame is processed (not the poses after final global optimization),
and the output of poses should be in the same frame rate as the input camera images (Otherwise, the completeness evaluation would be affected).
The format for each running time file is described as follows
timestamp[i] t_pose
where t_pose
denotes the system time (or cumulative running time in seconds and at least three decimal places, even for black frames in D6) when the pose is estimated.
Submission
Please submit a zip file containing all the poses and running time files. The structure of zip file should follow the form described as follows:
YourSLAMName/sequence_name/Roundpose.txt
YourSLAMName/sequence_name/Roundtime.txt
e.g.
MYSLAM/C0_test/0pose.txt
MYSLAM/C0_test/0time.txt
You can click here to download the example.
Evaluation
We evaluate the overall performance of a SLAM system considering tracking accuracy, initialization quality, tracking robustness, relocalization time and the computation efficiency. The criteria are as follows:
for absolute positional / rotational error
for relative positional / rotational error
for the ratio of bad poses (100%  completeness)
for the initialization quality
for the tracking robustness
for the relocalization time
for more detail please refer to the paper[1]. We convert each criteria error into a normalized score by:
where is the variance controlling the normalization function shape. The complete score is a weighted sum of all the individual scores as:
where weight and variance for each criteria are listed below:
1.0  1.0  0.5  0.5  1.0  1.0  1.0  1.0  
small scenes  55.83  2.48  2.92  0.17  2.38  1.85  0.95  1.42 
large scenes  3626.32  19.74  5.43  0.42  2.38  7.70  18.13  1.42 
The variance list above is obtained by computing the median of our previous evaluation results, which contain the results of 4 VISLAM systems (MSCKF, OKVIS, VINSMono, SenseSLAM) evaluated on dataset of smallscale scenes and largescale scenes respectively. You can evaluate your SLAM system with our training dataset using the evaluation tool.
In the final round competition, we will test all systems on benchmarking PCs with the same hardware configuration. The running time will be taken into account for computing the final score according to the following equation:
where denotes the average framerate of the system. It should be noted that not all sequences are evaluated for all the critera.
Acknowledgement

[1] Jinyu Li, Bangbang Yang, Danpeng Chen, Nan Wang, Guofeng Zhang, Hujun Bao. Survey and evaluation of monocular visualinertial SLAM algorithms for augmented reality. Journal of Virtual Reality & Intelligent Hardware, 2019, 1(4): 386410. DOI:10.3724/SP.J.20965796.2018.0011.