🎉 Welcome to ICCV’23 Visual-Inertial SLAM Challenge! 🎉

In the visual-inertial track, we exclusively offer access to high-quality visual-inertial datasets sourced from SubT-MRS and TartanAir. These datasets encompass various challenging conditions such as “lighting changes, darkness, smoke, self-similar environments and more” providing a test from simulation to real-world. For the other two tracks, see here: Lidar-Inertial SLAM Challenge and Sensor Fusion SLAM Challenge.

Seize this chance to demonstrate your skills and compete among the finest in the field!

Three separate awards will be given for each track. Join us now to become a vital part of cutting-edge advancements in robotics and sensor fusion! 🤖💡 Let your expertise shine in this thrilling competition!

Important Latest Updates:

Hello everyone!

Thanks for participating the SLAM challenge and try our dataset!

The first ICCV SLAM Challenge has concluded. We have summarized the winning teams’ solutions and provided a detailed analysis of the submitted results. For more insights, please review our Paper.

In response to your requests, we have open-sourced our Ground Truth Trajectory data to facilitate easier evaluation. This will be valuable for your research and benchmarking.

Looking ahead, we will release more datasets and Ground Truth Map. Stay tuned! To stay informed with the latest updates, be sure to sign up for our mailing list

Thank you for your participation and contribution to the challenge! ^^

Previous Updates:

1. To facilitate better algorithm debugging for everyone recently, we allow multiple daily submissions until 20 Sept. Starting from 21 Sept, only one submission per day is allowed.

2. We’ve observed an issue with the extrinsics of three trajectories, i.e., Endofworld, Moon, and Westerndesert. Please download the extrinsics calibration file again. We are very sorry for any inconvenience caused.

If you have any question, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.

Time remaining:

File structure:

rosbag
├── TartanAir_visual_{places ...}_noise0.bag
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    └── (zipped) raw_data_{...}yyyy-mm-dd-hh-mm-ss{...}.bag

folder
├── TartanAir_visual_{places ...}.zip
│   ├── (zipped) imu
│   │   └── [acc/gyro/imu/imu_time].[npy/txt]
│   └── (zipped) image_lcam_front
│       ├── {...}_lcam_front.png
│       └── timestamps.txt
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    ├── (zipped) cam_0
    │   ├── {...}.png
    │   └── timestamps.txt
    ├── (zipped) imu
    │   └── imu_data.csv
    └── (zipped) tf
        └── tf_data.csv

SubT-MRS Datasets

• Multiple Modalities: Our dataset includes hardware time-synchronized data from 4 RGB cameras, 1 LiDAR, 1 IMU, and 1 thermal camera, providing diverse and precise sensor inputs.

• Diverse Scenarios: Collected from multiple locations, the dataset exhibits varying environmental setups, encompassing indoors, outdoors, mixed indoor-outdoor, underground, off-road, and buildings, among others.

• Multi-Degraded: By incorporating multiple sensor modalities and challenging conditions like fog, snow, smoke, and illumination changes, the dataset introduces various levels of sensor degradation.

• Heterogeneous Kinematic Profiles: The SubT-MRS Dataset uniquely features time-synchronized sensor data from diverse vehicles, including RC cars, legged robots, drones, and handheld devices, each operating within distinct speed ranges.

Tartan Air Datasets

This benchmark is based on the TartanAir dataset, which is collected in photo-realistic simulation environments based on the AirSim project. A special goal of this dataset is to focus on the challenging environments with changing light conditions, adverse weather, and dynamic objects. The four most important features of our dataset are:

• Large size diverse realistic data. We collect the data in diverse environments with different styles, covering indoor/outdoor, different weather, different seasons, urban/rural.
• Multimodal ground truth labels. We provide RGB stereo, depth, optical flow, and semantic segmentation images, which facilitates the training and evaluation of various visual SLAM methods.
• Diversity of motion patterns. Our dataset covers much more diverse motion combinations in 3D space, which is significantly more difficult than existing datasets.
• Challenging Scenes. We include challenging scenes with difficult lighting conditions, day-night alternating, low illumination, weather effects (rain, snow, wind and fog) and seasonal changes.Please refer to the TartanAir Dataset and the paper for more information.

Folder structure inside the Tartan Air dataset:

    visual_envname
    ├── image_lcam_front                        # image folder
    │   ├── timestamps.txt                      # image timestamp
    │   ├── 000000_lcam_front.png               # RGB image 000000
    │   ├── 000001_lcam_front.png               # RGB image 000001
    │   ├── ... ...
    │   └── 000xxx_lcam_front.png               # RGB image 000xxx
    │
    └── imu                                     # IMU folder
        ├── acc.npy                             # IMU acceleration
        ├── acc.txt                             # IMU acceleration
        ├── gyro.npy                            # IMU gyroscope
        ├── gyro.txt                            # IMU gyroscope
        ├── imu.npy                             # IMU acceleration and gyroscope
        ├── imu.txt                             # IMU acceleration and gyroscope
        ├── imu_time.npy                        # IMU timestamp
        └── imu_time.txt                        # IMU timestamp

Download

ROS bag format: Google Baidu
Folder format:     Google Baidu

Ground Truth: Google Baidu

Bonus Tracks

🚀 We also provide 3 extra datasets from Sensor Fusion Challenge as bonuses in the competition. You will get extra scores if you test your algoithm on Bonus Track and submit the results to us.

Evaluation

The submission will be ranked based on Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). Specifically, The ATE and RPE of every trajectory in the visual inertial track and its bonus track will be evaluated. The final score for a submitted trajectory will be assigned according to which interval the weighted sum of the ATE and RPE lies in.

Submit the results.

Prepare the trajectory

For each of the 9 trajectories of visual-inertial track, you need to compute the poses in IMU coordinate frame, and save them in the text file with the name sequnce_name.txt. Put all 9 files into a zip file with the following structure:

    visual_inertial_track.zip
    ├── SubT_MRS_Laurel_Caverns_Handheld1.txt     # result file for the trajectory Laurel_Caverns_Handheld1
    ├── SubT_MRS_Laurel_Caverns_Handheld2.txt     # result file for the trajectory Laurel_Caverns_Handheld2
    ├── SubT_MRS_OverExposure_LegRobot.txt        # result of te trajectory OverExposure_LegRobot
    ├── TartanAir_visual_endofworld.txt           # result file for the trajectory Urban_Challenge_UGV1
    ├── TartanAir_visual_moon.txt                 # result file for the trajectory Urban_Challenge_UGV2
    ├── TartanAir_visual_westerndesert.txt        # result file for the trajectory Laurel_Cavern
    │   (Below are Bonuses)
    ├── SubT_MRS_Flash_Light_LegRobot.txt         # result file for the trajectory Flash Light
    ├── SubT_MRS_Hawkins_Smoke_Handheld.txt       # result file for the smoke room 
    └── Subt_MRS_Outdoor_Night_LegRobot.txt       # result file for the outdoor night

The text file should have the following format:

# timestamp_s tx ty tz qx qy qz qw
1.403636580013555527e+09 0.0 0.0 0.0 0.0 0.0 0.0 0.0

It is a text file containing the translation and orientation of the IMU in a fixed coordinate frame. The estimated trajectory file should satisfy the following requirements.

• Each line in the text file contains a single pose.
• The format of each line is ‘timestamp_s tx ty tz qx qy qz qw’.
• tx ty tz (3 floats) give the position of IMU sensor to the world origin in the world frame.
• qx qy qz qw (4 floats) give the orientation of IMU in the form of a unit quaternion with respect to the world frame.
• The trajectory can have an arbitrary initial position and orientation. However, we are using the IMU frame to define the motion. That is to say, the x-axis is pointing to forward, the y-axis is pointing left, the z-axis is pointing up.

Submit in Gradescope

To submit the estimated trajectory into the submission system, you can follow the steps listed below:

1. Register a account in the GradeScope and log into the website.
2. Click the right-bottom Add Course button and enter the course-entry code: V5NPPX , Then you can find the iccv-vi courses in your GradeScope homepage.
3. Click the iccv-vi course and you will see the assignment named Trajectory-result-submission in the dashboard.
4. Click the assignment and upload your visual-inertial-track.zip file. Also please remember to input the group name as the leaderboard name. Then click the upload button.
   • You should directly compress the estimated result files of the trajectories into a zip file, not the folder containing the result files.
5. After around 1 minutes, you will see the APE and RPE result of your trajectory in the leaderboard.

• Note:
  1. You must submit all the 9 trajectories for visual inertial track.
  2. The trajecotry should be complete. The duration of estimated trajecotry should be roughly same with ground truth trajectory.

Submit Report

Participants are requested to submit a report describing their methods along with the gradescope submission. A template for the same is provided here : ICCV_Template_Report . Please include your report pdf in the visual-inertial-track.zip file.

Challenge Rules

1. Participants are welcome to form teams. A participant cannot be in multiple teams and a team must make submissions under a single account.
2. Every day a team can submit for at most once on gradescope and the submission must be in a certain time window: 12:00 P.M. - 11:59 P.M. UTC.
3. The size of every trajectory file submitted should be no more than 2 MB.
4. Every team must submit a report along with the gradescope submissions.
5. Organizers reserve the right to make changes to the rules and timeline.
6. Violation of the rules or other unfair activities may result in disqualification.

🎉Visual-inertial Leaderboard🎉

As of 11:59 PM EDT, October 15th, 2023.

Note: The ATE(RPE) rank is according to the mean of the reciprocal of each trajectory’s ATE(RPE) score, weighted by the trajectory length.

Summary

We summarized the SLAM performance of above top teams. Feel free to check our Paper and cite our work.

@misc{zhao2023subtmrs,
      title={SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments}, 
      author={Shibo Zhao and Yuanjun Gao and Tianhao Wu and Damanpreet Singh and Rushan Jiang and Haoxiang Sun and Mansi Sarawata and Yuheng Qiu and Warren Whittaker and Ian Higgins and Yi Du and Shaoshu Su and Can Xu and John Keller and Jay Karhade and Lucas Nogueira and Sourojit Saha and Ji Zhang and Wenshan Wang and Chen Wang and Sebastian Scherer},
      year={2023},
      eprint={2307.07607},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

Contact us

If you have any question or see anything wrong, please do not hesitate to post issues on this github website . We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.

Disclaimer

Due to the rapidly changing nature of the SLAM, the above competitation results and SLAM summary is updated on October 15th 2023. In addition, due to the rich body of literature, there may be inaccuracies or mistakes in the paper. We welcome readers to send pull requests to our GitHub repository (inside https://github.com/water-horse/ICCV2023_SLAM_Challenge.git) so we may continue to update our references, correct the mistakes and inaccuracies, as well as updating the entries of the studies in the paper.