🎉 Welcome to ICCV’23 LiDAR-Inertial SLAM Challenge! 🎉

In the Lidar-inertial track, we exclusively offer access to high-quality LiDAR-inertial datasets sourced from SubT-MRS and TartanAir. These datasets encompass various challenging conditions such as “multi-floor,long corridor, self-similar environments and more” providing a test from simulation to real-world. For the other two tracks, see here: Visual-Inertial SLAM Challenge and Sensor Fusion SLAM Challenge.

Seize this chance to demonstrate your skills and compete among the finest in the field!

Three separate awards will be given for each track. Join us now to become a vital part of cutting-edge advancements in robotics and sensor fusion! 🤖💡 Let your expertise shine in this thrilling competition!

Important Latest Updates:

Hello everyone!

Thanks for participating the SLAM challenge and try our dataset!

The first ICCV SLAM Challenge has concluded. We have summarized the winning teams’ solutions and provided a detailed analysis of the submitted results. For more insights, please review our Paper.

In response to your requests, we have open-sourced our Ground Truth Trajectory data to facilitate easier evaluation. This will be valuable for your research and benchmarking.

Looking ahead, we will release more dataset with Ground Truth Map. Stay tuned! To stay informed with the latest updates, be sure to sign up for our mailing list

Thank you for your participation and contribution to the challenge! ^^

Previous Updates:

1. To facilitate better algorithm debugging for everyone recently, we allow multiple daily submissions until 20 Sept. Starting from 21 Sept, only one submission per day is allowed.

2. We’ve observed an issue with the extrinsics of three trajectories, i.e., Lidar_factory, Lidar_ocean, and Lidar_sewerage. Please download the calibration file again. We are very sorry for any inconvenience caused.

3. We’ve observed an issue with one trajectory that is released in the LiDAR-inertial track. The length of the Final_Challenge_UGV3 trajectory recorded by ROS bags is about 80 seconds shorter than the same trajectory recorded by the folder-format data. To solve this issue, we have updated the ZIP file containing ROS bags of that trajectory to match the length of the folder-format data. If you have already downloaded that ZIP file, you can avoid downloading lots of Gigabytes again by simply downloading the last two missing ROS bag files here. We are very sorry for any inconvenience caused.

If you have any question, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.

Time remaining:

File structure:

rosbag
├── TartanAir_lidar_{places ...}_noise0.bag
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    └── (zipped) raw_data_{...}yyyy-mm-dd-hh-mm-ss{...}.bag

folder
├── TartanAir_lidar_{places ...}.zip
│   ├── (zipped) imu
│   │   └── [acc/gyro/imu/imu_time].[npy/txt]
│   └── (zipped) lidar
│       ├── {...}_lcam_front_lidar.ply
│       └── timestamps.txt
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    ├── (zipped) imu
    │   └── imu_data.csv
    ├── (zipped) lidar
    │   ├── {...}.las
    │   └── timestamps.txt
    └── (zipped) tf
        └── tf_data.csv

SubT-MRS Datasets

• Multiple Modalities: Our dataset includes hardware time-synchronized data from 4 RGB cameras, 1 LiDAR, 1 IMU, and 1 thermal camera, providing diverse and precise sensor inputs.

• Diverse Scenarios: Collected from multiple locations, the dataset exhibits varying environmental setups, encompassing indoors, outdoors, mixed indoor-outdoor, underground, off-road, and buildings, among others.

• Multi-Degraded: By incorporating multiple sensor modalities and challenging conditions like fog, snow, smoke, and illumination changes, the dataset introduces various levels of sensor degradation.

• Heterogeneous Kinematic Profiles: The SubT-MRS Dataset uniquely features time-synchronized sensor data from diverse vehicles, including RC cars, legged robots, drones, and handheld devices, each operating within distinct speed ranges.

TartanAir Dataset

This benchmark is based on the TartanAir dataset, which is collected in photo-realistic simulation environments based on the AirSim project. A special goal of this dataset is to focus on the challenging environments with changing light conditions, adverse weather, and dynamic objects. The four most important features of our dataset are:

• Large size diverse realistic data. We collect the data in diverse environments with different styles, covering indoor/outdoor, different weather, different seasons, urban/rural.
• Multimodal ground truth labels. We provide RGB stereo, depth, optical flow, and semantic segmentation images, which facilitates the training and evaluation of various visual SLAM methods.
• Diversity of motion patterns. Our dataset covers much more diverse motion combinations in 3D space, which is significantly more difficult than existing datasets.
• Challenging Scenes. We include challenging scenes with difficult lighting conditions, day-night alternating, low illumination, weather effects (rain, snow, wind and fog) and seasonal changes.Please refer to the TartanAir Dataset and the paper for more information.

Folder structure inside the Tartan Air dataset:

    lidar_envname
    ├── lidar                                   # LiDAR folder
    │   ├── timestamps.txt                      # LiDAR timestamp
    │   ├── 000000_lcam_front_lidar.png         # RGB LiDAR 000000
    │   ├── 000001_lcam_front_lidar.png         # RGB LiDAR 000001
    │   ├── ... ...
    │   └── 000xxx_lcam_front_lidar.png         # RGB LiDAR 000xxx
    │
    └── imu                                     # IMU folder
        ├── acc.npy                             # IMU acceleration
        ├── acc.txt                             # IMU acceleration
        ├── gyro.npy                            # IMU gyroscope
        ├── gyro.txt                            # IMU gyroscope
        ├── imu.npy                             # IMU acceleration and gyroscope
        ├── imu.txt                             # IMU acceleration and gyroscope
        ├── imu_time.npy                        # IMU timestamp
        └── imu_time.txt                        # IMU timestamp

Download

ROS bag format: Google Baidu
Folder format:     Google Baidu

Ground Truth: Google Baidu

Bonus Tracks

🚀 We also provide 3 extra datasets from Sensor Fusion Challenge as bonuses in the competition. You will get extra scores if you test your algoithm on Bonus Track and submit the results to us.

Evaluation

The submission will be ranked based on Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). Specifically, The ATE and RPE of every trajectory in the lidar inertial track and its bonus track will be evaluated. The final score for a submitted trajectory will be assigned according to which interval the weighted sum of the ATE and RPE lies in.

Submit the results.

Prepare the trajectory

For each of the 12 trajectories of LiDAR-inertial track, you need to compute the poses in IMU coordinate frame, and save them in the estimated trajectory text file with the name sequnce_name.txt. Put all 12 files into a zip file with the following structure:

    lidar_inertial_track.zip
    ├── SubT_MRS_Final_Challenge_UGV1.txt         # result file for the trajectory Final_Challenge_UGV1
    ├── SubT_MRS_Final_Challenge_UGV2.txt         # result file for the trajectory Final_Challenge_UGV2
    ├── SubT_MRS_Final_Challenge_UGV3.txt         # result file for the Final_Challenge_UGV3
    ├── SubT_MRS_Urban_Challenge_UGV1.txt         # result file for the Urban_Challenge_UGV1
    ├── SubT_MRS_Urban_Challenge_UGV2.txt         # result file for the Urban_Challenge_UGV2 
    ├── SubT_MRS_Laurel_Caverns_Handheld3.txt     # result file for the Laureal Cavern  
    ├── TartanAir_lidar_factory.txt               # result file for the trajectory lidar_factory 
    ├── TartanAir_lidar_ocean.txt                 # result file for the trajectory lidar_ocean 
    ├── TartanAir_lidar_sewerage.txt              # result file for the trajectory lidar_sewerage 
    │   (Below are Bonuses)
    ├── SubT_MRS_Hawkins_Long_Corridor_RC.txt     # result file for the trajectory Long Corridor 
    ├── SubT_MRS_Hawkins_Multi_Floor_LegRobot.txt # result file for the trajectory Multi Floor 
    └── SubT_MRS_MILL19_Block_LiDAR.txt           # result file for the trajectory Block LiDAR  

The estimated_trajecotry.txt file should have the following format:

# timestamp_s tx ty tz qx qy qz qw
1.403636580013555527e+09 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Here are some requirements for your estimated_trajectory.txt

• Each line in the text file contains a single pose.
• The format of each line is ‘timestamp_s tx ty tz qx qy qz qw’.
• tx ty tz (3 floats) give the position of IMU sensor to the world origin in the world frame.
• qx qy qz qw (4 floats) give the orientation of IMU in the form of a unit quaternion with respect to the world frame.
• The trajectory can have an arbitrary initial position and orientation. However, we are using the IMU frame to define the motion. That is to say, the x-axis is pointing to forward, the y-axis is pointing left, the z-axis is pointing up.

Submit in Gradescope

• To submit the estimated trajectory into the submission system, you can follow the steps listed below:
  1. Register a account in the GradeScope and log into the website.
  2. Click the right-bottom Add Course button and enter the course-entry code: K3EGGJ, Then you can find the iccv-lii courses in your GradeScope homepage.
  3. Click the iccv-lii course and you will see the assignment named Trajectory-result-submission in the dashboard.
  4. Click the assignment and upload your lidar_inertial_track.zip file. Also please remember to input the group name as the leaderboard name. Then click the upload button.
     • You should directly compress the estimated result files of the trajectories into a zip file, not the folder containing the result files.
  5. After around 1 minutes, you will see the APE and RPE result of your trajectory in the leaderboard.
• Note:
  1. You must submit all the 12 trajectories for lidar inertial track.
  2. The trajecotry should be complete. The duration of estimated trajecotry should be roughly same with ground truth trajectory.

Submit Report

Participants are requested to submit a report describing their methods along with the gradescope submission. A template for the same is provided here : ICCV_Template_Report . Please include your report pdf in the lidar_inertial_track.zip file.

Challenge Rules

1. Participants are welcome to form teams. A participant cannot be in multiple teams and a team must make submissions under a single account.
2. Every day a team can submit for at most once on gradescope and the submission must be in a certain time window: 12:00 P.M. - 11:59 P.M. UTC.
3. The size of every trajectory file submitted should be no more than 2 MB.
4. Every team must submit a report along with the gradescope submissions.
5. Organizers reserve the right to make changes to the rules and timeline.
6. Violation of the rules or other unfair activities may result in disqualification.

🎉Lidar-inertial Leaderboard🎉

As of 11:59 PM EDT, October 15th, 2023.

Note: The ATE(RPE) rank is according to the mean of the reciprocal of each trajectory’s ATE(RPE) score, weighted by the trajectory length.

Summary

We summarized the SLAM performance of above top teams. Feel free to check our Paper and cite our work.

@misc{zhao2023subtmrs,
      title={SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments}, 
      author={Shibo Zhao and Yuanjun Gao and Tianhao Wu and Damanpreet Singh and Rushan Jiang and Haoxiang Sun and Mansi Sarawata and Yuheng Qiu and Warren Whittaker and Ian Higgins and Yi Du and Shaoshu Su and Can Xu and John Keller and Jay Karhade and Lucas Nogueira and Sourojit Saha and Ji Zhang and Wenshan Wang and Chen Wang and Sebastian Scherer},
      year={2023},
      eprint={2307.07607},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

Contact us

If you have any question or see anything wrong, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.

Disclaimer

Due to the rapidly changing nature of the SLAM, the above competitation results and SLAM summary is updated on October 15th 2023. In addition, due to the rich body of literature, there may be inaccuracies or mistakes in the paper. We welcome readers to send pull requests to our GitHub repository (inside https://github.com/water-horse/ICCV2023_SLAM_Challenge.git) so we may continue to update our references, correct the mistakes and inaccuracies, as well as updating the entries of the studies in the paper.