🎉 Welcome to ICCV’23 Sensor Fusion SLAM Challenge! 🎉

In the sensor fusion track, we exclusively offer access to high-quality sensor-fusion datasets sourced from SubT-MRS. These datasets encompass various challenging conditions such as “lighting changes, darkness, smoke, self-similar environments and more”. For the other two tracks, see here: Visual-Inertial SLAM Challenge and Lidar-Inertial SLAM Challenge.

Seize this chance to demonstrate your skills and compete among the finest in the field!

Three separate awards will be given for each track. Your SLAM performance in the Sensor Fusion track will not impact the scores in other tracks. Join us now to become a vital part of cutting-edge advancements in robotics and sensor fusion! 🤖💡 Let your expertise shine in this thrilling competition!

Important Latest Updates:

Hello everyone!

Thanks for participating the SLAM challenge and try our dataset!

The first ICCV SLAM Challenge has concluded. We have summarized the winning teams’ solutions and provided a detailed analysis of the submitted results. For more insights, please review our Paper.

In response to your requests, we have open-sourced our Ground Truth Trajectory data to facilitate easier evaluation. Looking ahead, we will release more datasets and Ground Truth Map. Stay tuned! To stay informed with the latest updates, be sure to sign up for our mailing list Thank you for your participation and contribution to the challenge! ^^

Feel free to check our Paper and cite our work if it is useful ^ ^.

@InProceedings{Zhao_2024_CVPR,
    author    = {Zhao, Shibo and Gao, Yuanjun and Wu, Tianhao and Singh, Damanpreet and Jiang, Rushan and Sun, Haoxiang and Sarawata, Mansi and Qiu, Yuheng and Whittaker, Warren and Higgins, Ian and Du, Yi and Su, Shaoshu and Xu, Can and Keller, John and Karhade, Jay and Nogueira, Lucas and Saha, Sourojit and Zhang, Ji and Wang, Wenshan and Wang, Chen and Scherer, Sebastian},
    title     = {SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {22647-22657}
}

Previous Updates:

1. To facilitate better algorithm debugging for everyone recently, we allow multiple daily submissions until 20 Sept. Starting from 21 Sept, only one submission per day is allowed.

If you have any question, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.

Time remaining:

File structure:

rosbag
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    └── (zipped) raw_data_{...}yyyy-mm-dd-hh-mm-ss{...}.bag
folder
├── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
│   ├── (zipped) cam_0
│   │   ├── {...}.png
│   │   └── timestamps.txt
│   ├── (zipped) imu
│   │   └── imu_data.csv
│   ├── (zipped) lidar
│   │   ├── {...}.las
│   │   └── timestamps.txt
│   └── (zipped) tf
│       └── tf_data.csv
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    ├── (zipped) cam_0
    │   ├── {...}.png
    │   └── timestamps.txt
    ├── (zipped) imu
    │   └── imu_data.csv
    ├── (zipped) tf
    │   └── tf_data.csv
    └── (zipped) thermal
        ├── {...}.png
        └── timestamps.txt

SubT-MRS Datasets

• Multiple Modalities: Our dataset includes hardware time-synchronized data from 4 RGB cameras, 1 LiDAR, 1 IMU, and 1 thermal camera, providing diverse and precise sensor inputs.

• Diverse Scenarios: Collected from multiple locations, the dataset exhibits varying environmental setups, encompassing indoors, outdoors, mixed indoor-outdoor, underground, off-road, and buildings, among others.

• Multi-Degraded: By incorporating multiple sensor modalities and challenging conditions like fog, snow, smoke, and illumination changes, the dataset introduces various levels of sensor degradation.

• Heterogeneous Kinematic Profiles: The SubT-MRS Dataset uniquely features time-synchronized sensor data from diverse vehicles, including RC cars, legged robots, drones, and handheld devices, each operating within distinct speed ranges.

Download

ROS bag format: Google Baidu
Folder format:     Google Baidu

Ground Truth:     Google Baidu

Evaluation

The submission will be ranked based on Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). Specifically, The ATE and RPE of every trajectory in the sensor fusion track and its bonus track will be evaluated. The final score for a submitted trajectory will be assigned according to which interval the weighted sum of the ATE and RPE lies in.

Submit the results.

Prepare the trajectory

For each of the 6 trajectories of sensor-fusion track, you need to compute the poses in IMU coordinate frame, and save them in the text file with the name sequnce_name.txt. Put all 6 files into a zip file with the following structure:

    Multi_Model_Sensor_Fusion.zip
    ├── SubT_MRS_Hawkins_Long_Corridor_RC.txt            # result file for the trajectory Long Corridor 
    ├── SubT_MRS_Hawkins_Multi_Floor_LegRobot.txt        # result file for the trajectory Multi Floor 
    ├── SubT_MRS_MILL19_Block_LiDAR.txt                  # result file for the trajectory Block LiDAR 
    ├── SubT_MRS_MILL19_Block_Visual.txt                 # result file for the trajectory Block Visual   
    ├── SubT_MRS_Flash_Light_LegRobot.txt                # result file for the trajectory Flash Light
    ├── SubT_MRS_Hawkins_Smoke_Handheld.txt              # result file for the smoke room 
    └── Subt_MRS_Outdoor_Night_LegRobot.txt              # result file for the outdoor night
  

The text file should have the following format:

# timestamp_s tx ty tz qx qy qz qw
1.403636580013555527e+09 0.0 0.0 0.0 0.0 0.0 0.0 0.0

It is a text file containing the translation and orientation of the IMU in a fixed coordinate frame. The estimated trajectory file should satisfy the following requirements.

• Each line in the text file contains a single pose.
• The number of lines/poses must be the same as the number of image frames in that trajectory.
• The format of each line is ‘timestamp_s tx ty tz qx qy qz qw’.
• tx ty tz (3 floats) give the position of IMU sensor to the world origin in the world frame.
• qx qy qz qw (4 floats) give the orientation of IMU in the form of a unit quaternion with respect to the world frame.
• The trajectory can have an arbitrary initial position and orientation. However, we are using the IMU frame to define the motion. That is to say, the x-axis is pointing to forward, the y-axis is pointing left, the z-axis is pointing up.

Submit in Gradescope

To submit the estimated trajectory into the submission system, you can follow the steps listed below:

1. Register a account in the GradeScope and log into the website.
2. Click the right-bottom Add Course button and enter the course-entry code: G2YGGB, Then you can find the iccv-mul courses in your GradeScope homepage.
3. Click the iccv-mul course and you will see the assignment named Trajectory-result-submission in the dashboard.
4. Click the assignment and upload your Sensor_Fusion.zip file. Also please remember to input the group name as the leaderboard name. Then click the upload button.
   • You should directly compress the estimated result files of the trajectories into a zip file, not the folder containing the result files.
5. After around 1 minutes, you will see the APE and RPE result of your trajectory in the leaderboard.

Submit Report

Participants are requested to submit a report describing their methods along with the gradescope submission. A template for the same is provided here : ICCV_Template_Report . Please include your report pdf in the Sensor_Fusion.zip file.

Challenge Rules

1. Participants are welcome to form teams. A participant cannot be in multiple teams and a team must make submissions under a single account.
2. Every day a team can submit for at most once on gradescope and the submission must be in a certain time window: 12:00 P.M. - 11:59 P.M. UTC.
3. The size of every trajectory file submitted should be no more than 2 MB.
4. Every team must submit a report along with the gradescope submissions.
5. Organizers reserve the right to make changes to the rules and timeline.
6. Violation of the rules or other unfair activities may result in disqualification.

🎉Sensor-Fusion Leaderboard 🎉

As of Sept 25, 11 AM, EDT.

Note: The ATE(RPE) rank is according to the mean of the reciprocal of each trajectory’s ATE(RPE) score, weighted by the trajectory length.

Summary

We summarized the SLAM performance of above top teams. Feel free to check our Paper and cite our work.

@InProceedings{Zhao_2024_CVPR,
    author    = {Zhao, Shibo and Gao, Yuanjun and Wu, Tianhao and Singh, Damanpreet and Jiang, Rushan and Sun, Haoxiang and Sarawata, Mansi and Qiu, Yuheng and Whittaker, Warren and Higgins, Ian and Du, Yi and Su, Shaoshu and Xu, Can and Keller, John and Karhade, Jay and Nogueira, Lucas and Saha, Sourojit and Zhang, Ji and Wang, Wenshan and Wang, Chen and Scherer, Sebastian},
    title     = {SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {22647-22657}
}

Disclaimer

Due to the rapidly changing nature of the SLAM, the above competitation results and SLAM summary is updated on October 15th 2023. In addition, due to the rich body of literature, there may be inaccuracies or mistakes in the paper. We welcome readers to send pull requests to our GitHub repository (inside https://github.com/water-horse/ICCV2023_SLAM_Challenge.git) so we may continue to update our references, correct the mistakes and inaccuracies, as well as updating the entries of the studies in the paper.

License

The SubT-MRS Dataset has been licensed under the CC by 4.0 License and has been collected by the AirLab.

Contact us

If you have any question or see anything wrong, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.