🎉 Welcome to ICCV’23 LiDAR-Inertial SLAM Challenge! 🎉

In the Lidar-inertial track, we exclusively offer access to high-quality LiDAR-inertial datasets sourced from SubT-MRS and TartanAir. These datasets encompass various challenging conditions such as “multi-floor,long corridor, self-similar environments and more” providing a test from simulation to real-world. For the other two tracks, see here: Visual-Inertial SLAM Challenge and Sensor Fusion SLAM Challenge.

Seize this chance to demonstrate your skills and compete among the finest in the field!

Three separate awards will be given for each track. Join us now to become a vital part of cutting-edge advancements in robotics and sensor fusion! 🤖💡 Let your expertise shine in this thrilling competition!

Important Latest Updates:

Hello everyone!

Thanks for participating the SLAM challenge!

As the competition deadline approaches, many teams asked for an extension. To ensure fairness, we’ve extended the deadline to 11:59 PM EDT, October 15th, 2023. It is important to note that each team must provide answers to all questions in the report template. A high-quality report will also be factored into your final score and assessed as a reference for the submitted trajectory.

Starting from September 25th, each team is limited to a maximum of 30 submissions. Please be aware that exceeding this limit will result in the 30th submission being considered as the final one.

Additionally, registering multiple accounts is strictly prohibited now. Feel free to let us know if you have more questions.

We wish you the best of luck in the challenge! ^^

Previous Updates:

1. To facilitate better algorithm debugging for everyone recently, we allow multiple daily submissions until 20 Sept. Starting from 21 Sept, only one submission per day is allowed.

2. We’ve observed an issue with the extrinsics of three trajectories, i.e., Lidar_factory, Lidar_ocean, and Lidar_sewerage. Please download the calibration file again. We are very sorry for any inconvenience caused.

3. We’ve observed an issue with one trajectory that is released in the LiDAR-inertial track. The length of the Final_Challenge_UGV3 trajectory recorded by ROS bags is about 80 seconds shorter than the same trajectory recorded by the folder-format data. To solve this issue, we have updated the ZIP file containing ROS bags of that trajectory to match the length of the folder-format data. If you have already downloaded that ZIP file, you can avoid downloading lots of Gigabytes again by simply downloading the last two missing ROS bag files here. We are very sorry for any inconvenience caused.

If you have any question, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.

Time remaining:

File structure:

rosbag
├── TartanAir_lidar_{places ...}_noise0.bag
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    └── (zipped) raw_data_{...}yyyy-mm-dd-hh-mm-ss{...}.bag

folder
├── TartanAir_lidar_{places ...}.zip
│   ├── (zipped) imu
│   │   └── [acc/gyro/imu/imu_time].[npy/txt]
│   └── (zipped) lidar
│       ├── {...}_lcam_front_lidar.ply
│       └── timestamps.txt
└── SubT_MRS_{trajectory names ...}_{robot types ...}.zip
    ├── (zipped) imu
    │   └── imu_data.csv
    ├── (zipped) lidar
    │   ├── {...}.las
    │   └── timestamps.txt
    └── (zipped) tf
        └── tf_data.csv

SubT-MRS Datasets

• Multiple Modalities: Our dataset includes hardware time-synchronized data from 4 RGB cameras, 1 LiDAR, 1 IMU, and 1 thermal camera, providing diverse and precise sensor inputs.

• Diverse Scenarios: Collected from multiple locations, the dataset exhibits varying environmental setups, encompassing indoors, outdoors, mixed indoor-outdoor, underground, off-road, and buildings, among others.

• Multi-Degraded: By incorporating multiple sensor modalities and challenging conditions like fog, snow, smoke, and illumination changes, the dataset introduces various levels of sensor degradation.

• Heterogeneous Kinematic Profiles: The SubT-MRS Dataset uniquely features time-synchronized sensor data from diverse vehicles, including RC cars, legged robots, drones, and handheld devices, each operating within distinct speed ranges.

TartanAir Dataset

This benchmark is based on the TartanAir dataset, which is collected in photo-realistic simulation environments based on the AirSim project. A special goal of this dataset is to focus on the challenging environments with changing light conditions, adverse weather, and dynamic objects. The four most important features of our dataset are:

• Large size diverse realistic data. We collect the data in diverse environments with different styles, covering indoor/outdoor, different weather, different seasons, urban/rural.
• Multimodal ground truth labels. We provide RGB stereo, depth, optical flow, and semantic segmentation images, which facilitates the training and evaluation of various visual SLAM methods.
• Diversity of motion patterns. Our dataset covers much more diverse motion combinations in 3D space, which is significantly more difficult than existing datasets.
• Challenging Scenes. We include challenging scenes with difficult lighting conditions, day-night alternating, low illumination, weather effects (rain, snow, wind and fog) and seasonal changes.Please refer to the TartanAir Dataset and the paper for more information.

Folder structure inside the Tartan Air dataset:

    lidar_envname
    ├── lidar                                   # LiDAR folder
    │   ├── timestamps.txt                      # LiDAR timestamp
    │   ├── 000000_lcam_front_lidar.png         # RGB LiDAR 000000
    │   ├── 000001_lcam_front_lidar.png         # RGB LiDAR 000001
    │   ├── ... ...
    │   └── 000xxx_lcam_front_lidar.png         # RGB LiDAR 000xxx
    │
    └── imu                                     # IMU folder
        ├── acc.npy                             # IMU acceleration
        ├── acc.txt                             # IMU acceleration
        ├── gyro.npy                            # IMU gyroscope
        ├── gyro.txt                            # IMU gyroscope
        ├── imu.npy                             # IMU acceleration and gyroscope
        ├── imu.txt                             # IMU acceleration and gyroscope
        ├── imu_time.npy                        # IMU timestamp
        └── imu_time.txt                        # IMU timestamp

Download

ROS bag format: Google Baidu
Folder format:     Google Baidu

Bonus Tracks

🚀 We also provide 3 extra datasets from Sensor Fusion Challenge as bonuses in the competition. You will get extra scores if you test your algoithm on Bonus Track and submit the results to us.

Evaluation

The submission will be ranked based on Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). Specifically, The ATE and RPE of every trajectory in the lidar inertial track and its bonus track will be evaluated. The final score for a submitted trajectory will be assigned according to which interval the weighted sum of the ATE and RPE lies in.

Submit the results.

Prepare the trajectory

For each of the 12 trajectories of LiDAR-inertial track, you need to compute the poses in IMU coordinate frame, and save them in the estimated trajectory text file with the name sequnce_name.txt. Put all 12 files into a zip file with the following structure:

    lidar_inertial_track.zip
    ├── SubT_MRS_Final_Challenge_UGV1.txt         # result file for the trajectory Final_Challenge_UGV1
    ├── SubT_MRS_Final_Challenge_UGV2.txt         # result file for the trajectory Final_Challenge_UGV2
    ├── SubT_MRS_Final_Challenge_UGV3.txt         # result file for the Final_Challenge_UGV3
    ├── SubT_MRS_Urban_Challenge_UGV1.txt         # result file for the Urban_Challenge_UGV1
    ├── SubT_MRS_Urban_Challenge_UGV2.txt         # result file for the Urban_Challenge_UGV2 
    ├── SubT_MRS_Laurel_Caverns_Handheld3.txt     # result file for the Laureal Cavern  
    ├── TartanAir_lidar_factory.txt               # result file for the trajectory lidar_factory 
    ├── TartanAir_lidar_ocean.txt                 # result file for the trajectory lidar_ocean 
    ├── TartanAir_lidar_sewerage.txt              # result file for the trajectory lidar_sewerage 
    │   (Below are Bonuses)
    ├── SubT_MRS_Hawkins_Long_Corridor_RC.txt     # result file for the trajectory Long Corridor 
    ├── SubT_MRS_Hawkins_Multi_Floor_LegRobot.txt # result file for the trajectory Multi Floor 
    └── SubT_MRS_MILL19_Block_LiDAR.txt           # result file for the trajectory Block LiDAR  

The estimated_trajecotry.txt file should have the following format:

# timestamp_s tx ty tz qx qy qz qw
1.403636580013555527e+09 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Here are some requirements for your estimated_trajectory.txt

• Each line in the text file contains a single pose.
• The format of each line is ‘timestamp_s tx ty tz qx qy qz qw’.
• tx ty tz (3 floats) give the position of IMU sensor to the world origin in the world frame.
• qx qy qz qw (4 floats) give the orientation of IMU in the form of a unit quaternion with respect to the world frame.
• The trajectory can have an arbitrary initial position and orientation. However, we are using the IMU frame to define the motion. That is to say, the x-axis is pointing to forward, the y-axis is pointing left, the z-axis is pointing up.

Submit in Gradescope

• To submit the estimated trajectory into the submission system, you can follow the steps listed below:
  1. Register a account in the GradeScope and log into the website.
  2. Click the right-bottom Add Course button and enter the course-entry code: K3EGGJ, Then you can find the iccv-lii courses in your GradeScope homepage.
  3. Click the iccv-lii course and you will see the assignment named Trajectory-result-submission in the dashboard.
  4. Click the assignment and upload your lidar_inertial_track.zip file. Also please remember to input the group name as the leaderboard name. Then click the upload button.
     • You should directly compress the estimated result files of the trajectories into a zip file, not the folder containing the result files.
  5. After around 1 minutes, you will see the APE and RPE result of your trajectory in the leaderboard.
• Note:
  1. You must submit all the 12 trajectories for lidar inertial track.
  2. The trajecotry should be complete. The duration of estimated trajecotry should be roughly same with ground truth trajectory.

Submit Report

Participants are requested to submit a report describing their methods along with the gradescope submission. A template for the same is provided here : ICCV_Template_Report . Please include your report pdf in the lidar_inertial_track.zip file.

Challenge Rules

1. Participants are welcome to form teams. A participant cannot be in multiple teams and a team must make submissions under a single account.
2. Every day a team can submit for at most once on gradescope and the submission must be in a certain time window: 12:00 P.M. - 11:59 P.M. UTC.
3. The size of every trajectory file submitted should be no more than 2 MB.
4. Every team must submit a report along with the gradescope submissions.
5. Organizers reserve the right to make changes to the rules and timeline.
6. Violation of the rules or other unfair activities may result in disqualification.

🎉Lidar-inertial Leaderboard🎉

As of Sept 25, 11 AM, EDT.

Note: The ATE(RPE) rank is according to the mean of the reciprocal of each trajectory’s ATE(RPE) score, weighted by the trajectory length.

Contact us

If you have any question or see anything wrong, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.