π Welcome to ICCVβ23 Visual-Inertial SLAM Challenge! π
In the visual-inertial track, we exclusively offer access to high-quality visual-inertial datasets sourced from SubT-MRS and TartanAir. These datasets encompass various challenging conditions such as βlighting changes, darkness, smoke, self-similar environments and moreβ providing a test from simulation to real-world. For the other two tracks, see here: Lidar-Inertial SLAM Challenge and Sensor Fusion SLAM Challenge.
Seize this chance to demonstrate your skills and compete among the finest in the field!
Three separate awards will be given for each track. Join us now to become a vital part of cutting-edge advancements in robotics and sensor fusion! π€π‘ Let your expertise shine in this thrilling competition!
Important Latest Updates:
- To facilitate better algorithm debugging for everyone recently, we allow multiple daily submissions until 20 Sept. Starting from 21 Sept, only one submission per day is allowed.
- Weβve observed an issue with the extrinsics of three trajectories, i.e., Endofworld, Moon, and Westerndesert. Please download the extrinsics calibration file again. We are very sorry for any inconvenience caused.
If you have any question, please do not hesitate to post issues on this github website. We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.
Please note challenge deadline: 25th September 2023 11:59 PM EST
Time remaining:
File structure:
rosbag
βββ TartanAir_visual_{places ...}_noise0.bag
βββ SubT_MRS_{trajectory names ...}_{robot types ...}.zip
βββ (zipped) raw_data_{...}yyyy-mm-dd-hh-mm-ss{...}.bag
folder
βββ TartanAir_visual_{places ...}.zip
β βββ (zipped) imu
β β βββ [acc/gyro/imu/imu_time].[npy/txt]
β βββ (zipped) image_lcam_front
β βββ {...}_lcam_front.png
β βββ timestamps.txt
βββ SubT_MRS_{trajectory names ...}_{robot types ...}.zip
βββ (zipped) cam_0
Β Β βΒ Β βββ {...}.png
Β Β βΒ Β βββ timestamps.txt
Β Β βββ (zipped) imu
Β Β βΒ Β βββ imu_data.csv
βββ (zipped) tf
βββ tf_data.csv
SubT-MRS Datasets
-
Multiple Modalities: Our dataset includes hardware time-synchronized data from 4 RGB cameras, 1 LiDAR, 1 IMU, and 1 thermal camera, providing diverse and precise sensor inputs.
-
Diverse Scenarios: Collected from multiple locations, the dataset exhibits varying environmental setups, encompassing indoors, outdoors, mixed indoor-outdoor, underground, off-road, and buildings, among others.
-
Multi-Degraded: By incorporating multiple sensor modalities and challenging conditions like fog, snow, smoke, and illumination changes, the dataset introduces various levels of sensor degradation.
-
Heterogeneous Kinematic Profiles: The SubT-MRS Dataset uniquely features time-synchronized sensor data from diverse vehicles, including RC cars, legged robots, drones, and handheld devices, each operating within distinct speed ranges.
Tartan Air Datasets
This benchmark is based on the TartanAir dataset, which is collected in photo-realistic simulation environments based on the AirSim project. A special goal of this dataset is to focus on the challenging environments with changing light conditions, adverse weather, and dynamic objects. The four most important features of our dataset are:
- Large size diverse realistic data. We collect the data in diverse environments with different styles, covering indoor/outdoor, different weather, different seasons, urban/rural.
- Multimodal ground truth labels. We provide RGB stereo, depth, optical flow, and semantic segmentation images, which facilitates the training and evaluation of various visual SLAM methods.
- Diversity of motion patterns. Our dataset covers much more diverse motion combinations in 3D space, which is significantly more difficult than existing datasets.
- Challenging Scenes. We include challenging scenes with difficult lighting conditions, day-night alternating, low illumination, weather effects (rain, snow, wind and fog) and seasonal changes.Please refer to the TartanAir Dataset and the paper for more information.
Folder structure inside the Tartan Air dataset:
visual_envname
βββ image_lcam_front # image folder
β βββ timestamps.txt # image timestamp
β βββ 000000_lcam_front.png # RGB image 000000
β βββ 000001_lcam_front.png # RGB image 000001
β βββ ... ...
β βββ 000xxx_lcam_front.png # RGB image 000xxx
β
βββ imu # IMU folder
βββ acc.npy # IMU acceleration
βββ acc.txt # IMU acceleration
βββ gyro.npy # IMU gyroscope
βββ gyro.txt # IMU gyroscope
βββ imu.npy # IMU acceleration and gyroscope
βββ imu.txt # IMU acceleration and gyroscope
βββ imu_time.npy # IMU timestamp
βββ imu_time.txt # IMU timestamp
Download
ROS bag format:βGoogle Baidu
Folder format:βββΒ Β Google Baidu
Name | Source | Location | Robot | Sensor | Description | Trajectory Length (m) | Duration (s) | Video | Calibration (Extrinsics) | Calibration (Intrinsics) |
---|---|---|---|---|---|---|---|---|---|---|
Handheld1 | SubT-MRS | Lauren Cavern | RC7 | IMU,RGB | Darkness | 400.61 | 816 | link | Google Baidu | Google Baidu |
Handheld2 | SubT-MRS | Lauren Cavern | RC7 | IMU,RGB | Darkness | 583.19 | 739 | link | Google Baidu | Google Baidu |
OverExposure | SubT-MRS | Hawkins | RC7 | IMU,RGB | Over Exposure | 456.26 | 2128 | link | Google Baidu | Google Baidu |
Endofworld | TartanAir | Simulation | Virtual Sensors | IMU,RGB | Fog | 280 | 70.8 | link | Google Baidu | Google Baidu |
Moon | TartanAir | Simulation | Virtual Sensors | IMU,RGB | Shaddow | 850 | 346.9 | link | Google Baidu | Google Baidu |
Westerndesert | TartanAir | Simulation | Virtual Sensors | IMU,RGB | Day-night Circle | 600 | 180.5 | link | Google Baidu | Google Baidu |
Bonus Tracks
π We also provide 3 extra datasets from Sensor Fusion Challenge as bonuses in the competition. You will get extra scores if you test your algoithm on Bonus Track and submit the results to us.
Name | Source | Location | Robot | Sensor | Description | Trajectory | Duration | Video | Calibration (Extrinsics) | Calibration (Intrinsics) |
---|---|---|---|---|---|---|---|---|---|---|
Smoke_Room | SubT-MRS | Hawkins | RC7 | RGB,Thermal,IMU | Visual Degraded | 104.84 | 418 | link | Google Baidu | Google Baidu |
Outdoor_Night | SubT-MRS | Hawkins | SP1 | RGB,Thermal,IMU | Visual Degraded | 254.03 | 484 | link | Google Baidu | Google Baidu |
FlashLight | SubT-MRS | Hawkins | SP1 | RGB,Thermal,IMU | Visual Degraded | 147.75 | 279 | link | Google Baidu | Google Baidu |
Evaluation
The submission will be ranked based on Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). Specifically, The ATE and RPE of every trajectory in the visual inertial track and its bonus track will be evaluated. The final score for a submitted trajectory will be assigned according to which interval the weighted sum of the ATE and RPE lies in.
Submit the results.
Prepare the trajectory
For each of the 9 trajectories of visual-inertial track, you need to compute the poses in IMU coordinate frame, and save them in the text file with the name sequnce_name.txt. Put all 9 files into a zip file with the following structure:
visual_inertial_track.zip
βββ SubT_MRS_Laurel_Caverns_Handheld1.txt # result file for the trajectory Laurel_Caverns_Handheld1
βββ SubT_MRS_Laurel_Caverns_Handheld2.txt # result file for the trajectory Laurel_Caverns_Handheld2
βββ SubT_MRS_OverExposure_LegRobot.txt # result of te trajectory OverExposure_LegRobot
βββ TartanAir_visual_endofworld.txt # result file for the trajectory Urban_Challenge_UGV1
βββ TartanAir_visual_moon.txt # result file for the trajectory Urban_Challenge_UGV2
βββ TartanAir_visual_westerndesert.txt # result file for the trajectory Laurel_Cavern
β (Below are Bonuses)
βββ SubT_MRS_Flash_Light_LegRobot.txt # result file for the trajectory Flash Light
βββ SubT_MRS_Hawkins_Smoke_Handheld.txt # result file for the smoke room
βββ Subt_MRS_Outdoor_Night_LegRobot.txt # result file for the outdoor night
The text file should have the following format:
# timestamp_s tx ty tz qx qy qz qw
1.403636580013555527e+09 0.0 0.0 0.0 0.0 0.0 0.0 0.0
It is a text file containing the translation and orientation of the IMU in a fixed coordinate frame. The estimated trajectory file should satisfy the following requirements.
- Each line in the text file contains a single pose.
- The format of each line is βtimestamp_s tx ty tz qx qy qz qwβ.
- tx ty tz (3 floats) give the position of IMU sensor to the world origin in the world frame.
- qx qy qz qw (4 floats) give the orientation of IMU in the form of a unit quaternion with respect to the world frame.
- The trajectory can have an arbitrary initial position and orientation. However, we are using the IMU frame to define the motion. That is to say, the x-axis is pointing to forward, the y-axis is pointing left, the z-axis is pointing up.
Submit in Gradescope
To submit the estimated trajectory into the submission system, you can follow the steps listed below:
- Register a account in the GradeScope and log into the website.
- Click the right-bottom
Add Course
button and enter the course-entry code:V5NPPX
, Then you can find theiccv-vi
courses in your GradeScope homepage. - Click the
iccv-vi
course and you will see the assignment namedTrajectory-result-submission
in the dashboard. - Click the assignment and upload your
visual-inertial-track.zip
file. Also please remember to input the group name as the leaderboard name. Then click the upload button.- You should directly compress the estimated result files of the trajectories into a zip file, not the folder containing the result files.
- After around 1 minutes, you will see the APE and RPE result of your trajectory in the leaderboard.
- Note:
- You must submit all the 9 trajectories for visual inertial track.
- The trajecotry should be complete. The duration of estimated trajecotry should be roughly same with ground truth trajectory.
Submit Report
Participants are requested to submit a report describing their methods along with the gradescope submission. A template for the same is provided here : ICCV_Template_Report . Please include your report pdf in the visual-inertial-track.zip
file.
Challenge Rules
- Participants are welcome to form teams. A participant cannot be in multiple teams and a team must make submissions under a single account.
- Every day a team can submit for at most once on gradescope and the submission must be in a certain time window: 12:00 P.M. - 11:59 P.M. UTC.
- The size of every trajectory file submitted should be no more than 2 MB.
- Every team must submit a report along with the gradescope submissions.
- Organizers reserve the right to make changes to the rules and timeline.
- Violation of the rules or other unfair activities may result in disqualification.
πVisual-inertial Leaderboardπ
Leaderboard will be open on Gradescope when there is still enough time before the challenge ends.
Contact us
If you have any question or see anything wrong, please do not hesitate to post issues on this github website . We would love to hear from your feedback! Every post will be responded with no spared effort within 36 hours.