MID-3K
MID-3K (Multimodal ISR Dataset with 3000 images)
https://github.com/kennedyk1/MID-3K
This dataset, collected by the ISR (Institute of Systems and Robotics) Team, is a new multi-sensory dataset that was organized, calibrated, curated, and annotated. The sensory data was collected using ROS and a Jackal Clearpath mobile robot (see Fig. 1), operating in two indoor environments: three floors of the DEEC building and two floors of DEI building at the University of Coimbra, Polo 2, Portugal.
This main repository provides information about the Dataset, which consists of 3,083 scenes, each containing 4 images from different modalities (RGB, thermal, depth, and intensity). The images are organized by modality into separate repositories, with the links listed below:
- MID-3K RGB Modality: Contains RGB images.
- MID-3K Thermal Modality: Contains thermal images.
- MID-3K Depth Modality: Contains depth images data captured by LiDAR.
- MID-3K Intensity Modality: Contains intensity images data captured by LiDAR.
You can clone any of these modalities to your local environment using the git clone command. Simply open the terminal and follow these steps:
git clone https://github.com/kennedyk1/MID-3K-rgb
git clone https://github.com/kennedyk1/MID-3K-thermal
git clone https://github.com/kennedyk1/MID-3K-depth
git clone https://github.com/kennedyk1/MID-3K-intensity
This dataset includes a metainfo.csv file that provides detailed information about each image, including the collection date and time, department, floor, and the number of thermal and RGB annotations. This file can be useful for splitting the dataset into training, validation, and test sets for CNN training, allowing for organized and efficient dataset management.
|
| Fig. 1: Sensors on mobile robot, Clearpath Jackal model: (RGB) Ximea MQ013CG-E2, (Thermal) Flir Boson 640 LWIR, and (LiDAR) Ouster OS1-64-U. |
|
| Fig. 2: Sample images used for sensors calibration. The composite image shows the RGB (left) and thermal (center) images, and the projection of 3D-LiDAR points over the RGB image (right). The first two images are used for sensor calibrations, while the last one is used to verify if the LiDAR-camera calibration is properly aligned. |
The dataset consists of 3083 selected frames, containing RGB, thermal, depth-map and intensity-map images (last two generated from LiDAR) totaling 12332 image files (see Fig. 2). The sensors have been calibrated, but a small temporal misalignment is present due to hardware limitations.
 |
 |
 |
 |
| Fig. 2: Dataset frame-examples composed of RGB, Thermal, Depth-Map and Intensity-Map modalities. | |||
The depth and intensity images representation was generated from the projected LiDAR point clouds using a modified bilateral filter. In particular, each LiDAR point cloud is projected on the RGB image-plane, considering the Camera-LiDAR calibration matrix, and then a sliding-window based weighing function, dependent on the range dispersion, is used to interpolate the points inside the mask therefore generating a dense representation.
| RGB | Thermal | Depth | Intensity | |
|---|---|---|---|---|
| Model | Ximea MQ013CG-E2 | FLIR BOSON 640 LWIR | OUSTER OS1-64-U | OUSTER OS1-64-U |
| Type | Colour Camera | Thermal Camera | LiDAR | LiDAR |
| Spec. | 1280x1024 pixels, 1.3 MP | 640x512 pixels | Vert. Res.: 64 channels Hor. Res.: 2048 points |
Vert. Res.: 64 channels Hor. Res.: 2048 points |
| Image Type | .png | .png | .png | .png |
| Total Images | 3,083 | 3,083 | 3,083 | 3,083 |
| Total Files Annotations | 3,083 txt files | 3,083 txt files | 3,083 txt files | 3,083 txt files |
| Total Annotations (people) | 10,824 | 10,881 | 10,824¹ | 10,824¹ |
| Dataset Size | ~1.3 GB | ~1.3 GB | ~977 MB | ~1.7 GB |
| Images Resolution | 640x512 | 640x512 | 640x512 | 640x512 |
| Annotation Format | YOLO xywhn² | YOLO xywhn² | YOLO xywhn² | YOLO xywhn² |
| ¹ The labels from the RGB modality were used because the LiDAR was calibrated with the RGB camera. ² YOLO normalized xywh format class x_center y_center width height |
||||
| Daily Distribution of Images in the Dataset | ||||||
| Day | Date | Images | Thermal Annotations | RGB Annotations | Folder | |
| #1 | 29-Apr | 368 (11.9%) | 1368 (12.6%) | 1404 (13.0%) | Test | |
| #2 | 07-May | 332 (10.8%) | 1075 (9.9%) | 1072 (9.9%) | Train | |
| #3 | 08-May | 337 (10.9%) | 1658 (15.2%) | 1651 (15.3%) | Test | |
| #4 | 09-May | 1333 (43.2%) | 4199 (38.6%) | 4125 (38.1%) | Train | |
| #5 | 16-May | 713 (23.1%) | 2581 (23.7%) | 2572 (23.8%) | Train | |
| - | - | 3083 | 10881 | 10824 | ||
Visit our datasets through the links below.
- MID-1K - Multimodal ISR Dataset with 1100 images (RGB, thermal and depth)
- MID-3K - Multimodal ISR Dataset with 3000 images (RGB, thermal, depth and intensity)
References
- Sousa, E., Mota, K. O., Gomes, I. P., Garrote, L., Wolf, D. F., & Premebida, C. (2023, September). Late-Fusion Multimodal Human Detection Based on RGB and Thermal Images for Robotic Perception. In 2023 European Conference on Mobile Robots (ECMR) (pp. 1-6).
- Kennedy O. S. Mota, Luís Garrote, Cristiano Premebida. Multimodal Human Detection using RGB, Thermal and LiDAR modalities for Robotic Perception. In 2024 IEEE 20th International Conference on Automation Science and Engineering (CASE), Bari, Italy.
- Kennedy O. S. Mota, Diogo S. de Oliveira, Luís Garrote, Cristiano Premebida. Multimodal Human Detection Using YOLO and Representation Learning for Robot Perception. In Seventh Iberian Robotics Conference (ROBOT 2024), Madrid, Spain. [Preview Document]