MEVDT: Multi-Modal Event-Based Vehicle Detection and Tracking Dataset

El Shair, Zaid A.; Rawashdeh, Samir A.

Work Description

Title: MEVDT: Multi-Modal Event-Based Vehicle Detection and Tracking Dataset Open Access Deposited

Attribute	Value
Methodology	To collect the MEVDT dataset, we used the DAVIS 240c hybrid event-based camera which captures both grayscale images at 24 FPS and asynchronous events at a microsecond resolution. The camera was stationary, positioned at two different scenes within the University of Michigan-Dearborn campus. The combined Active Pixel Sensor (APS) and Dynamic Vision Sensor (DVS) within the camera allowed for synchronous grayscale images and asynchronous high-temporal resolution event data. Data recording was managed using the ROS DVS package. For labeling, we manually annotated vehicles using the dLabel Annotation Tool, ensuring sub-pixel precision for object bounding boxes and the continuity of unique object IDs across frames. Annotations were saved initially in the COCO JSON format and later converted to various formats, including MOT Challenge and a custom format designed for synchronization with event data. The annotation process yielded ground truth labels for object detection and tracking at a frequency of 24 Hz.
Description	The MEVDT dataset was created to fill a critical gap in event-based computer vision research by supplying a high-quality, real-world labeled dataset. Intended to facilitate the development of advanced algorithms for object detection and tracking applications, MEVDT includes multi-modal traffic scene data with synchronized grayscale images and high-temporal-resolution event streams. Additionally, it provides annotations for object detection and tracking with class labels, pixel-precise bounding box coordinates, and unique object identifiers. The dataset is organized into directories containing sequences of images and event streams, comprehensive ground truth labels, fixed-duration event samples, and data indexing sets for training and testing. To access and utilize the dataset, researchers need specific software or scripts compatible with the data formats included, such as PNG for grayscale images, CSV for event stream data, AEDAT for the encoded fixed-duration event samples, and TXT for annotations. Recommended tools include standard image processing libraries for PNG files and CSV or text parsers for event data. A specialized Python script for reading AEDAT files is available at: https://github.com/Zelshair/cstr-event-vision/blob/main/scripts/data_processing/read_aedat.py, which streamlines access to the encoded event sample data.
Creator	El Shair, Zaid A. Rawashdeh, Samir A.
Creator ORCID iD	https://orcid.org/0000-0001-9518-2828 https://orcid.org/0000-0002-3473-6978
Depositor	zelshair@umich.edu
Contact information	zelshair@umich.edu
Discipline	Engineering
Keyword	Computer Vision Event-Based Vision Object Detection Object Tracking Multi-Modal Vision Dataset
Citations to related material	El Shair, Z. and Rawashdeh, S., 2024. MEVDT: Multi-Modal Event-Based Vehicle Detection and Tracking Dataset. Data In Brief (under review). El Shair, Z. and Rawashdeh, S.A., 2022. High-temporal-resolution object detection and tracking using images and events. Journal of Imaging, 8(8), p.210. El Shair, Z. and Rawashdeh, S., 2023. High-temporal-resolution event-based vehicle detection and tracking. Optical Engineering, 62(3), pp.031209-031209. El Shair, Z.A., 2024. Advancing Neuromorphic Event-Based Vision Methods for Robotic Perception Tasks (Doctoral dissertation, University of Michigan-Dearborn).
Resource type	Dataset
Last modified	06/26/2024
Published	06/25/2024
Language	English
DOI	https://doi.org/10.7302/d5k3-9150
License	http://creativecommons.org/licenses/by/4.0/

To Cite this Work:
El Shair, Z. A., Rawashdeh, S. A. (2024). MEVDT: Multi-Modal Event-Based Vehicle Detection and Tracking Dataset [Data set], University of Michigan - Deep Blue Data. https://doi.org/10.7302/d5k3-9150

Relationships


This work is not a member of any user collections.

Files (Count: 3; Size: 1.26 GB)

Title	Original Upload	Last Modified	File Size	Access	Actions
MEVDT.zip	2024-06-18	2024-06-18	1.26 GB	Open Access	View Details Download
Readme.txt	2024-06-25	2024-06-25	4.46 KB	Open Access	View Details Download
Dataset_Structure.txt	2024-06-25	2024-06-25	2.9 KB	Open Access	View Details Download

Date: 21 June, 2024

Dataset Title: MEVDT: Multi-Modal Event-Based Vehicle Detection and Tracking Dataset

Dataset Creators: Z.A. EL Shair & S.A. Rawashdeh

Dataset Contact: Zaid A. El Shair zelshair@umich.edu

Funding: N/A

Key Points:
- The MEVDT dataset addresses the critical need for annotated datasets in the domain of event-based computer vision, particularly for automotive applications.
- The dataset comprises 63 multi-modal traffic sequences featuring approximately 13k images, 5M of high-temporal resolution events, 10k manually-annotated object labels, and 85 unique object tracking trajectories.
- Labels for object detection and tracking, alongside multiple data formats, are included to support the development of advanced computer vision algorithms.

Research Overview:
The MEVDT dataset was developed to support advancements in event-based computer vision by providing a rich source of annotated visual data specifically targeting the automotive domain. Its primary goal is to enable the development of event-based and multi-modal algorithms focused on object detection and tracking in dynamic traffic environments, leveraging the unique capabilities of event-based sensors. This dataset includes a significant volume of labeled data essential for advancing research in a novel field where such resources are scarce.

Methodology:
Data was collected using the hybrid sensor DAVIS 240c, which combines an Active Pixel Sensor (APS) and a Dynamic Vision Sensor (DVS) within the same pixel array. The APS captures grayscale images at 24 FPS, while the DVS records pixel brightness changes (i.e., events) at microsecond resolution. The collection process took place at the University of Michigan-Dearborn campus, in two scenes under clear daylight conditions. Data recording was managed using the Robot Operating System (ROS) DVS package running on a laptop. The camera was fixed capturing moving vehicles which were manually labeled with 2D bounding boxes and unique IDs.

Files contained here:
The MEVDT dataset is organized into four directories, each with training and testing splits:
- sequences/: Holds grayscale images and sequence-long event streams. Within each scene, sequences are titled by the first sample's timestamp, including image files (.png) and events stream lists (.csv).
- labels/: Contains ground truth data for object detection and tracking computer vision tasks. The tracking_labels/ includes annotations in formats like COCO JSON, MOT Challenge, and a custom format for sequence-long object tracking; while detection_labels/ provides per-image object class and bounding box coordinates in x_min, y_min, x_max, y_max format for each event sample file in event_samples/.
- event_samples/: Consists of fixed-duration batch-sampled event files in AEDAT format, organized by sampling interval, enabling research at various temporal resolutions.
- data_splits/: Features CSV files that index the dataset splits for object detection, guiding users through the existing training and testing structure for different sampling durations.

Note: each line in the sequence-long event stream CSV files corresponds to a single event in comma-separated in the following format: ts, x, y, p.
In this format, ts denotes the event's timestamp in nanoseconds, x and y correspond to its two-dimensional pixel coordinate, and p indicates its polarity as either positive (p=1) or negative (p=0).

For more information about the dataset's structure please see Dataset_Structure.txt. Additionally, refer to the dataset manuscript for full details about this dataset.

Related publication(s):
El Shair, Z. and Rawashdeh, S., 2024. MEVDT: Multi-Modal Event-Based Vehicle Detection and Tracking Dataset. Data In Brief (under review).
El Shair, Z. and Rawashdeh, S.A., 2022. High-temporal-resolution object detection and tracking using images and events. Journal of Imaging, 8(8), p.210.
El Shair, Z. and Rawashdeh, S., 2023. High-temporal-resolution event-based vehicle detection and tracking. Optical Engineering, 62(3), pp.031209-031209.
El Shair, Z.A., 2024. Advancing Neuromorphic Event-Based Vision Methods for Robotic Perception Tasks (Doctoral dissertation, University of Michigan-Dearborn).

Use and Access:
The MEVDT dataset is made available under Creative Commons Attribution 4.0 International (CC BY 4.0) and is freely available for research usage.

To Cite Data:
El Shair, Z.A. (2024). MEVDT: Multi-Modal Event-Based Vehicle Detection and Tracking Dataset [Data set]. University of Michigan - Deep Blue. https://doi.org/10.7302/d5k3-9150

Update Provenance Log Entries

Download All Files (To download individual files, select them in the “Files” panel above)

Remediation of Harmful Language

The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.