Installing CARLA from source allows full access to its underlying code and assets, enabling customization of maps, vehicle models, and other resources. It provides the flexibility to modify and extend the simulator's functionality, such as adding new features or adjusting existing ones, which cannot be done with pre-built versions.
We utilize YOLO for our experiments due to its superior performance compared to R-CNNs. YOLO (You Only Look Once) offers faster inference and efficiency by processing the entire image in a single pass, making it ideal for real-time applications
Download a pre-trained adversarial patch, crafted using the CART dataset, for attacking YOLOv5's detection of a car in the CARLA environment. We use this patch in all our experiments.
CARLA's Python API Client is an interface that enables users to interact with simulator programmatically using Python. It provides a high-level API to control various aspects of simulation, like spawning, managing or controlling actors and accessing their sensor data. This client facilitates seamless integration of custom code. Hence, the object detection must be integrated into the Python API client. This allows the vehicle to autonomously navigate the map while detecting objects in the scene.
Source: CARLA Python API Docs
Allows evaluation of the adversarial patch's impact on object detection, while blending the patch visually with the scene for realism. It enables testing autonomous driving systems in near-real-world conditions.
Material Generation and Modified Adversarial Vehicle Blueprint
The GUI serves as the primary entry point for interacting with the AVATAR framework. It seamlessly integrates with the underlying PythonAPI, automatically invoking the components with relevant parameters to start a PyGame window simulation displaying the scene in CARLA.
Typically, the PythonAPI is sufficient to interact with CARLA, eliminating the need to directly work with Unreal Engine. In our case, we modify CARLA assets (e.g., for adversarial patch attacks). This requires access to Unreal Engine asset files to specify whether an adversarial patch attack should be conducted and to select the patch image to use.
Within the AVATAR GitHub repository, there is a patches folder. Inside, there is a flag file used to signal Unreal Engine whether an attack should be performed, and a patch image file is transferred to Unreal Engine, converted into an material asset, and applied to a vehicle asset.
NOTE: Do not delete the flag file, as it is critical for attack execution. one can replace the patch image with a custom image to be used as adversarial patch in simulations.
The Adversarial Trace Classifier (ATC) is developed for post simulation analysis to robustly differentiate between original vehicle behavior and the behavior affected by adversarial patches.
If you find our contribution useful, please consider citing our work.
@article{sharma2024avatar,
title={AVATAR: Autonomous Vehicle Assessment through Testing of Adversarial Patches in Real-time},
author={Sharma, Abhijith and Narayan, Apurva and Azad, Nasser Lashgarian and Fischmeister, Sebastian and Marksteiner, Stefan},
journal={IEEE Transactions on Intelligent Vehicles},
year={2024},
publisher={IEEE}
}