Open-Vocabulary USAR Victim Detection and Rescue Scene Understanding

Urban Search and Rescue (USAR) operations require rapid and reliable identification of victims in complex disaster environments. Earthquakes, structural collapses, explosions, and other emergencies can create highly cluttered scenes in which people may be partially occluded, trapped beneath debris, or visible only through small portions of the body. Conventional object-detection models are often limited to a predefined set of classes and may therefore struggle to recognize unusual victim conditions that were not explicitly represented during training.

Open-Vocabulary USAR Victim Detection and Rescue Scene Understanding explores an AI-assisted approach that combines open-vocabulary object detection, deep-learning-based victim localization, and Vision-Language Models (VLMs) to provide richer situational awareness for rescue teams. Instead of detecting only a generic person class, the system can use semantic prompts such as “trapped person,” “partially buried victim,” “injured person,” “person under debris,” or “visible human body part.”

The proposed framework integrates visual detection with rescue-scene interpretation. Detected victims are localized using bounding boxes and confidence scores, while the vision-language component analyzes the broader environment to identify rubble, structural damage, obstacles, possible access routes, and operational hazards. This information can then be organized into a structured USAR report to support first responders' decision-making. A typical processing pipeline includes:

• Image acquisition from UAVs, mobile robots, fixed cameras, or rescue personnel.

• Image preprocessing for efficient inference in resource-constrained or real-time environments.

• Victim detection using a dedicated deep-learning detector such as a SimAM-FPN Faster R-CNN architecture.

• Open-vocabulary detection to search for victim conditions that are not restricted to predefined labels.

• IoU and confidence evaluation for assessing localization quality.

• Vision-language scene understanding for describing debris, hazards, accessibility, and victim context.

• Structured USAR reporting to present victim observations and operationally relevant information to rescue personnel.

The key advantage of an open-vocabulary approach is its ability to extend victim recognition beyond the categories available in the original training dataset. Rescue operators can dynamically introduce textual concepts according to the current mission, allowing the perception system to adapt to different disaster scenarios without retraining the entire detector. For UAV-assisted rescue missions, this capability can be particularly valuable. A drone can inspect areas that may be dangerous or inaccessible to human responders while the AI system highlights potential victims and generates contextual information about their surroundings. Rather than replacing rescue specialists, the system serves as a decision-support tool, helping teams prioritize areas requiring further inspection.

Ultimately, Open-Vocabulary USAR Victim Detection and Rescue Scene Understanding represents a step toward more flexible and context-aware AI systems for disaster response. By combining conventional victim detection, semantic open-vocabulary recognition, and vision-language reasoning, the framework aims to transform raw imagery into actionable information to support faster, better-informed USAR operations.

---