A toolkit for generating computer vision ground truth (Bounding Box, Semantic Segmentation, Depth) and integrating it with ML-Agents as observations for reinforcement learning training.Require Unity 6 render graphPerception is designed to empower Unity ML-Agents with advanced computer vision capabilities. It allows a standard Unity Camera to generate rich Ground Truth data including 2D Bounding Boxes, Semantic Segmentation masks, and Gray-scale Depth maps.Perception is specifically engineered for runtime integration with Unity ML-Agents. It wraps these ground truth generators into custom Sensor Components, allowing Agents to directly observe the segmented labeled, and depth world. The observations are serialized and sent to the external Python environment via ML-Agents API, enabling the training of complex vision-based reinforcement learning models. The Python samples is available on GitHub https://github.com/BlueFisher/Perception-for-ML-Agents-Sample/blob/master/Perception_unity_wrapper.ipynb.Perception is a fork of the now-deprecated com.unity.perception. It strips out the dataset capture (offline data collection) features to focus on lightweight, real-time ground truth generation and has been migrated to support Unity 6 render graph.Require com.unity.ml-agents >= 4.0.0 (ML-Agents Release 23)Perception CapabilitiesBounding Box Generation: Real-time calculation of 2D bounding boxes for labeled game objects (BoundingBoxPerceptionCamera).Semantic Segmentation: Pixel-perfect object classification rendered to textures (SemanticSegmentationPerceptionCamera).Depth Render: Grayscale depth map rendering (DepthRenderPerceptionCamera).ML-Agents IntegrationCustom Sensors: Ready-to-use components that attach to Agents:BoundingBoxComponent: Feeds bounding box coordinates as Buffer Sensor observations.SemanticSegmentationComponent: Feeds segmentation masks as Render Texture observations.DepthRenderComponent: Feeds depth maps as Render Texture observations.Architecture & ToolsLabeling System: Robust configuration via LabelConfig and IdLabelConfig to manage object classes and IDs.URP Support: Utilizes PerceptionRendererFeature and custom render passes for efficient ground truth generation.
