Support of a Research Project: VLM Evaluation & Domain Randomization
Evaluated open-source vision-language models for robotic perception and built domain randomization for sim-to-real transfer in Isaac Sim.
Contributed to a robot learning research codebase (loco-manipulation with vision-language-action policies in a RL-loop) in two areas, supporting Nico Bohlinger from the IAS group.
Open-source VLM evaluation. Tested open-source VLMs (Florence-2 and CLIP) for object detection, grounding, and captioning. Used synthetic MuJoCo scenes built from Google Scanned Objects. Goal: assess their fit as perception backbones for downstream policies. Built a small evaluation harness covering Florence-2's task prompts (OCR, captioning, object detection, segmentation, phrase grounding), including rendering, postprocessing, and result visualization.
Domain randomization in Isaac Sim. Implemented visual and physical randomization for a loco-manipulation environment to help close the sim-to-real gap. Randomized object and table colors/materials per episode. Randomized dome-light (HDR environment map) settings. Added an image randomizer applying GPU-side augmentations (brightness, contrast, hue, saturation, noise, blur) to rendered camera observations. Added camera extrinsics randomization (e.g. position and rotation noise on a RGB-D camera, following the VIRAL paper's parameterization) to make policies robust to camera mounting variance.