$ cat vlm-robotics/README.md
Research Assistant · University of Michigan · Jan 2026 – Present
Putting vision-language models to work on a real robot arm
Material classification and object detection for a UR5E cobot — where a model is only useful if it's fast, structured, and reliable downstream. In collaboration with UM Health.
Role
Research Assistant — VLM & robotics integration
Goal
VLM-driven material classification for robotic handling
Key outcome
64% micro-F1 across 10 material categories, Qwen3-VL 8B + LoRA
Context
University of Michigan · UM Health collaboration
~0.95s
per image · Qwen3-VL 8B, RTX 6000 Blackwell (bs=1, greedy, bf16)
64%
micro-F1 across 10 material categories — Qwen3-VL 8B + LoRA (UnSloth, rank 16)
9,285
image fine-tuning dataset · structured JSONL detections for reliable parsing
$ cat problem.md
Off-the-shelf VLMs are too slow and too loose for robotics
A robot arm can't wait 24 seconds per frame — which is what the 30B prototype cost on an RTX 2000 Ada — and it can't parse freeform prose. The pipeline needed detections a planner can consume: structured, low-latency, and grounded in the physical scene, with each detection expressed in robot-base-frame coordinates rather than pixels.
$ ./approach.sh
01Prototyped object detection with Qwen3-VL 30B via prompt tuning on RGB frames from the fixed wall-mounted Orbbec Gemini 335 that views the flat work surface, emitting structured JSONL for reliable downstream parsing.
02Fine-tuned Qwen3-VL 8B with LoRA (UnSloth, rank 16, 9,285 images) to classify materials across 10 categories → 64% micro-F1.
03Settled the model choice on measurement, not size: the 30B prototype ran at 24 s per frame on an RTX 2000 Ada, the LoRA-tuned 8B at 6 s per frame on the same RTX 2000 Ada at comparable accuracy — one machine, two models, so the difference is attributable to the model.
04The lab later moved to an RTX 6000 Blackwell, where the same 8B model runs at ~0.95 s per image (batch size 1, greedy decoding, bf16, base model with the adapter not merged). Different hardware — not comparable to the two figures above.
05Calibrated the fixed wall-mounted camera to the robot with a planar homography (3×3, 8 DOF) mapping pixels to base-frame X/Y on the work surface — fitted from 9 point correspondences (ArUco markers for a repeatable pixel coordinate, tool tip touched to each marker for the robot coordinate). 5.3 mm leave-one-out accuracy. The camera sits ~50° off nadir, which is why a homography is needed rather than a similarity transform; Z is a fixed hover height, never sensed.
06Collected multi-angle captures with the separate flange-mounted Orbbec on the UR5E — the arm moves through waypoints and rotates around each object, with per-frame viewpoint taken from UR5E forward kinematics and each capture keyed to the TCP coordinate.
07Prototyped AS7265x spectral sensing (Raspberry Pi, I2C) as a complementary material-classification signal.
$ cat results.json
- Model choice made on measurement: 24 s per frame (30B) vs 6 s per frame (8B + LoRA) at comparable accuracy, both on an RTX 2000 Ada
- Current runtime ~0.95 s per image — the same 8B model on an RTX 6000 Blackwell (bs=1, greedy, bf16, adapter not merged); different hardware, so not comparable to the figures above
- 64% micro-F1 on 10-way material classification from a 9,285-image LoRA fine-tune
- Detections emitted as structured JSONL — robust parsing for the robotics stack
- Pixel-to-robot mapping at 5.3 mm leave-one-out on the fixed camera; multi-angle capture keyed to TCP coordinates and spectral sensing prototyped alongside it
Fig 1 — the Orbbec Gemini 335 depth camera on its 3D-printed mount
Fig 2 — fine-tuned Qwen3-VL material detections (paper / plastic)
Fig 3 — the same frame as structured JSONL — what the robotics stack actually consumes
$ grep -r skills
vision-language modelsQwen3-VLLoRA / QLoRA · UnSlothprompt engineeringlatency/accuracy trade-offsUR5E cobotRGB-D sensingcamera-to-robot calibration · planar homographyRaspberry Pi · I2C