Benchmarking and enhancing multimodal LLMs on comprehensive mechanical engineering drawing understanding — with MechVL, a 4B domain-specialized VLM trained via SFT → DAPO RL self-play.
| Model | Total | |
|---|---|---|
| MechVL-4B-RL (ours) | 84.85 | best open & closed |
| GLM-4.6V | 78.91 | best closed-source |
| Gemini-3-Pro-Preview | 77.28 | |
| MechVL-4B-SFT | 76.36 | |
| GPT-5 | 75.44 |
pip install -U transformers accelerate torch
from transformers import AutoProcessor, AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained(
"XiaofengAlg/MechVL-4B-RL", dtype="bfloat16", device_map="auto")
processor = AutoProcessor.from_pretrained("XiaofengAlg/MechVL-4B-RL")
messages = [{"role": "user", "content": [
{"type": "image", "url": "drawing.png"},
{"type": "text", "text": "图纸中标注的零件总长度是多少？"},
]}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096)
print(processor.decode(out[0], skip_special_tokens=True))
This Space hosts the project page. The interactive Gradio demo
(gradio-app/ in this repo) is ready to deploy —
toggle the Space SDK to gradio under Space settings
once a Gradio runtime is available on the account, or run it locally:
git clone https://huggingface.co/spaces/XiaofengAlg/mechvqa-project cd mechvqa-project/gradio-app && pip install -r requirements.txt && python app.py