PReMind: When Personal Memory Meets Visual Hazards
AuthorsYeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon
AffiliationsDepartment of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University
PReMind tests and trains visual assistants to connect what they see with the personal context needed to give safer advice.
Key results
Total requests in the main test set.
Cases requiring a hazard response informed by a personal fact.
Joint success on the interdependency set before PReMind training.
Joint success on the interdependency set after training.
Adequacy on acute hazards before PReMind training.
Adequacy on acute hazards after PReMind training.
What the paper found
PReMind targets a failure in personalized visual assistants: safe advice can depend on both what an image shows and what the assistant remembers about the specific person. The paper introduces PReMind-Bench, with 1024 test requests and 140 interdependency cases that require combining a visible hazard with a person-specific fact. It separately scores memory retrieval, identity grounding, abstention, hazard response, and joint success. PReMind trains Qwen3.5-4B with two-phase, group-relative DAPO reinforcement learning, mixing on-demand memory calls with forced-call episodes so the model learns both when to retrieve and how to use retrieved facts. Phase II adds difficulty-based curriculum, replay of memory-use failures, and credit for scene-specific hazard actions. On the interdependency set, joint success rises from 12.9% for the base to 28.6% for PReMind; on acute hazards, response adequacy increases from 34.4% to 56.9%. The gains persist under forced-call evaluation and a Claude Sonnet 5.5 cross-check. Evaluation uses OpenAI’s GPT-6-Luna as the primary judge, and compares against systems including Anthropic’s Claude Opus 5.5. The results show that jointly training personalization and situational safety helps, while the model still over-flags some safe requests and does not establish reduced harm on interdependency cases.
Original abstract
Personalized vision-language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
Read the original paperMore in AI Benchmarks
Browse all 60 papers →Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Bowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell Huang
A new benchmark tests whether coding agents can make whole scientific programs faster without breaking them.
DrawScore: Evaluating Generated CAD Models against Engineering Drawings
Haorui Chen, Shiteng Lu, Zhanyu Guo, Zaixi Li, Yunhe Ni, Yaosong Du, Yichen Xiang, Xiande Huang, Rui Huang
DrawScore checks generated CAD models through engineering-style views, making small but manufacturing-critical mistakes harder to overlook.
AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
Qiming Shi, Yikun Li, Beidi Luan, Rui Sun, Sinuo Wang, Delin Wang, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai
AutoVend tests whether LLM agents can run vending-machine businesses profitably over six months—and finds that most struggle to beat a simple inventory baseline.