Weimin Lyu

Weimin Lyu

Ph.D. Candidate

About Me

Hi! I am currently an Applied Scientist at Amazon Search, where I focus on leveraging Large Language Models (LLMs) to enhance the shopping experience. I completed my PhD Study in Computer Science at Stony Brook University, advised by Prof. Chao Chen. I have also been fortunate to collaborate with Professors Haibin Ling, Fusheng Wang, and Tengfei Ma.

Research Interests

My research centers on how training objectives shape the internal behaviors of LLMs and VLMs, and how to make those behaviors controllable, efficient, and safe:

  • Training Dynamics & Alignment: Instruction/post-training for representation alignment and safe controllability in LLMs, ViTs, and VLMs; connecting training objectives with model reliability and behavior shaping.
  • Efficient Multimodal Learning: Token compression and resource-efficient model design for whole slide pathology image analysis using vision-language models like LLaVA.
  • Model Explainability in Clinical AI: Interpretable modeling of electronic health records using transformers and multimodal fusion.
  • LLM-based Personalization: Customer understanding for search/navigation, including personalization data generation, signal compression/memorization, post-training and evaluation.

News

  • 2025-08: Successfully defended my PhD thesis!
  • 2025-06: I’m honored to receive the prestigious Catacosinos Fellowship, awarded annually to outstanding PhD students at Stony Brook University - only three winners this year!
  • 2025-05: Two papers accepted to ACL 2025! Congrats to all collaborators! CalD, led by Chenlu and Prof. Ritwik Banerjee, proposes an efficient framework for detecting deviant or nuanced language using smaller models. RPA Evaluation, led by Chaoran, Bingsheng and Prof. Dakuo Wang, presents a comprehensive guideline for evaluating LLM-based role-playing agents.
  • 2025-01: Three papers are accepted by ICLR 2025, including one first-authored paper: VLOOD!
  • 2024-07: My first-authored paper, TrojVLM, is accepted by ECCV 2024! We investigate the vulnerabilities in the generative capabilities of Vision-Language Models, with a focus on image captioning and visual question answering (VQA) tasks.
  • 2024-07: One paper is accepted by WACV 2025!
  • 2024-06: My first-authored paper, BadCLM, is nominated as the Best Student Paper by AMIA 2024! We investigate the clinical language model’s vulnerabilities.
  • 2024-03: One first-authored paper is accepted by NAACL 2024! We introduce a task-agnostic method for detecting textual backdoors, targeting a range of language models and traditional NLP tasks.
  • 2023-10: My first-authored TAL is accepted by EMNLP 2023!
  • 2023-03: Two papers are accepted by ICLR 2023 Workshop on BANDS!
  • 2022-10: Paper “An Integrated LSTM-HeteroRGNN Model for Interpretable Opioid Overdose Risk Prediction” is accepted by Artificial Intelligence in Medicine!
  • 2022-06: One first-authored paper is nominated as the Best Student Paper by AMIA 2022! We propose a multimodal transformer to fuse clinical notes and traditional EHR data for interpretable mortality prediction. AMIA is the world’s premier meeting for research and practice of biomedical and health informatics.
  • 2022-04: One first-authored paper “A Study of the Attention Abnormality in Trojaned BERTs” is accepted by NAACL 2022!
  • 2020-09: Start my Computer Science Ph.D. journey at Stony Brook University!

Industry Experience

Amazon, Seattle, USA (May 2025 - Present)
Applied Scientist

- Building personalization-oriented LLMs to improve navigation/search relevance across Amazon’s shopping ecosystem.
- Developing post-training (instruction tuning, RL) and personalized training-data generation pipelines leveraging behavioral/contextual signals.
- Designing LLM-as-Judge evaluation for automated quality assessment, covering 55% of Search traffic.
Amazon, Seattle, USA (May 2024 - May 2025)
Applied Scientist Intern

- Built the first-gen LLM Seller Foundation Model for seller-risk automation, integrating tabular signals, text, and attribution features.
- Designed continuous pre-training (LM + attribution-level contrastive) and multi-task post-training to align representations and improve controllability.
- Deployed to production with business impact: +22% automation coverage, $184K cost reduction (NA).

Selected Publications

Full publications can be found in Google Scholar.

Conference/Workshop/Journal

ICLR2025
Efficient Whole Slide Pathology VQA via Token Compression
Weimin Lyu, Qingqiao Hu, Kehan Qi, Zhan Shi, Wentao Huang, Saumya Gupta, Chao Chen
In Submission
[ArXiv]
ACL2025
Class Distillation with Mahalanobis Contrast: An Efficient Training Paradigm for Pragmatic Language Understanding Tasks
Chenlu Wang, Weimin Lyu, Ritwik Banerjee
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)
[ACL]
ACL2025
Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents
Chaoran Chen, Bingsheng Yao, Ruishi Zou, Wenyue Hua, Weimin Lyu, Toby Jia-Jun Li, Dakuo Wang
Findings of the Association for Computational Linguistics: ACL 2025 (ACL 2025)
[ACL]
Geometry of Long-Tailed Representation Learning: Rebalancing Features for Skewed Distributions
Lingjie Yi, Jiachen Yao, Weimin Lyu, Haibin Ling, Raphael Douady, Chao Chen
The Thirteenth International Conference on Learning Representations (ICLR 2025)
[ICLR]
ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Language
Yuxin Wang, Xiaomeng Zhu*, Weimin Lyu*, Saeed Hassanpour, Soroush Vosoughi
The Thirteenth International Conference on Learning Representations (ICLR 2025)(Spotlight)
[ICLR]
Backdooring Vision-Language Models with Out-Of-Distribution Data
Weimin Lyu, Jiachen Yao, Saumya Gupta, Lu Pang, Tao Sun, Lingjie Yi, Lijie Hu, Haibin Ling, Chao Chen
The Thirteenth International Conference on Learning Representations (ICLR 2025)
[ICLR]
PivotAlign: Improve Semi-Supervised Learning by Learning Intra-Class Heterogeneity and Aligning with Pivots
Lingjie Yi, Tao Sun, Yikai Zhang, Songzhu Zheng, Weimin Lyu, Haibin Ling, Chao Chen
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2025)
[WACV]
ECCV2024
TrojVLM: Backdoor Attack Against Vision Language Models
Weimin Lyu, Lu Pang, Tengfei Ma, Haibin Ling, Chao Chen
The 18th European Conference on Computer Vision (ECCV 2024)
[ECCV]
BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Records
Weimin Lyu, Zexin Bi, Fusheng Wang, Chao Chen
American Medical Informatics Association Annual Symposium (AMIA 2024) (Best Student Paper Finalist)
[arXiv]
Task-Agnostic Detector for Insertion-Based Backdoor Attacks
Weimin Lyu, Xiao Lin, Songzhu Zheng, Lu Pang, Haibin Ling, Susmit Jha, Chao Chen
The Findings of 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024)
[NAACL]
Attention-Enhancing Backdoor Attacks Against BERT-based Models
Weimin Lyu, Songzhu Zheng, Lu Pang, Haibin Ling, Chao Chen
The Findings of 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) (A short version is accepted as Oral at ICLR 2023 Workshop on BANDS)
[EMNLP][Code]
An Integrated LSTM-HeteroRGNN Model for Interpretable Opioid Overdose Risk Prediction
Xinyu Dong, Rachel Wong, Weimin Lyu, Kayley Abell-Hart, Janos G Hajagos, Richard N Rosenthal, Chao Chen, Fusheng Wang
Artificial Intelligence in Medicine (AIIM 2022)
[AIIM]
A Multimodal Transformer: Fusing Clinical Notes With Structured EHR Data for Interpretable In-Hospital Mortality Prediction
Weimin Lyu, Xinyu Dong, Rachel Wong, Songzhu Zheng , Kayley Abell-Hart, Fusheng Wang, Chao Chen
American Medical Informatics Association Annual Symposium (AMIA 2022) (Best Student Paper Finalist)
[AMIA][Code]
A Study of the Attention Abnormality in Trojaned BERTs
Weimin Lyu, Songzhu Zheng, Tengfei Ma, Chao Chen
The 2022 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL 2022)(Oral)
[NAACL][Code]