README.md

Awesome LLMs for Anomaly and OOD Detection

Tracking advancements in "Large Language Models for Anomaly and Out-of-Distribution Detection", based on our detailed survey found at Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey.

Table of Contents

• Introduction
• Taxonomy
  • LLMs for Augmentation
  • LLMs for Detection
  • LLMs for Explanation
• Citation

Introduction

Detecting anomalies or out-of-distribution (OOD) samples is critical for maintaining the reliability and trustworthiness of machine learning systems. Recently, Large Language Models (LLMs) have demonstrated their effectiveness not only in natural language processing but also in broader applications due to their advanced comprehension and generative capabilities. The integration of LLMs into anomaly and OOD detection marks a significant shift from the traditional paradigm in the field. This survey focuses on the problem of anomaly and OOD detection under the context of LLMs. We propose a new taxonomy to categorize existing approaches into three classes based on the role played by LLMs. Following our proposed taxonomy, we further discuss the related work under each of the categories and finally discuss potential challenges and directions for future research in this field.

Taxonomy

LLMs for Augmentation

Exploring how LLMs support the augmentation of detection capabilities without being direct detectors.

Paper	Authors	Backbone Model	Task Category	Dataset Type	Venue	Code
Envisioning outlier exposure by large language models for out-of-distribution detection	Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, Bo Han	GPT-3.5-turbo-16k; CLIP	OOD Detection	Images	ICML, 2024	Code
Out-of-Distribution Detection Using Peer-Class Generated by Large Language Model	K Huang, G Song, Hanwen Su, Jiyan Wang	GPT-3; CLIP	OOD Detection	Images	ArXiv, 2024	N/A
On the Powerfulness of Textual Outlier Exposure for Visual OoD Detection	Sangha Park, Jisoo Mok, Dahuin Jung, Saehyung Lee, Sungroh Yoon	BERT; BLIP-2; GPT-3; CLIP	OOD Detection	Images	NeurIPS, 2023	Code
Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Models	Albert Xu, Xiang Ren, Robin Jia	GPT-3; GPT-J; BERT	OOD Detection	Texts	ACL, 2023	Code
Tagfog: Textual anchor guidance and fake outlier generation for visual out-of-distribution detection	Jiankang Chen, Tong Zhang, Weishi Zheng, Ruixuan Wang	ChatGPT; CLIP	OOD Detection	Images	AAAI, 2024	Code
Do LLMs Understand Visual Anomalies? Uncovering LLM Capabilities in Zero-shot Anomaly Detection	Jiaqi Zhu, Shaofeng Cai, Fang Deng, Junran Wu	GPT-3.5; CLIP	Anomaly Detection	Images	ArXiv, 2024	N/A
Exploring large language models for multi-modal out-of-distribution detection	Yi Dai, Hao Lang, Kaisheng Zeng, Fei Huang, Yongbin Li	text-davinci-003; CLIP	OOD Detection	Images	EMNLP, 2023	N/A
LogGPT: Exploring ChatGPT for log-based anomaly detection	Jiaxing Qi et al.	ChatGPT	Anomaly Detection	Log Data	IEEE HPCC, 2023	Code
LogFiT: Log Anomaly Detection Using Fine-Tuned Language Models	Crispin Almodovar et al.	Various LLMs	Anomaly Detection	Log Data	IEEE TNSM 2024	N/A
How good are LLMs at out-of-distribution detection?	Andi Zhang et al.	LLaMA etc.	OOD Detection	Various	COLING, 2024	Code
Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector	Andi Zhang, Tim Z Xiao, Weiyang Liu, Robert Bamler, Damon Wischik	Various LLMs	OOD Detection	Texts	arXiv, 2024	N/A

---

LLMs for Detection

Highlighting how LLMs directly contribute to detecting anomalies and out-of-distribution samples.

Paper	Authors	Backbone Model	Task Category	Dataset Type	Venue	Code
WinCLIP: Zero-/few-shot anomaly classification and segmentation	Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, Onkar Dabeer	CLIP	Anomaly Detection	Images	CVPR, 2023	Code
CLIP-AD: A language-guided staged dual-path model for zero-shot anomaly detection	Xuhai Chen, Jiangning Zhang, Guanzhong Tian, Haoyang He, Wuhao Zhang, Yabiao Wang, Chengjie Wang, Yunsheng Wu, Yong Liu	CLIP	Anomaly Detection	Images	arXiv, 2023	N/A
Exploring grounding potential of VQA-oriented GPT-4V for zero-shot anomaly detection	Jiangning Zhang, Xuhai Chen, Zhucun Xue, Yabiao Wang, Chengjie Wang, Yong Liu	GPT-4V	Anomaly Detection	Images	arXiv, 2023	Code
CLIPScope: Enhancing Zero-Shot OOD Detection with Bayesian Scoring	Hao Fu, Naman Patel, Prashanth Krishnamurthy, Farshad Khorrami	CLIP	OOD Detection	Images	ArXiv, 2024	N/A
AnomalyCLIP: Object-agnostic prompt learning for zero-shot anomaly detection	Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, Jiming Chen	CLIP	Anomaly Detection	Images	ICLR, 2024	Code
Toward generalist anomaly detection via in-context residual learning with few-shot sample prompts	Jiawen Zhu, Guansong Pang	CLIP	Anomaly Detection	Images	CVPR, 2024	Code
PromptAD: Learning prompts with only normal samples for few-shot anomaly detection	Xiaofan Li, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yuan Xie, Lizhuang Ma	CLIP	Anomaly Detection	Images	CVPR, 2024	Code
Text prompt with normality guidance for weakly supervised video anomaly detection	Zhiwei Yang, Jing Liu, Peng Wu	CLIP	Anomaly Detection	Videos	arXiv, 2024	N/A
LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning	Miyai et al.	CLIP	OOD Detection	Images	NeurIPS, 2023	Code
CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No	Wang et al.	CLIP	OOD Detection	Images	ICCV, 2023	Code
Out-Of-Distribution Detection With Negative Prompts	Nie et al.	CLIP	OOD Detection	Images	ICLR, 2024	Code
ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection	Yichen Bai, Zongbo Han, Changqing Zhang, Bing Cao, Xiaoheng Jiang, Qinghua Hu	CLIP	OOD Detection	Images	CVPR, 2024	Code
Learning transferable negative prompts for out-of-distribution detection	Tianqi Li, Guansong Pang, Xiao Bai, Wenjun Miao, Jin Zheng	CLIP	OOD Detection	Images	CVPR, 2024	Code
AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models	Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, Jinqiao Wang	ImageBind-Huge; Vicuna-7B	Anomaly Detection	Images	ArXiv, 2024	Code
Adapting visual-language models for generalizable anomaly detection in medical images	Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xinchao Wang, Yanfeng Wang	CLIP	Anomaly Detection	Images	CVPR, 2024	Code
Adapting Contrastive Language-Image Pretrained (CLIP) Models for Out-of-Distribution Detection	Nikolas Adaloglou, Felix Michels, Tim Kaiser, Markus Kollmann	CLIP	OOD Detection	Images	TMLR, 2024	Code
Video anomaly detection and explanation via large language models	Hui Lv, Qianru Sun	Video-LLaMA	Anomaly Detection	Videos	arXiv, 2024	N/A
VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection	Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, Yanning Zhang	CLIP	Anomaly Detection	Videos	AAAI, 2023	Code
Delving into Out-of-Distribution Detection with Vision-Language Representations	Ming et al.	CLIP	OOD Detection	Images	NeurIPS, 2022	Code
Text prompt with normality guidance for weakly supervised video anomaly detection	Zhiwei Yang, Jing Liu, Peng Wu	CLIP	Anomaly Detection	Videos	arXiv, 2024	N/A
Negative Label Guided OOD Detection with Pretrained Vision-Language Models	Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, Bo Han	CLIP; ALIGN; GroupViT; AltCLIP	OOD Detection	Images	ICLR, 2024	Code
Zero-shot out-of-distribution detection based on the pretrained model CLIP	Sepideh Esmaeilpour, Bing Liu, Eric Robertson, Lei Shu	CLIP	OOD Detection	Images	AAAI, 2022	Code
Large language models can be zero-shot anomaly detectors for time series?	Sarah Alnegheimish, Linh Nguyen, Laure Berti-Equille, Kalyan Veeramachaneni	Mistral-7B-Instruct-v0.2; gpt-3.5-turbo-instruct	Anomaly Detection	Time Series	arXiv, 2024	N/A
Semantic anomaly detection with large language models	Amine Elhafsi, Rohan Sinha, Christopher Agia, Edward Schmerling, Issa Nesnas, Marco Pavone	text-davinci-003	Anomaly Detection	Videos	arXiv, 2023	N/A
Harnessing large language models for training-free video anomaly detection	Luca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, Elisa Ricci	Llama-2-13b-chat; ImageBind	Anomaly Detection	Videos	CVPR, 2024	Code
Large language models can deliver accurate and interpretable time series anomaly detection	Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, Ying-Cong Chen	GPT-4-1106-preview	Anomaly Detection	Time Series	arXiv, 2024	N/A
FiLo: Zero-shot anomaly detection by fine-grained description and high-quality localization	Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Hao Li, Ming Tang, Jinqiao Wang	CLIP	Anomaly Detection	Vision	arXiv, 2024	Code
Can LLMs Serve As Time Series Anomaly Detectors?	Manqing Dong, Hao Huang, Longbing Cao	GPT-4	Anomaly Detection	Time Series	arXiv, 2024	N/A

---

LLMs for Explanation

Detailing how LLMs aid in explaining the detection results, enhancing understanding and trust.

Paper	Authors	Backbone Model	Task Category	Dataset Type	Venue	Code
Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM	Huaxin Zhang, Xiaohao Xu, Xiang Wang, Jialong Zuo, Chuchu Han, Xiaonan Huang, Changxin Gao, Yuehuan Wang, Nong Sang	Video-LLaVA	Anomaly Detection	Videos	ArXiv, 2024	Code
Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models	Yuchen Yang, Kwonjoon Lee, Behzad Dariush, Yinzhi Cao, Shao-Yuan Lo	CogVLM-17B; GPT-4; Mistral-7B-Instruct-v0.2	Anomaly Detection	Videos	ArXiv, 2024	Code
Video Anomaly Detection and Explanation via Large Language Models	Lv et al.	LLaMA	Anomaly Detection	Videos	ICCV, 2024	N/A
Real-Time Anomaly Detection and Reactive Planning with Large Language Models	Sinha et al.	BERT, Llama 2 etc.	Anomaly Detection	Robotic Data	ArXiv, 2024	N/A

Citation

If you find this work useful, please cite our survey paper:

@article{xu2024large,
      title={Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey}, 
      author={Ruiyao Xu and Kaize Ding},
      journal={arXiv preprint arXiv:2409.01980},
      year={2024},
}