Welcome to the ultimate list of resources for AI in cybersecurity. This repository aims to provide an organized collection of high-quality resources to help professionals, researchers, and enthusiasts stay updated and advance their knowledge in the field.
Inspired by awesome-security, Awesome-AI-for-cybersecurity !
AI applications in cybersecurity can be categorized using Gartner’s PPDR model:
- Prediction
- Prevention
- Detection
- Response
- Monitoring
Additionally, AI applications can be divided by technical layers:
- Network (network traffic analysis and intrusion detection)
- Endpoint (anti-malware)
- Application (WAF or database firewalls)
- User (UBA)
- Process behavior (anti-fraud)
- DeepExploit - DeepExploit
- Fully automated penetration testing framework using Machine Learning. It uses reinforcement learning to improve its attack strategies over time.
- open-appsec - open-appsec is an open source machine-learning security engine that preemptively and automatically prevents threats against Web Application & APIs.
- OpenVAS - OpenVAS
- An open-source vulnerability scanner and vulnerability management solution. AI can be used to improve the identification and prioritization of vulnerabilities based on their potential impact and likelihood of exploitation.
- SEMA - SEMA
- ToolChain using Symbolic Execution for Malware Analysis. SEMA provides a framework for symbolic execution to extract execution traces and build system call dependency graphs (SCDGs). These graphs are used for malware classification and analysis, enabling the detection of malware based on symbolic execution and machine learning techniques.
- [python] Malware environment for OpenAI Gym: Create an AI that learns through reinforcement learning which functionality-preserving transformations to make on a malware sample to break through / bypass machine learning static-analysis malware detection.
- Snort IDS - Snort IDS
- An open-source network IDS and IPS capable of real-time traffic analysis and packet logging. Snort can leverage AI for anomaly detection and to enhance its pattern matching algorithms for better intrusion detection.
- PANTHER - PANTHER
- PANTHER combines advanced techniques in network protocol verification, integrating the Shadow network simulator with the Ivy formal verification tool. This framework allows for detailed examination of time properties in network protocols and identifies real-world implementation errors. It supports multiple protocols and can simulate advanced persistent threats (APTs) in network protocols.
- OSSEC - OSSEC
- An open-source host-based intrusion detection system (HIDS). AI can enhance OSSEC by providing advanced anomaly detection and predictive analysis to identify potential threats before they materialize.
- Zeek Network Security Monitor - Zeek GitHub
- A powerful network analysis framework focused on security monitoring. AI can be integrated to analyze network traffic patterns and detect anomalies indicative of security threats.
- AIEngine - AIEngine GitHub
- Next-generation interactive/programmable packet inspection engine with IDS functionality. AIEngine uses machine learning to improve packet inspection and anomaly detection, adapting to new threats over time.
- Sophos Intercept X - Sophos Intercept X
- Advanced endpoint protection combining traditional signature-based detection with AI-powered behavioral analysis to detect and prevent malware and ransomware attacks.
- MARK - Easy BigData Ranking - MARK
- The multi-agent ranking framework (MARK) aims to provide all the building blocks required to build large-scale detection and ranking systems. It includes distributed storage suited for BigData applications, a web-based visualization and management interface, a distributed execution framework for detection algorithms, and an easy-to-configure triggering mechanism. This allows data scientists to focus on developing effective detection algorithms.
- Metasploit Framework - Metasploit
- A tool for developing and executing exploit code against a remote target machine. AI can be used to automate the selection of exploits and optimize the attack vectors based on target vulnerabilities.
- PentestGPT - PentestGPT
- PentestGPT provides advanced AI and integrated tools to help security teams conduct comprehensive penetration tests effortlessly. Scan, exploit, and analyze web applications, networks, and cloud environments with ease and precision, without needing expert skills.
- Cortex - Cortex
- A powerful and flexible observable analysis and active response engine. AI can be used in Cortex to automate the analysis of observables and enhance threat detection capabilities.
- Nmap - Nmap
- A free and open-source network scanner used to discover hosts and services on a computer network. AI can enhance Nmap's capabilities by automating the analysis of scan results and suggesting potential security weaknesses.
- Burp Suite - Burp Suite
- A leading range of cybersecurity tools, brought to you by PortSwigger. Burp Suite can integrate AI to automate vulnerability detection and improve the efficiency of web application security testing.
- Nikto - Nikto
- An open-source web server scanner which performs comprehensive tests against web servers for multiple items. AI can help Nikto by automating the identification of complex vulnerabilities and enhancing detection accuracy.
- MISP - MISP
- Open source threat intelligence platform for gathering, sharing, storing, and correlating Indicators of Compromise (IoCs). AI can enhance the efficiency of threat detection and response by automating data analysis and correlation.
- Scammer-List - A free open source AI based Scam and Spam Finder with a free API
- [article] Review: machine learning techniques applied to cybersecurity
- [article] Cybersecurity data science: an overview from machine learning perspective
- [article] Machine learning approaches to IoT security: A systematic literature review
- [sequence] AI infosec: first strikes, zero-day markets, hardware supply chains, adoption barriers
- [post] AI Safety in a World of Vulnerable Machine Learning Systems
- IBM Cybersecurity Analyst Professional Certificate - IBM Cybersecurity Analyst
- Get ready to launch your career in cybersecurity. Build job-ready skills for an in-demand role in the field, no degree or prior experience required.
- NIST AI RMF - NIST AI RMF
- A framework for managing risks associated with AI in SaaS. It provides guidelines on how to implement AI securely, focusing on risk assessment, mitigation, and governance.
- Microsoft AI Security - Microsoft AI Security
- Case studies on securing AI applications in SaaS environments. These case studies demonstrate how AI can be used to enhance security and protect against evolving threats.
- Google AI Security Practices - Google AI Security
- Insights and case studies from Google on how to secure AI applications in the cloud.
- IBM Watson for Cybersecurity - IBM Watson
- Tools and solutions for securing AI applications. Watson uses AI to analyze vast amounts of security data and identify potential threats, providing actionable insights for cybersecurity professionals.
- Azure Security Center - Azure Security Center
- Comprehensive security management system for cloud environments. AI and machine learning are used to identify threats and vulnerabilities in real-time.
Machine learning in network security focuses on Network Traffic Analytics (NTA) to analyze traffic and detect anomalies and attacks.
Examples of ML techniques:
- Regression to predict network packet parameters and compare them with normal values.
- Classification to identify different classes of network attacks.
- Clustering for forensic analysis.
Research Papers:
- Machine Learning Techniques for Intrusion Detection: A comprehensive survey on various ML techniques used for intrusion detection.
- A Survey of Network Anomaly Detection Techniques: Discusses various techniques and methods for detecting anomalies in network traffic.
- Shallow and Deep Networks Intrusion Detection System: A Taxonomy and Survey: A taxonomy and survey of shallow and deep learning techniques for intrusion detection.
- A Taxonomy and Survey of Intrusion Detection System Design Techniques, Network Threats and Datasets: An in-depth review of IDS design techniques and relevant datasets.
Machine learning applications for endpoint protection can vary depending on the type of endpoint.
Common tasks:
- Regression to predict the next system call for executable processes.
- Classification to categorize programs into malware, spyware, or ransomware.
- Clustering for malware detection on secure email gateways.
Research Papers:
- Deep Learning at the Shallow End: Malware Classification for Non-Domain Experts: Discusses deep learning techniques for malware classification.
- Malware Detection by Eating a Whole EXE: Presents a method for detecting malware by analyzing entire executable files.
Machine learning can be applied to secure web applications, databases, ERP systems, and SaaS applications.
Examples:
- Regression to detect anomalies in HTTP requests.
- Classification to identify known attack types.
- Clustering user activity to detect DDOS attacks.
Research Papers:
- Adaptively Detecting Malicious Queries in Web Attacks: Proposes methods for detecting malicious web queries.
User behavior analysis involves detecting anomalies in user actions, which is often an unsupervised learning problem.
Tasks:
- Regression to detect anomalies in user actions.
- Classification for peer-group analysis.
- Clustering to identify outlier user groups.
Research Papers:
- Detecting Anomalous User Behavior Using an Extended Isolation Forest Algorithm: Discusses an extended isolation forest algorithm for detecting anomalous user behavior.
Process behavior monitoring involves detecting anomalies in business processes to identify fraud.
Tasks:
- Regression to predict user actions and detect outliers.
- Classification to identify known fraud types.
- Clustering to compare business processes and detect outliers.
Research Papers:
- A Survey of Credit Card Fraud Detection Techniques: A survey on various techniques for credit card fraud detection.
- Anomaly Detection in Industrial Control Systems Using CNNs: Discusses the use of convolutional neural networks for anomaly detection in industrial control systems.
IDS/IPS systems detect and prevent malicious network activities using machine learning to reduce false positives and improve accuracy.
Research Papers:
- Next-Generation Intrusion Detection Systems: Discusses advancements in intrusion detection systems.
- AI for Cybersecurity by Cylance (2017): An introduction to AI for cybersecurity by Cylance.
- Machine Learning and Security: Discusses the application of machine learning in security.
- Mastering Machine Learning for Penetration Testing: A guide on using machine learning for penetration testing.
- Malware Data Science: Covers data science techniques for malware analysis.
- AI for Cybersecurity: A Handbook of Use Cases: A handbook on various use cases of AI in cybersecurity.
- Deep Learning Algorithms for Cybersecurity Applications: A Technological and Status Review: Reviews the state of deep learning algorithms in cybersecurity applications.
- Machine Learning and Cybersecurity: Hype and Reality: Discusses the real-world applications and limitations of machine learning in cybersecurity.
- Malware Env for OpenAI Gym - Malware Env for OpenAI Gym: A platform for creating agents that manipulate PE files to achieve objectives like bypassing antivirus software.
- Deep-pwning - Deep-pwning: A lightweight framework for evaluating machine learning model robustness against adversarial attacks.
- Counterfit - Counterfit: An automation layer for assessing the security of machine learning systems.
- DeepFool - DeepFool: A method to fool deep neural networks.
- garak - garak: A security probing tool for large language models (LLMs).
- Snaike-MLFlow - Snaike-MLflow: A suite of red team tools for MLflow.
- HackGPT - HackGPT: A tool leveraging ChatGPT for hacking purposes.
- HackingBuddyGPT - HackingBuddyGPT: An automated penetration tester.
- Charcuterie - Charcuterie: Code execution techniques for machine learning libraries.
- Exploring the Space of Adversarial Images - Exploring the Space of Adversarial Images: A tool to experiment with adversarial images.
- Adversarial Machine Learning Library (Ad-lib) - Adversarial Machine Learning Library (Ad-lib): A game-theoretic library for adversarial machine learning.
- EasyEdit - EasyEdit: A tool to modify the ground truths of large language models (LLMs).
- BadDiffusion - BadDiffusion: Official repository to reproduce the paper "How to Backdoor Diffusion Models?" published at CVPR 2023.
- PrivacyRaven - PrivacyRaven: A privacy testing library for deep learning systems.
- Guardrail.ai - Guardrail.ai: A Python package to add structure, type, and quality guarantees to the outputs of large language models (LLMs).
- ProtectAI's model scanner - ProtectAI's model scanner: A security scanner for detecting suspicious actions in serialized ML models.
- rebuff - rebuff: A prompt injection detector.
- langkit - langkit: A toolkit for monitoring language models and detecting attacks.
- StringSifter - StringSifter: A tool that ranks strings based on their relevance for malware analysis.
- Python Differential Privacy Library - Python Differential Privacy Library: A library for implementing differential privacy.
- Diffprivlib - Diffprivlib: IBM's differential privacy library.
- PLOT4ai - PLOT4ai: A threat modeling library for building responsible AI.
- TenSEAL - TenSEAL: A library for performing homomorphic encryption operations on tensors.
- SyMPC - SyMPC: A secure multiparty computation library.
- PyVertical - PyVertical: Privacy-preserving vertical federated learning.
- Cloaked AI - Cloaked AI: Open source property-preserving encryption for vector embeddings.
- MLSecOps podcast - MLSecOps podcast: A podcast dedicated to the intersection of machine learning and security operations.
- OWASP ML TOP 10 - OWASP ML TOP 10: The top 10 machine learning security risks identified by OWASP.
- OWASP LLM TOP 10 - OWASP LLM TOP 10: The top 10 security risks for large language models as identified by OWASP.
- OWASP AI Security and Privacy Guide - OWASP AI Security and Privacy Guide: A guide to securing AI systems and ensuring privacy.
- OWASP WrongSecrets LLM exercise - OWASP WrongSecrets LLM exercise: An exercise for testing AI model security.
- NIST AIRC - NIST AIRC: NIST Trustworthy & Responsible AI Resource Center.
- ENISA Multilayer Framework for Good Cybersecurity Practices for AI - ENISA Multilayer Framework for Good Cybersecurity Practices for AI: A framework for good cybersecurity practices in AI.
- The MLSecOps Top 10 by Institute for Ethical AI & Machine Learning - The MLSecOps Top 10: Top 10 security practices for machine learning operations.
- High Dimensional Spaces, Deep Learning and Adversarial Examples: Discusses the challenges of adversarial examples in high-dimensional spaces.
- Adversarial Task Allocation: Explores adversarial task allocation in machine learning systems.
- Robust Physical-World Attacks on Deep Learning Models: Examines physical-world attacks on deep learning models.
- The Space of Transferable Adversarial Examples: Discusses transferable adversarial examples in deep learning.
- RHMD: Evasion-Resilient Hardware Malware Detectors: Explores hardware-based malware detectors resilient to evasion.
- Generic Black-Box End-to-End Attack against RNNs and Other API Calls Based Malware Classifiers: Investigates black-box attacks on RNNs and malware classifiers.
- Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks: Examines policy induction attacks on deep reinforcement learning models.
- Can you fool AI with adversarial examples on a visual Turing test?: Tests the robustness of AI models using a visual Turing test.
- Explaining and Harnessing Adversarial Examples: A foundational paper on adversarial examples in machine learning.
- Delving into Adversarial Attacks on Deep Policies: Analyzes adversarial attacks on deep policies.
- Crafting Adversarial Input Sequences for Recurrent Neural Networks: Discusses adversarial attacks on RNNs.
- Practical Black-Box Attacks against Machine Learning: Explores practical black-box attacks on machine learning models.
- Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN: Uses GANs to generate adversarial malware examples.
- Data Driven Exploratory Attacks on Black Box Classifiers in Adversarial Domains: Explores data-driven attacks on black-box classifiers.
- Fast Feature Fool: A Data-Independent Approach to Universal Adversarial Perturbations: Proposes a method for creating universal adversarial perturbations.
- Simple Black-Box Adversarial Perturbations for Deep Networks: Discusses simple methods for black-box adversarial perturbations.
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning: A retrospective on the evolution of adversarial machine learning.
- One Pixel Attack for Fooling Deep Neural Networks: Demonstrates how a single-pixel modification can fool deep neural networks.
- FedMLSecurity: A Benchmark for Attacks and Defenses in Federated Learning and LLMs: A benchmark for evaluating the security of federated learning and LLMs.
- Jailbroken: How Does LLM Safety Training Fail?: Analyzes the failure modes of LLM safety training.
- Bad Characters: Imperceptible NLP Attacks: Discusses imperceptible adversarial attacks on NLP models.
- Universal and Transferable Adversarial Attacks on Aligned Language Models: Explores universal adversarial attacks on language models.
- Exploring the Vulnerability of Natural Language Processing Models via Universal Adversarial Texts: Investigates the vulnerability of NLP models to adversarial texts.
- Adversarial Examples Are Not Bugs, They Are Features: Argues that adversarial examples are inherent features of models.
- Adversarial Attacks on Tables with Entity Swap: Discusses adversarial attacks on tabular data.
- Here Comes the AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications: Explores zero-click worms targeting AI-powered applications.
- Stealing Machine Learning Models via Prediction APIs: Discusses methods for extracting machine learning models via prediction APIs.
- On the Risks of Stealing the Decoding Algorithms of Language Models: Investigates the risks of extracting decoding algorithms from language models.
- Adversarial Demonstration Attacks on Large Language Models: Explores evasion attacks on large language models.
- Looking at the Bag is not Enough to Find the Bomb: An Evasion of Structural Methods for Malicious PDF Files Detection: Discusses evasion of PDF malware detection methods.
- Adversarial Generative Nets: Neural Network Attacks on State-of-the-Art Face Recognition: Investigates adversarial attacks on face recognition models.
- Query Strategies for Evading Convex-Inducing Classifiers: Discusses query strategies for evading convex-inducing classifiers.
- Adversarial Prompting for Black Box Foundation Models: Explores adversarial prompting for foundation models.
- Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers: Case study on evading PDF malware classifiers.
- Generic Black-Box End-to-End Attack against RNNs and Other API Calls Based Malware Classifiers: Investigates black-box attacks on RNNs and malware classifiers.
- Fast Feature Fool: A Data-Independent Approach to Universal Adversarial Perturbations: Proposes a method for creating universal adversarial perturbations.
- GPTs Don’t Keep Secrets: Searching for Backdoor Watermark Triggers in Autoregressive Language Models: Investigates backdoor triggers in autoregressive language models.
- Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models: Discusses backdoor vulnerabilities in instruction-tuned language models.
- BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT: Explores backdoor attacks on ChatGPT.
- Towards Poisoning of Deep Learning Algorithms with Back-Gradient Optimization: Proposes back-gradient optimization for poisoning deep learning algorithms.
- Efficient Label Contamination Attacks Against Black-Box Learning Models: Discusses efficient label contamination attacks on black-box models.
- Text-to-Image Diffusion Models Can be Easily Backdoored through Multimodal Data Poisoning: Explores backdooring diffusion models through data poisoning.
- UOR: Universal Backdoor Attacks on Pre-Trained Language Models: Discusses universal backdoor attacks on language models.
- Analyzing And Editing Inner Mechanisms of Backdoored Language Models: Investigates the inner mechanisms of backdoored language models.
- Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models: Discusses backdoor vulnerabilities in instruction-tuned language models.
- How to Backdoor Diffusion Models?: Explores methods for backdooring diffusion models.
- On the Exploitability of Instruction Tuning: Discusses the exploitability of instruction tuning.
- Defending against Insertion-based Textual Backdoor Attacks via Attribution: Proposes defenses against textual backdoor attacks.
- A Gradient Control Method for Backdoor Attacks on Parameter-Efficient Tuning: Discusses gradient control methods for backdoor attacks.
- BadNL: Backdoor Attacks Against NLP Models with Semantic-Preserving Improvements: Explores semantic-preserving backdoor attacks on NLP models.
- Be Careful About Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models: Discusses the vulnerability of word embeddings to poisoning.
- BadPrompt: Backdoor Attacks on Continuous Prompts: Investigates backdoor attacks on continuous prompts.
- Extracting Training Data from Diffusion Models: Discusses the extraction of training data from diffusion models.
- Prompt Stealing Attacks Against Text-to-Image Generation Models: Explores prompt stealing attacks on text-to-image generation models.
- Are Diffusion Models Vulnerable to Membership Inference Attacks?: Investigates the vulnerability of diffusion models to membership inference attacks.
- Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures: Discusses model inversion attacks and countermeasures.
- Multi-Step Jailbreaking Privacy Attacks on ChatGPT: Explores multi-step jailbreaking privacy attacks on ChatGPT.
- Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models: Discusses differentially private prompt learning for language models.
- ProPILE: Probing Privacy Leakage in Large Language Models: Investigates privacy leakage in large language models.
- Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence: Discusses embedding inversion attacks on sentence embeddings.
- Text Embeddings Reveal (Almost) As Much As Text: Explores the information leakage of text embeddings.
- Vec2Face: Unveil Human Faces from Their Blackbox Features in Face Recognition: Discusses the reconstruction of human faces from face recognition features.
- Realistic Face Reconstruction from Deep Embeddings: Explores face reconstruction from deep embeddings.
- DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection: Discusses backdoor attacks on deep learning models through neural payload injection.
- Black Box Adversarial Prompting for Foundation Models: Explores black-box adversarial prompting for foundation models.
- Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection: Discusses indirect prompt injection attacks on LLM-integrated applications.
- Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models: Proposes a benchmark for evaluating the safety and robustness of large language models.
- Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots: Discusses automated jailbreak attacks on multiple large language model chatbots.
- (Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs: Explores indirect instruction injection using images and sounds in multi-modal LLMs.
- Summoning Demons: The Pursuit of Exploitable Bugs in Machine Learning: Discusses the pursuit of exploitable bugs in machine learning.
- Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers: Case study on evading PDF malware classifiers.
- capAI - A Procedure for Conducting Conformity Assessment of AI Systems in Line with the EU Artificial Intelligence Act: Proposes a procedure for AI system conformity assessment.
- A Study on Robustness and Reliability of Large Language Model Code Generation: Investigates the robustness and reliability of LLM code generation.
- Getting pwn'd by AI: Penetration Testing with Large Language Models: Explores penetration testing with large language models.
- Evaluating LLMs for Privilege-Escalation Scenarios: Evaluates LLMs for privilege-escalation scenarios.
Your contributions are always welcome! Feel free to submit a pull request with your suggestions.