Machine Learning-Based CyberDefenses

Special Topics, TAMU, 2023

In this course, we will navigate through the applications of ML in the security field: the pros, the cons, and the future yet to come.

What to expect

  • Lots of malware analysis stuff (50% of the course).
  • Discussions about ML limits in general.
  • To be requested to map the introduced concepts to your own work.


  • Pitfalls of ML, dataset size, generalization, and more.
  • Malware detection, streams, concept drift.
  • Adversarial ML (attacks and defenses)
  • Biometrics, Authentication, and related applications.
  • Large Language Models, GPT-3, and other fun stuff.


  • Seminar presentation + competition

The competition

Let’s make a small version of the MLSEC competition, with students playing together to create attacks and defenses.

TAMU students can already enroll in CSCE689 via the Howdy! system.

Course Progress


  • Results (2023)
    • Defense
      • Syed Wall and Yasir Farrukh won the defense challenge!
    • Attack
      • Veronika Maragulova, Sidharth Baveja, Sidharth Anil, and Soumyajyoti Dutta won the attack challenge!
      • See the winner report here
  • Results (2024)
    • Defense
      • Sidharth Arivarasan, Sahil Salunkhe, Ali Ayati won the defense challenge!
    • Attack
      • Akshat Punjabi, Akshat Pandey, and Ayushri Jain won the attack challenge!


  • Public Detection Models (2023): a * Multiple Students: DockerHub - yasirali12/malwaredetector - felzek/malware-classifier - amithmkini/cyberai - yasirali12/pipeline - sidbav/689-final-submission - yasirali12/model - felzek/defender

  • Public Detection Models (2024):

    • Bhavan Dondapati, Vishal Vardhan Adepu, and Rohith Yogi Nomula
      • DockerHub: vva2/defender - Version: 1.0.2

ChatGPT Fun

  • Results:
    • Syed Rizvi and Yasir Farrukh were able tp ,ale ChatGPT to create a Python Ransomware.
      • Prompt: Prompt
      • Execution: Sandbox Execution
    • Amith KMattar, Chunkai Fu, amd Mason Jerome were able to make ChatGPT create a Dropper.
      • The Tool: Tool
      • Code Generation: Code