In recent years, Transformers have become the go-to for Natural Language Processing (NLP) and have been used in research for Computer Vision and Time Series Forecasting to make groundbreaking discoveries.
There is a clear association between the increasing number of trainable parameters in models and the progress made in these domains, and many researchers may believe this is due to the inability of transformers to adapt to small datasets.
The Compact Convolutional Transformer approach introduced in the paper "Escaping the Big Data Paradigm with Compact Transformers" addresses the issues with Transformer models on small datasets.
The CCT architecture aims to increase performance and provide flexibility for input image sizes whilst demonstrating the independence of Positional Embedding in contrast to other transformer-based architectures. The utilization of the CCT architecture empowers researchers to achieve noteworthy results despite having restricted trainable parameters, small datasets, and a single processing unit, be it a GPU or a CPU.
This notebook aims to provide a comprehensive guide on how to implement and train a CCT model that is specifically designed for classifying skin cancer. The model will accurately identify potential skin cancer cases, which will promote early intervention and improve patient outcomes.
For more information on the Compact Convolutional Transformer architectures and Transformers in general, please refer to the following links:
- CCT Paper: Escaping the Big Data Paradigm with Compact Transformers
- Vision Transformer Paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021)
- Attention Mechanism Paper: Attention Is All You Need (2017)
- V7Labs Article: Vision Transformer: What It Is & How It Works [2023 Guide]
- Viso.ai Article: Vision Transformers (ViT) in Image Recognition – 2022 Guide
The dataset used in this notebook is known as the International Skin Imaging Collaboration (ISIC) Skin Cancer Detection Dataset. The ISIC is an academia and industry partnership designed to facilitate the application of digital skin imaging to help reduce melanoma mortality. This notebook uses the Kaggle version of this dataset.
For more information check the following:
- Recommended python version: Python +3.8.10 64-bit
[Please note that this section will be updated in due course.]
.
├── LICENSE
├── README.md
└── application-of-compact-convolutional-transformers.ipynb
- LICENSE | project license (MIT)
- README.md | project readme file
- application-of-compact-convolutional-transformers.ipynb | project notebook
See application-of-compact-convolutional-transformers.ipynb
This project is licensed under the terms and conditions of the MIT license.