Natural Language Processing (NLP) focuses on enabling computers to understand, interpret and process human language. It underpins many modern technologies, including search engines, recommendation systems, chatbots and large language models. As the amount of textual data generated online continues to grow, the ability to automatically organize and analyze text has become increasingly important.
One fundamental task in NLP is text classification. This task involves automatically assigning predefined categories or topics to textual data based on its content. Topic classification systems are widely used in applications such as news categorization, content moderation, sentiment analysis and information filtering.
Developing effective text classification models typically involves transforming raw text into numerical representations that machine learning algorithms can process. Techniques such as TF-IDF vectorization or text embeddings allow models to capture patterns in language and distinguish between different topics.
For this reason, evaluating different feature extraction methods and classification algorithms plays an important role in building robust NLP pipelines, while comparing multiple approaches helps identify which combinations of text representation and model architecture perform best for the given classification task.
This project was developed using a Notebook. Therefore if you're looking forward to test it out yourself, keep in mind to either use a Anaconda Distribution or a 3rd party software that helps you inspect and execute it.
Therefore, for more informations regarding the Virtual Environment used in Anaconda, consider checking the DEPENDENCIES.md file.
Within this assignment, text topic classification is formulated as a supervised multi-class NLP task, where the objective is to automatically classify prompts into their corresponding semantic categories.
The project is divided into two major phases. The first phase focuses on traditional machine learning approaches, exploring multiple preprocessing strategies, sparse and dense feature representations and classical classifiers such as Logistic Regression, Support Vector Machines and Random Forests. Different text representations, including TF-IDF and Word2Vec embeddings, are evaluated to understand how feature engineering impacts predictive performance.
The second phase extends the project toward modern transformer-based architectures, leveraging pre-trained language models such as DistilRoBERTa-base, BERT-base-uncased and microsoft/deberta-v3-small. These models are fine-tuned for downstream classification using the Hugging Face ecosystem and optimized through techniques such as LoRA (Low-Rank Adaptation) to improve parameter efficiency.
Additionally, the project explores more advanced NLP paradigms including domain adaptation and prompting-based inference through both zero-shot and few-shot prompting strategies. Throughout the project, particular attention is given to dataset exploration, feature representation analysis, hyperparameter optimization and extensive error analysis, allowing a deeper understanding of the strengths and limitations of each modelling strategy.
For this project, we used the Tanaos Topic Classification Training Dataset, publicly available on Hugging Face. It was created to support experiments in text topic classification, providing labeled text samples that can be used to train and evaluate machine learning models.
The data consists of synthetically generated text examples, where each sample is associated with a specific topic label. These labels represent different thematic categories, allowing models to learn how to distinguish between topics based on the content of the text.
Additionally, because the dataset is already structured and labeled, it is well suited for building and testing supervised text classification pipelines, enabling the comparison of different feature extraction approaches and classification models in order to evaluate their performance on a topic classification task.
Two major feature extraction strategies were explored during the first phase of the project.
The first approach relies on TF-IDF vectorization, which encodes the importance of words according to their frequency and discriminative value within the corpus. Uni-grams and bi-grams are used to capture both individual keywords and short contextual phrases, while dimensionality constraints help reduce noise and control feature complexity.
The second approach relies on Word2Vec embeddings, where words are mapped into dense continuous vector spaces capable of capturing semantic relationships between terms. Document representations are generated through embedding aggregation strategies, producing compact semantic representations of the prompts.
Although embeddings provide richer semantic information, the use of simple aggregation techniques such as mean pooling may compress relevant contextual details, potentially reducing discriminative power in fine-grained classification tasks.
The first phase of the project evaluates several traditional supervised learning models, including Logistic Regression, Support Vector Machines and Random Forests.
These models are evaluated under both TF-IDF and Word2Vec embedding representations. Hyperparameter optimization is additionally performed to identify the strongest classical machine learning pipeline.
The second phase of the project focuses on evaluating the impact of modern transformer architectures on the text classification task. Three pre-trained transformer models were explored DistilRoBERTa-base, BERT-base-uncased and microsoft/deberta-v3-small
All models were fine-tuned using the Hugging Face Transformers ecosystem, leveraging GPU acceleration and parameter-efficient fine-tuning through LoRA (Low-Rank Adaptation). Beyond standard supervised fine-tuning, the project also investigates:
- Domain Adaptation, where transformers are further pre-trained on domain-specific text before downstream classification.
- Prompting-based approaches, including both zero-shot and few-shot inference using instruction-based language models.
These experiments allow a comparison between traditional machine learning pipelines and modern contextual language models in terms of classification performance, semantic understanding, robustness and computational complexity.
The strongest traditional machine learning pipeline combined TF-IDF features with a hyperparameter-tuned Support Vector Machine, achieving approximately 89.9% Accuracy, 87.9% Macro F1-score and 87.9% Balanced Accuracy.
The experiments revealed that TF-IDF consistently outperformed Word2Vec embeddings, indicating that sparse lexical representations remain highly effective for structured topic classification tasks.
Among the transformer architectures, DistilRoBERTa-base achieved the strongest overall performance, reaching approximately 90.4% Accuracy, 87.4% Macro F1-score and 87.1% Balanced Accuracy.
The transformer models demonstrated stronger contextual understanding and semantic reasoning capabilities, particularly in prompts containing ambiguous or overlapping vocabulary.
However, the performance gap relative to the optimized TF-IDF + SVM pipeline remained surprisingly small, highlighting that simpler lexical approaches can still remain highly competitive in NLP classification tasks.
Due to their large size, the trained models could not be hosted directly on GitHub. Instead, they have been made available via Google Drive and can be accessed here.
After downloading, ensure that all checkpoints and training logs are placed within the directory of the Tanaos/notebooks project to guarantee proper integration and reproducibility.
Extensive error analysis revealed that most remaining misclassifications originate from semantic overlap between related domains, particularly between Science and Health, Science and Society/Culture, Lifestyle and Work/Careers and Other and Lifestyle/Hobbies
Clustering and discriminative term analysis showed that highly influential cross-domain terms such as AI, climate change, productivity, medical research, and sustainability frequently push the models toward semantically adjacent classes.
Overall, the analysis demonstrated that the remaining errors are primarily associated with fine-grained semantic ambiguity rather than a failure to capture broader topic structures.
Two main feature extraction approaches were explored: TF-IDF and Word2Vec embeddings. TF-IDF encodes word importance based on frequency and leverages uni and bi-grams to capture both individual keywords and short contextual phrases, while also controlling dimensionality and reducing noise.
In contrast, Word2Vec embeddings provide richer semantic representations by capturing relationships between words in a continuous vector space. However, the use of simple aggregation strategies (mean pooling) compresses this information into a single vector, which can lead to a loss of fine-grained contextual detail. As a result, while embeddings offer a complementary semantic perspective, they were less effective in this classification setting.
Although several data augmentation strategies were explored, including synonym replacement and back-translation, they were not included in the final pipeline due to the already sufficient dataset size and the limited expected performance gains relative to their computational cost. Regarding preprocessing, lemmatization proved more effective than stemming, as it preserves the semantic structure of the text while still reducing vocabulary complexity, ultimately supporting better generalization.
The best-performing setup overall corresponds to a hyperparameter-tuned TF-IDF + Support Vector Machine pipeline, achieving approximately 89.9% accuracy and 88.0% macro F1-score.
Interestingly, despite the strong semantic capabilities of transformer architectures, the performance difference between the optimized SVM pipeline and the best transformer model (DistilRoBERTa-base) remained relatively small, which highlights that carefully engineered sparse lexical representations can still rival substantially more complex deep learning approaches in multi-class text classification tasks.
At the same time, the transformer architectures demonstrated clear advantages in terms of semantic understanding, contextual reasoning, and robustness against vocabulary variation.
The transformer-based experiments provided valuable insight into the strengths and limitations of contextual language models for topic classification.
Models such as DistilRoBERTa-base and BERT-base-uncased demonstrated strong semantic reasoning capabilities and improved contextual understanding when handling prompts containing overlapping or ambiguous vocabulary.
Additionally, experiments involving LoRA fine-tuning, domain adaptation, and prompting-based inference highlighted different paradigms of modern NLP systems, illustrating both their strengths and their current limitations in complex multi-class classification scenarios.
The results also demonstrated that, although transformer architectures provide stronger contextual modelling, they remain sensitive to semantically overlapping domains and highly discriminative cross-domain vocabulary.
The conducted error analyses revealed that class separability is generally strong, but the models consistently struggle with semantically related domains such as science and technology or health and science.
These misclassifications are largely driven by overlapping vocabulary and shared contextual patterns across topics, particularly in prompts discussing artificial intelligence, healthcare, sustainability, climate, productivity, and societal impact.
Both the traditional pipelines and the transformer architectures were heavily influenced by prominent cross-domain keywords, which limited their ability to fully capture subtle semantic distinctions when prompts span multiple thematic areas.
Overall, the project demonstrates that both traditional machine learning pipelines and modern transformer architectures remain highly effective approaches for multi-class text classification.
The results reinforce the importance of balancing representation quality, model complexity, and task-specific characteristics when designing robust NLP systems.
While transformer models provide impressive contextual understanding and semantic modelling capabilities, the experiments also highlight that simpler and computationally efficient approaches such as TF-IDF + Linear SVM can still achieve highly competitive performance in structured NLP classification tasks.
Future work could explore larger transformer architectures, retrieval-augmented approaches, hybrid sparse+dense representations, and more advanced embedding aggregation strategies.
Additionally, incorporating explainability techniques, calibration methods, and larger real-world datasets could further improve both the interpretability and robustness of the classification pipelines. More sophisticated domain adaptation and retrieval-based contextual augmentation strategies may also help reduce the remaining semantic overlap between closely related classes.
- Author → Gonçalo Esteves, Pedro Afonseca and Rafael Pacheco
- Course → Natural Language Processing [M.IA004]
- University → Faculty of Engineering, University of Porto
README.md by Gonçalo Esteves