Dataset creation and benchmarking for Kashmiri news snippet classification using fine-tuned transformer and LLM models in a low resource setting
Unlocking Kashmiri NLP: A Landmark Dataset and Benchmarking Study
This research addresses the critical scarcity of linguistic resources for Kashmiri, a low-resource Indo-Aryan language. It introduces the first manually curated and labeled dataset of 15,036 Kashmiri news snippets across 10 diverse domains (Medical, Politics, Sports, Tourism, Education, Art and Craft, Environment, Entertainment, Technology, and Culture). The study benchmarks various ML, DL, transformer, and LLM models for multi-class news snippet classification. Fine-tuned ParsBERT-Uncased emerged as the top-performing transformer, achieving an F1-score of 0.98, significantly advancing Kashmiri NLP. This foundational work provides a robust dataset and effective methodologies, paving the way for future advancements in digital inclusion for Kashmiri speakers.
Executive Impact: Key Metrics
0
News Snippets Created
0
Diverse Categories
0
Best F1 Score Achieved
0
Manually Labeled Corpus
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
The study initiated by manually creating a labeled dataset of 15,036 Kashmiri news snippets. English news snippets were collected from various sources, translated into Kashmiri using Microsoft Bing Translator, and then meticulously refined by native speakers and language experts to ensure linguistic accuracy and cultural authenticity across ten diverse domains. This rigorous process addressed the scarcity of standardized linguistic materials for Kashmiri NLP.
Various machine learning, deep learning, transformer-based, and large language models (LLMs) were benchmarked for multi-class news snippet classification. This involved comprehensive experimentation with different embeddings (IndicBERT v2, FastText), hyperparameters, and training methodologies. The objective was to identify the most effective approaches for accurate text classification in a low-resource language context.
Transformer models, including mBERT, DistilBERT, ParsBERT, and BERT-Base-ParsBERT-Uncased, demonstrated superior performance. Fine-tuned BERT-Base-ParsBERT-Uncased achieved an F1-score of 0.98, outperforming other models. Its pre-training on linguistically similar languages (Persian, Arabic, Urdu) and task-specific fine-tuning allowed it to capture intricate linguistic features of Kashmiri effectively.
Large Language Models (LLMs) like BLOOM-560m and Flan-T5-Base were also evaluated. While Flan-T5-Base showed poor performance in zero-shot settings due to its limited Kashmiri-specific pre-training, fine-tuned BLOOM-560m achieved a competitive F1-score of 0.97. This highlights the importance of multilingual pre-training and fine-tuning for effective LLM application in low-resource languages.
0
F1-Score achieved by ParsBERT-Uncased, setting a new benchmark for Kashmiri NLP.
Enterprise Process Flow
Data Collection (English)
→
Machine Translation (Kashmiri)
→
Manual Refinement & Labeling
→
Domain Categorization
→
Dataset Publication
| Model Type | Key Strengths | Performance (F1-score) |
|---|---|---|
| Machine Learning |
|
0.95 |
| Deep Learning (GRU) |
|
0.92 |
| Transformers (ParsBERT-Uncased) |
|
0.98 |
| LLMs (Fine-tuned BLOOM-560m) |
|
0.97 |
Impact on Digital Inclusion
The creation of this comprehensive dataset and the identification of high-performing models directly contributes to digital inclusion for the Kashmiri language. By providing reliable NLP resources, this work enables the development of tools for education, content organization, and accessibility, fostering greater use and preservation of this underrepresented language. This also sets a precedent for resource generation strategies in other low-resource contexts.
Advanced ROI Calculator
Estimate the potential savings and reclaimed productivity your enterprise could achieve by implementing AI solutions tailored to document processing and data classification.
Implementation Roadmap: Strategic Phases
Our phased approach ensures a smooth and effective integration of advanced AI, aligning with your enterprise goals for maximum impact.
Phase 1: Dataset Expansion & Automation
Expand the Kashmiri dataset further and explore automated data collection methods, potentially integrating advanced OCR for historical texts.
Phase 2: Cross-Domain Transfer Learning
Investigate cross-domain transfer learning techniques to leverage knowledge from high-resource languages for improved Kashmiri NLP model performance.
Phase 3: Speech & Multimodal NLP
Extend research into speech recognition and multimodal NLP for Kashmiri, building upon the established text classification foundation.
Phase 4: Community & Developer Tools
Develop open-source tools and APIs to empower Kashmiri developers and researchers, fostering a vibrant NLP ecosystem.
Ready to Transform Your Enterprise with AI?
Schedule a personalized consultation with our AI strategists to explore how these insights can be tailored to your specific business needs and drive tangible ROI.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat