When we first built the news sentiment layer for MicroStocks, we tested general-purpose BERT models — the same ones that power many chatbots and review systems. The results were surprisingly poor. Phrases like "profit warning", "margin compression", and "elevated cost pressures" were being consistently misclassified. The problem wasn't the model architecture — it was the training data.
Why General NLP Fails on Financial Text
General sentiment models like VADER, TextBlob, or standard BERT are trained on consumer review data (Amazon reviews, movie ratings, Twitter posts) where positivity and negativity map predictably to everyday language. Financial text uses a completely different vocabulary where the sentiment of a phrase depends heavily on context.
Consider: "The company is highly leveraged." In a startup context, leverage can be positive — you're scaling efficiently with OPM. In a high-interest rate environment for a cyclical business, it's existential risk. A general model has no way to make this distinction. FinBERT does.
What is FinBERT?
FinBERT is a variant of BERT pre-trained on a large corpus of financial text including Reuters and Bloomberg news, financial analyst reports, SEC filings (10-K and 10-Q), and earnings call transcripts. It was originally developed by researchers at the Singapore Management University and has become the de facto standard for financial NLP tasks requiring sentiment classification.
Performance Comparison: General BERT vs FinBERT
On standard financial sentiment benchmarks (FiQA, Financial PhraseBank), FinBERT outperforms general BERT by 12–18% in F1 score for three-class sentiment classification (positive / neutral / negative). The biggest gains are in the "negative" class — where financial hedging language, passive voice, and technical jargon frequently fool general models.
How We Fine-Tune FinBERT for Indian Markets
The base FinBERT model is trained predominantly on US and global financial text. Indian market terminology — SEBI regulatory language, RBI monetary policy announcements, GST impact analysis, PLI scheme-related news — requires additional fine-tuning. We've built a supplementary training dataset from 5+ years of ET Markets, Business Standard, Moneycontrol, and NSE/BSE announcements. This specifically improves classification accuracy for:
- RBI policy impact on banking and NBFC stocks
- SEBI ASM/GSM announcements
- Budget and fiscal policy impact news
- PLI scheme approval and disbursement news
- Promoter buying/selling announcements
Limitations and Responsible Use
FinBERT, even fine-tuned, is not a market oracle. Sentiment scores are most reliable when aggregated over multiple articles and multiple days. A single article can be misleading — especially opinion pieces, paid content, or articles from partisan financial platforms. MicroStocks weights sentiment signals by source credibility (BSE announcements > ET wire > unattributed "expert" columns) to mitigate noise injection from low-quality sources.