List of Transformer-based Pretrained Language Models (T-PTLMs)

2 minute read

Published:

This information is from our survey paper “AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing”. In this survey paper, we have introduced a new taxonomy for transformer-based pretrained language models (T-PTLMs). Here is the list of all T-PTLMs with links for the paper and the pretrained model. For detailed information, please refer the survey paper.

General T-PTLMs

ModelPaper and and Link
GPTPaper Model
BERTPaper Model
XLNetPaper Model
RoBERTaPaper Model
ELECTRAPaper Model
ALBERTPaper Model
T5Paper Model
BARTPaper Model
PEGAUSUSPaper Model

Social Media T-PTLMs

ModelPaper and and Link
HateBERTPaper Model
RoBERTa-twitterPaper Model
RoBERTa-reviewsPaper Model
BERT-SentiXPaper Model
XLM-R-twitterPaper Model
BERTweetPaper Model
BERTweet-COVID19Paper Model
CT-BERTPaper Model

Monolingual T-PTLMs

ModelLanguagePaper and and Link
BanglaBERTBanglaPaper
IndoBERTIndonesianPaper Model
IndonesianBERTIndonesianPaper Model
IndonesianBERT-LiteIndonesianPaper Model
PhoBERTVietnamesePaper Model
RobBERTDutchPaper Model
RomanianBERTRomanianPaper Model
FlauBERTFrenchPaper Model
AraBERTArabicPaper Model
AraGPT2ArabicPaper Model
AraELECTRAArabicPaper Model
BERTjeDutchPaper Model
FinnishBERTFinnishPaper Model
ALBERTOIndonesianPaper Model
PortugueseBERTPortuguesePaper Model
BETOSpanishPaper Model
CamemBERTFrenchPaper Model
GermanBERTGermanModel
ARBERTArabicPaper Model
MARBERTArabicPaper Model
WangchanBERTaThaiPaper Model
KoreALBERTKoreanPaper Model
KoBERTKoreanModel
KoGPT2ArabicModel
PTT5PortuguesePaper Model
SweedishBERTSwedishPaper Model
SweedishALBERTSwedishPaper Model
SweedishELECTRASwedishPaper Model
HerBERTPolishPaper Model
PolishBERTPolishPaper Model
RobeCzechCzechPaper Model
BERTurkTurkishModel
TurkishELECTRATurkishModel
ConvBERTurkTurkishModel
CzechBERTCzechPaper Model
KLUE-BERTKoreanPaper Model
KLUE-RoBERTaKoreanPaper Model
ParsBERTPersianPaper Model

Multilingual T-PTLMs

ModelPaper and and Link
MuRILPaper Model
IndicBERTPaper Model
mT5Paper Model
mT6Paper Model
InfoXLMPaper Model
mBERTPaper Model
mBARTPaper Model
XLMPaper Model
XLM-RPaper Model
XLM-RXLPaper Model
XLM-RXXLPaper Model
UnicoderPaper Model
IndoBARTPaper

Domain-Specific T-PTLMs

ModelDomainPaper and and Link
FinBERTFinancePaper Model
ALeaseBERTLegalPaper Model
Legal-BERTLegalPaper Model
RoBERTa-NewsNewsPaper Model
CodeBERTProgrammingPaper Model
Code-GPTProgrammingPaper Model
Code-GPT-adaptedProgrammingPaper Model
PLBARTProgrammingPaper Model
GraphCodeBERTProgrammingPaper Model
CoTextProgrammingPaper Model
NetBERTNetworkingPaper
OAG-BERTAcademicPaper Model
SciBERTAcademicPaper Model
MathBERTAcademicPaper
TOD-BERTDialoguePaper Model
BioBERTBiomedicalPaper Model
ClinicalBERTBiomedicalPaper Model
PubMedBERTBiomedicalPaper Model

Extensions

ModelCategoryPaper and and Link
DistilBERTCompactPaper Model
TinyBERTCompactPaper Model
MobileBERTCompactPaper Model
MiniLMCompactPaper Model
CharBERTChar-basedPaper Model
CharRoBERTaChar-basedPaper Model
CharacterBERTChar-basedPaper Model
CANINETokenization-FreePaper Model
ByT5Tokenization-FreePaper Model
CharformerTokenization-FreePaper
SBERTSentence-basedPaper Model
DeBERTaEfficientPaper Model
ConvBERTEfficientPaper Model
LongformerLong SequencePaper Model
BigBirdLong SequencePaper Model

Hits