Language ModelClimateBERT

ClimateBERT is the name of our transformer-based language model adapted for use for climate-related text and has been fine-tuned on various downstream tasks.

Using the DistilRoBERTa model as starting point, the ClimateBERT Language Model is additionally pretrained on a text corpus comprising climate-related research paper abstracts, corporate and general news and reports from companies. The underlying methodology can be found in our language model research paper.

Language Model Weights

The pretrained domain-adapted language models with masked language model head are publicly available on ­čĄŚ Hugging Face Hub:

  1. ClimateBERTF:
  2. ClimateBERTS:
  3. ClimateBERTD:
  4. ClimateBERTD+S:

BibTeX entry and citation info

  title={ClimateBERT: A Pretrained Language Model for Climate-Related Text},
  author={Webersinke, Nicolas and Kraus, Mathias and Bingler, Julia and Leippold, Markus},
  journal={arXiv preprint arXiv:2110.12010},