Language ModelClimateBERT

ClimateBERT is the name of our transformer-based language model adapted for use for climate-related text and has been fine-tuned on various downstream tasks.

Using the DistilRoBERTa model as starting point, the ClimateBERT Language Model is additionally pretrained on a text corpus comprising climate-related research paper abstracts, corporate and general news and reports from companies. The underlying methodology can be found in our language model research paper.

Language Model Weights

The pretrained domain-adapted language models with masked language model head are publicly available on ­čĄŚ Hugging Face Hub:

  1. ClimateBERTF: huggingface.co/climatebert/distilroberta-base-climate-f
  2. ClimateBERTS: huggingface.co/climatebert/distilroberta-base-climate-s
  3. ClimateBERTD: huggingface.co/climatebert/distilroberta-base-climate-d
  4. ClimateBERTD+S: huggingface.co/climatebert/distilroberta-base-climate-d-s

BibTeX entry and citation info

@article{wkbl2021,
  title={ClimateBERT: A Pretrained Language Model for Climate-Related Text},
  author={Webersinke, Nicolas and Kraus, Mathias and Bingler, Julia and Leippold, Markus},
  journal={arXiv preprint arXiv:2110.12010},
  year={2021}
}