@inproceedings{thrush-etal-2022-dynatask,
    title = "Dynatask: A Framework for Creating Dynamic {AI} Benchmark Tasks",
    author = "Thrush, Tristan  and
      Tirumala, Kushal  and
      Gupta, Anmol  and
      Bartolo, Max  and
      Rodriguez, Pedro  and
      Kane, Tariq  and
      Gaviria Rojas, William  and
      Mattson, Peter  and
      Williams, Adina  and
      Kiela, Douwe",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.acl-demo.17",
    pages = "174--181",
    abstract = "We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers. Dynatask is integrated with Dynabench, a research platform for rethinking benchmarking in AI that facilitates human and model in the loop data collection and evaluation. To create a task, users only need to write a short task configuration file from which the relevant web interfaces and model hosting infrastructure are automatically generated. The system is available at https://dynabench.org/ and the full library can be found at https://github.com/facebookresearch/dynabench.",
}