Summarization Dataset

Summarization System Submission

Fill out this form to submit a system for evaluation on the unreleased test set. Systems must be submitted as Docker or GPU-enabled Docker containers (nvidia-docker). Docker is used to minimize differences between your training environment and the test environment. The NEWSROOM download tools include:

  1. A working Docker example for the TextRank summarization system.
  2. The CLI tool used to run submitted summarization systems (newsroom-run).
  3. The CLI tool used to evaluate summaries of submitted systems (newsroom-test).

These tools are provided to test systems prior to submission.

  You will receive email confirmation.

Input and Output

The test dataset is provided on standard input in a JSON line stream. This is similar to the training data, but only containing the "text" field. By default, this is the raw summary string, but can be tokenized if requested. From this input, the system must reproduce a list of JSON-encoded strings to standard output. The output must be in the same order as the input. If possible, standard output should be flushed between summaries.

Running the System

The Dockerfile for the container must specify an ENTRYPOINT to be run the container as an executable (see TextRank example). Docker containers are run using the provided newsroom-run tool, which runs docker or nvidia-docker as specified.

docker run \
    -a stdin -a stdout \
    -i --rm [IMAGE] \
    < unreleased.jsonl > summaries.jsonl

System Evaluation

System summaries are evaluated using ROUGE-1, ROUGE-2, and ROUGE-L both with and without stemming. If a tokenizer is specified, summaries are evaluated with and without tokenization. See the provided newsroom-test tool for the exact evaluation procedure. This tool can also be used for evaluation on the provided development and test datasets.

Additional Instructions

Additional instructions for using Docker, running, and evaluating systems are provided in the NEWSROOM download. Review these instructions before submission.