Fill out this form to submit a system for evaluation on the unreleased test set.
Systems must be submitted as Docker or GPU-enabled Docker containers (
Docker is used to minimize differences between your training environment and the test environment.
The NEWSROOM download tools include:
These tools are provided to test systems prior to submission.
The test dataset is provided on standard input in a JSON line stream. This is similar to the training data, but only containing the "text" field. By default, this is the raw summary string, but can be tokenized if requested. From this input, the system must reproduce a list of JSON-encoded strings to standard output. The output must be in the same order as the input. If possible, standard output should be flushed between summaries.
The Dockerfile for the container must specify an
ENTRYPOINT to be run the container as an executable (see TextRank example).
Docker containers are run using the provided
newsroom-run tool, which runs
nvidia-docker as specified.
docker run \ -a stdin -a stdout \ -i --rm [IMAGE] \ < unreleased.jsonl > summaries.jsonl
System summaries are evaluated using ROUGE-1, ROUGE-2, and ROUGE-L both with and without stemming.
If a tokenizer is specified, summaries are evaluated with and without tokenization.
See the provided
newsroom-test tool for the exact evaluation procedure.
This tool can also be used for evaluation on the provided development and test datasets.
Additional instructions for using Docker, running, and evaluating systems are provided in the NEWSROOM download. Review these instructions before submission.