Skip to main content

DocMaps Implementation Group: Pilot Retrospective

This overview of the DocMaps pilot demonstrates the ability of DocMaps to grow the nascent preprint review and curation ecosystem, facilitating the exchange, aggregation,and publishing of reviewed preprints.

Published onMar 20, 2023
DocMaps Implementation Group: Pilot Retrospective
·

DocMaps Implementation Group

DocMaps is a framework for representing the editorial processes used to create journal articles, preprints, peer reviews, and other documents in a machine-readable, extensible, interoperable format. Building on prior work by the DocMaps Technical Committee, the DocMaps Implementation Group (DIG) has piloted this framework with preprint evaluations aggregated by the eLife’s Sciety and EMBO’s Early Evidence Base aggregators and the bioRxiv preprint servers. 

The DocMaps pilot started in mid-2022. In addition to providing machine-readable data and context about how community groups and peer review platforms are evaluating preprints, the Implementation Group had four goals: 

  • Support the diverse preprint ecosystem with collaborations amongst review aggregators and preprint servers to leverage the former’s metadata indexing

  • Bring all evaluations of different groups for a preprint within an activity stream (offering original source as well as in-context view)

  • Simplify metadata ingest amongst many review parties with feed formats

  • Explore potential visualizations of DocMaps content on preprint servers, aggregators, and search results

This pilot has demonstrated the ability of DocMaps to grow the nascent preprint review and curation ecosystem, facilitating the exchange, aggregation,and publishing of reviewed preprints, increasing the number and kinds of evaluations displayed alongside preprints, and providing greater visibility and credibility. Read more about the pilot participants on the DocMaps Implementation Group page.

Overview

Here’s how DocMaps works in the bioRxiv/Sciety integration, using a preprint about plant-insect interaction networks as an example: 

  1. A researcher submits a preprint to bioRxiv

https://www.biorxiv.org/content/10.1101/2022.11.10.516059v1  

On 13 November 2022, authors Hsi-Cheng Ho and Florian Altermatt submitted a preprint to bioRxiv. 

  1. A review group uploads its evaluation of the preprint 

https://zenodo.org/record/7504294#.Y7wowOLMKWA 

A PREreview community member (an ecology journal) uploaded their peer review on January 4, 2023. 

In the pilot, a Sciety reviewer community (Group) selected the preprint for evaluation, curation, and screening. The group uploads its evaluation to a preprint review platform, publishing platform, or other repository. 

  1. An aggregator indexes the preprint & reviews, creating a DocMap for the preprint. 

https://sciety.org/articles/activity/10.1101/2022.11.10.516059 

Sciety aggregates the preprint and its evaluations (reviews) as an article, displaying the corresponding activity. Sciety uses the EuropePMC search function behind the scenes to return XML for preprints posted to bioRxiv and medRxiv, along with their version information. Returned preprints are appended to these results with any evaluations (namely reviews) fetched from external sources, generating an activity stream for each preprint. For preprints with relevant evaluations, Sciety creates a DocMap for the preprint from the evaluation.  Additional information is(at present) hard-coded into the DocMap.  

On the front-end, DocMaps assertions populate the Article Activity Feed, displaying the preprint’s editorial history. Users can refer back to the preprint on bioRxiv, read the review on Sciety, or read the review in context on the review management platform. 

  1. bioRxiv populates the “Community Reviews” tab for the preprint with the review group’s evaluation 

https://www.biorxiv.org/content/10.1101/2022.11.10.516059v1 

In addition to displaying metrics for article views, PDF downloads, and attention scores, bioRxiv includes information about public conversation and dissemination regarding a preprint. The content of Sciety’s DocMaps is displayed as Community Reviews (peer reviews for which the author did not opt in) for an individual preprint. Again, users can refer back to the review on the aggregator or in context on the review management platform.

Here’s the machine-readable DocMap for this preprint:

https://sciety.org/docmaps/v1/articles/10.1101/2022.11.10.516059.docmap.json  

Outputs

  • The DocMaps Project on GitHub, which contains JSON-LD contexts and framing for DocMaps that can be referenced in DocMaps.

  • Endpoints for available DocMaps from Sciety and EEB are available.

    • Sciety: DocMaps can be batch queried at https://sciety.org/docmaps/v1/index.

    • Early Evidence Base: Individual DocMaps can be queried at https://eeb-dev.embo.org/api/v2/docmap/[DOI of preprint].

  • docmap-tools developed by eLife use DocMap content, generate JATS XML from it, and other utility functions.

  • render-rev is a Web Components-based Javascript package that fetches data about the peer review process of a preprint and displays it in the form of a timeline. The timeline contains the most important points of the process such as reviews and author replies. All data about the review process is fetched from the Early Evidence Base (EEB) API in the DocMaps format.

  • Sciety’s API Prototype repo, which contains an experiment into creating an API using RDF for integration between Sciety, bioRxiv and others.

  • Kotahi, which works closely with Sciety as part of a Publish-Review-Curate ecosystem, released version 1.4.0 on 26 January 2023. This release can generate DocMaps on publish action for the elife instance archetype.

Next Steps

For this pilot, the bioRxiv/Sciety integration leverages the metadata available across the publishing ecosystem, adding important context and information to the research presented on each platform. At present, Sciety has generated over 2,000 DocMaps for reviewed preprints. Existing DocMaps can be accessed via Sciety’s API. Early Evidence Base has generated over 6,000 DocMaps for refereed preprints amongst its formal review platforms like Review Commons, Peer Ref, eLife, Rapid Reivews: COVID-19, Peer Community In, EMBO Press, and Peerage of Science. Together, we will continue to explore more complex representations of editorial processes, increasing assertions to create complete and accurate DocMaps.  

Thanks to new funding from the Chan Zuckerberg Initiative’s Open Science program, we will expand support for DocMaps implementation and integration. We will generate thorough documentation for developers creating and consuming DocMaps, expand the framework for mapping to community standards and vocabularies, and continue developing resources to support groups launching preprint evaluation and curation initiatives. You can view our roadmap and other documentation

Get In Touch

To learn more about the DocMaps framework, see the draft documentation. If you are interested in implementing DocMaps as a reviewer, aggregator, or consumer, please email us at [email protected].

Comments
1
Andrew Edstrom:

Dead link. The problem is that the period is within the link.

?
Emily Esten:

fixed - thanks for catching this!