SIA: a scalable interoperable annotation server for biomedical named entities

A vast amount of information on biomedical processes is scattered over millions of scientific publications. Manual curation of this information is expensive and cannot keep up with the ever increasing volume of biomedical literature [1]. To this end, several sophisticated natural language processing tools have been proposed to assist professionals in finding specific information from texts. Many of these highly specialized tools are provided as open source projects to the community. However, the integration of state-of-the-art open source extractors into customized text-mining workflows is often difficult and cumbersome [2, 3]. Standardized interchange formats, such as BioC [4], enable the exchange of text mining results but the initial set-up of these tools is still an unsolved issue. Exposing tools via public web services implementing common specifications bypasses this problem and allows a code-agnostic integration of specific tools by providing an interoperable interface to third parties. This enables simple integration, comparison, and aggregation of different state-of-the-art tools. In this publication we present SIA, our contribution to the BeCalm-Technical interoperability and performance of annotation servers (BeCalm-TIPS) task [5]. SIA is a robust, scalable, extensible, and generic framework to combine multiple named entity recognition tools into a single system.

The publication is organized as follows: First, we briefly introduce the BeCalm-TIPS task and its requirements. We then give an overview of the SIA system architecture, followed by a detailed description of the implementation and the error handling features. This is followed by a scalability experiment conducted on a large dump of PubMed articles and a discussion of the results. We end with a summary and a future work section.

The following section provides a short introduction to the BeCalm-TIPS task, focusing on the payloads annotation servers had to accept and respond with. A detailed description of the task is available in [5].

The task set out to define a testbed for comparing different annotation tools by making them accessible via public web endpoints which exchange standardized JSON messages. It required participants to register their endpoint and a set of supported named entity types with a system managed by the task organizers. Over the course of the task, this endpoint received a number of annotation requests. Each request was not required to be processed interactively, just the message reception had to be acknowledged. Once the annotations were generated by the annotation server, they had to be sent back to a dedicated endpoint—via a separate HTTP request.

Listing 1 shows an excerpt of the JSON payload for an annotation request. It consists of a list of document identifiers and their respective source. As no text was transmitted, participants where required to implement their own text retrieval component to fetch the title, abstract and potentially full text for each document prior to processing. A type field specified the list of named entities to be identified. A unique communication identifier was passed along, which had to be included in any outgoing messages in order to correlate individual requests and responses.

Once the annotation server acknowledged the reception of a request it had a specified amount of time to respond. Listing 2 shows a snippet of such a response. It contains a list of detected annotations across all requested documents, identifying the text source section (abstract A or title T), the start and end positions within it, a confidence score, and the extracted named entity type as well as the annotated text itself.

The task merely specified the required input—as well as output schemata and gave participants full control over the implementation of their system as well as which annotation types they wanted to support.

To connect these components, we opted for a message based architecture, which links each component to a central message bus, over which they exchange messages. Incoming annotation requests are translated into messages by the front end, and subsequently processed by the back end. Once processing is finished the final result is handled by the result handler. To this end, SIA defines a configurable message flow for each message, which incorporates fetching raw texts, running a set of annotators, aggregating the results and forwarding them to a result handler. The configuration defines the actual processing steps, the set of annotator components to use, which document fetchers to enable and how to deal with the results. For example, a processing flow could fetch PubMed articles from a public endpoint, handle all requests for Mutations with the SETH [6] tagger and send annotation results back to the requester. The overall processing flow is expressed as an ordered sequence of message transformation and aggregation steps, while the configuration allows to extend the actual processing flow with new annotator and document fetcher components. Interested readers are referred to Enterprise Integration Patterns [7] for a detailed discussion of the different message handling and transformation strategies that SIA employs.

To handle messages, persistent queues are defined as input and output buffers for all components, where a subsequent component consumes from the previous component’s output queue. These queues are stored for the entirety of the system’s lifetime. This architecture further provides fault tolerant and scalable processing. Fault tolerance is enabled through component wise acknowledgment of each successful message processing, which allows replaying all unacknowledged messages during system recovery, while scalability is achieved through component replication and round robin based message forwarding for increased message throughput.

Messages, the data objects in our architecture, carry information through the system and are composed of a Header and Payload part. The Header contains meta information, such as expiry date, global ids and requested annotation types, and is used by the system to route messages to the respective consumers. The Payload contains the actual data to be processed.

SIA is implemented in Java and uses RabbitMQ [8] as its message bus implementation. In the following each individual component of SIA is described in detail.

Result handler

The result handler processes the aggregated annotation results from the back end, by consuming from a dedicated output queue. We implemented a REST component according to the TIPS task specification, which posts these annotations back to a dedicated endpoint. Additional handlers, such as statistics gatherer or result archiver, can easily be added.

In the following we describe the failure handling strategies across the different components within SIA.

Invalid requests Client calls with wrong or missing information are handled in the front end using request validation. Such invalid requests are communicated back to the caller with detailed error descriptions.

Backpressure To avoid that a large number of simultaneous requests can temporarily overload the processing system, SIA buffers all accepted requests in the input queue - using priorities to represent deadlines.

Front end fails If the front end stops, new requests are simply not accepted, irrespective of any ongoing processing in the back end.

Back end unavailable Messages are still accepted and buffered when there is enough storage space, otherwise the front end denies any new annotation requests.

Back end fails If the back end stops while there are still messages being processed, these are not lost but retried upon restart. This is enabled by acknowledging each message only upon successful processing per component.

Corpus adapter fails Each adapter retries, using exponential backoff, to fetch a document before it is marked as unavailable. As the BeCalm-TIPS task does not specify how to signal unavailable documents, these are just internally logged. Any subsequent processing treats a missing document as one with no content.

Annotator fails If an annotator fails on a particular message, this can potentially harm the entire back end when annotators are embedded in the system. As annotators are software components not under the control of the processing pipeline, we catch all recoverable errors and return zero found annotations in these cases—logging the errors for later analysis.

Result Handling fails The BeCalm-TIPS task description expects the result of an annotation request to be delivered to a known endpoint. If this fails, the delivery is retried in a similar manner to the corpus adapter failure handling.

Message expired Clients can define a deadline for results. This is mapped to a time-to-live attribute of each message. This results in automatically dropping any expired messages from the message bus.

To test the scalability as well as extensibility of SIA we performed an offline evaluation, focusing on throughput. To this end we extended the front end to accept full text documents and added an identity corpus adapter which simply returns the document text from the request message itself. Furthermore, we added a result handler, which writes all results into a local file. By adding these components, we turned SIA into an offline annotation tool, that can be fed from a local collection of text documents without relying on external document providers.

SIA itself is very lightweight and runs anywhere given a Java environment and a connection to RabbitMQ. Annotators can be directly embedded or configured to run externally, exchanging messages through the bus. During the BeCalm-TIPS tasks, we deployed SIA into Cloud Foundry, a platform as a service provider, which enables deployments of cloud containers [18]. The front- and back end with embedded result handling were deployed as two separate application containers connected to a hosted instance of RabbitMQ. To limit the resource consumption, we only enabled the SETH, mirNER and DiseaseNER annotators.

Figure 2 shows the received and processed annotation requests over the course of a four week period during the task. It highlights that our system is capable of sustaining a high number of daily requests, with more than 14.000 daily requests received at maximum. Furthermore we observed that the request handling time during these weeks was dominated by individual corpus downloading times, which make up about \(50\%\) of the overall processing time. This validates our decision to support bulk downloading of documents, as this amortizes the networking overhead for each document, over a number of documents. Processing each annotation request in total took less than two seconds for the configured annotators. We observed higher annotation times for PubMed articles, which is partially due to higher server response times and the need for more sophisticated result parsing. We also estimated the message bus overhead to about \(10\%\), stemming from individual message serialization and persistence compared to running the annotators stand alone—an acceptable slowdown which is easily compensated by additional parallelism.

We described SIA, our contribution to the BeCalm-TIPS task, which provides scalability—through component replication, fault tolerance—through message acknowledgement, and extensibility—through well defined injection points—with a particular emphasis on failure handling. The message-based architecture proved to be a good design blueprint, which can be extended with additional components. To further provide scalable processing, a suggested improvement is to automate the back end scaling by coupling it with an input queue length monitoring. This would allow to scale the back end up or down in response to changes in observed load. One interesting further development path is to port SIA to a distributed streaming environment such as Flink [19] or Spark [20]. These systems reduce the overhead of the message bus at the expense of more complex stream processing and result aggregation. While many of the existing components could be reused, some engineering effort would need to be spent on implementing a fault tolerant aggregation, integrating the potentially unreliable corpus adapters.

To encourage further discussion, the source of our current solution is freely available under an Apache 2.0 license at https://github.com/Erechtheus/sia, along with detailed guides on how to run and deploy the system.