project · 2023 · MSc Project · RPTU Kaiserslautern

Multilingual Named Entity Recognition for Investigative Documents

A comparative study and implementation of NER approaches for extracting person names, organisations, and locations from multilingual documents, benchmarking NLTK, spaCy, BERT, BERT+CRF, and BERT+BiLSTM+CRF across English, German, Russian, and Chinese.

BERTBiLSTMCRFspaCyNLTKPyTorchHugging FaceArangoDBPython

Investigative journalism involves sifting through large volumes of documents across multiple languages to find connections between people, organisations, and events. Manual extraction of this information does not scale. This project explored automated named entity recognition as a solution, benchmarking five approaches against a consistent set of metrics to identify which generalises best across languages.

The problem

The task was to extract three entity types from multilingual documents: person names, organisation names, and locations. The challenge is that the same entity can appear differently across languages, writing systems, and document styles, and a model trained on English text frequently degrades on German, Russian, or Chinese. The goal was an approach that holds up across all four.

Dataset

The WikiANN dataset was used for training and evaluation: 165,804 tokens drawn from Wikipedia articles, annotated in IOB format (B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC) across 176 supported languages.

Five approaches compared

NLTK with Hidden Markov Model forms the statistical baseline. The pipeline runs sentence segmentation, tokenisation, POS tagging, and entity extraction through an HMM tagger. Straightforward to implement but limited by HMM’s inability to model long-range context and the lack of deep learning support.

spaCy with pre-trained language models uses large pre-built models for English, German, Russian, and Chinese. Fast and practical but prone to misclassification on domain-specific or uncommon entities, and limited for short or noisy text.

BERT fine-tuned on WikiANN is the core transformer approach. BERT’s bidirectional attention captures full sentence context, making it significantly more robust than rule-based or statistical methods. Fine-tuned with bert-base-cased for English, bert-base-multilingual-cased for German and Russian, and bert-base-chinese for Chinese.

BERT + CRF adds a Conditional Random Field layer on top of BERT’s token embeddings. The CRF models dependencies between adjacent output labels, which is useful for enforcing valid tag sequences (an I-PER should not follow a B-ORG, for instance).

BERT + BiLSTM + CRF adds a bidirectional LSTM between BERT and the CRF layer. BERT processes all tokens simultaneously and can lose positional dynamics over long sequences. The BiLSTM layer runs forward and backward over BERT’s embeddings, capturing sequential order and long-range dependencies before the CRF makes the final label prediction.

Input tokens
     │
     ▼
┌──────────┐
│  BERT    │  contextualised token embeddings
└────┬─────┘
     │
     ▼
┌──────────┐
│ BiLSTM   │  forward + backward sequential context
└────┬─────┘
     │
     ▼
┌──────────┐
│   CRF    │  optimal label sequence prediction
└────┬─────┘
     │
     ▼
 NER tags (B-PER, I-ORG, B-LOC ...)

Results

BERT base outperformed all other approaches across all four languages on F1, accuracy, precision, and recall. Adding CRF and BiLSTM layers did not significantly improve on BERT alone, though BERT+CRF showed better handling of contextual label dependencies in some cases.

Selected F1 scores (BERT base):

Inference time for BERT across all language variants averaged approximately 0.020 seconds per sentence (60 words), making it practical for batch document processing.

Knowledge graph output

The extracted entities were used to populate a knowledge graph in ArangoDB, connecting persons to organisations and to the source documents they appear in. This graph representation allows investigators to query relationships across a document corpus rather than reading each document individually.

← all work