Artificial Intelligence makes many things possible. We’re making them happen.

For example, in NLP in academic settings the usual approach is to concentrate exclusively on pure texts, such as news articles. The reality is often completely different, involving processing of bills, insurance claims, complaints, surveys, Excel tables, PowerPoint slides and documents with little linguistic information, but a lot of meaningful visual text block placement. The most common use case is the extraction of information from tables or other such semi-structured data. Extraction of information from pure texts with proper sentences, phrases or other linguistic structures is actually quite rare.

As a result, very advanced techniques common for NLP (such as high-end parsing solutions) often have little to no influence on the achieved scores. On the other hand, taking visual placement of text blocks on the original document into account has a tremendous influence. For us, this means that we must go beyond these well-trodden paths and combine multiple data modalities in unique ways, such as for visual document analysis.

Unsupervised training on large data sets

Unsupervised training may run against a full web crawl, consisting of hundreds of millions sentences. This would produce a language model. Something that encodes the general usage of words, sentences, etc. of the language in question. This language model is independent from what it is going to be used for later on. It encodes a lot of linguistic knowledge, such as word similarity (“house” and “building” are semantically similar), syntax of words (“running” and “tidying” have a similar sentence function), ambiguity, etc. The unsupervised training process derives all of this information by itself from the data. Imagine seeing many sentences with contexts like “the house was built in” or “the building was built in”. Even without any knowledge in common English everyone would be able to deduce that “house” and “building” appear to be interchangeable.

Unsupervised learning evolved a lot in the past 10 years. It went from complicated co-occurrence counting based methods specifically for semantic vs. syntactic unsupervised models towards the current state-of-art of Neural Network based, fully contextualized, character-level-based language model that assign an individual vector to each word.

This vector has many properties, such as being similar (but not necessarily equal) to the vector of the same word in a slightly different context. Or being different for the same word in a completely different context. The character-based approach allows to assign good vectors even to words that were not seen in the unsupervised training (so-called out-of-vocabulary words). It also means that words with OCR errors in them will get sensible vectors most of the time. These modern methods are simpler by design, but much more demanding in terms of required compute power.

As of 2019, training a good language model requires at least a small super computer with four Tesla GPUs and will train for a couple of weeks.

Supervised training

Supervised training is the task to find a model that reproduces a given set of training examples, and generalizes to another set of test data, which was not seen during training. An additional validation data set is often used in-between for tuning hyper parameters of the supervised training.

Traditionally, algorithms like Support Vector Machine (SVM), or Conditional Random Field (CRF) were used in this area. Recently, though, supervised training is usually implemented by using neural network solutions with components like convolutional neural networks (CNN), or Long-Short-Term-Memory (LSTM).

ExB developed an own architecture for separating feature generation (designed, or pre-learned) from actual machine learning, so that the machine learning algorithm can be easily exchanged. At the same time, feature generation can also be easily expanded without affecting the machine learning layer in any (negative) way.