Text Mining and NLP Fundamentals

LanguageLanguage is sequential, ambiguous, and enormous in vocabulary — three properties that make it a genuinely different kind of data from the tabular or image data covered elsewhere on this site. Natural language processing (NLP) is the discipline built around handling those three properties, and its methods have layered up over several decades: normalisation and counting first, statistical models next, and learned representations after that. Jurafsky and Martin's textbook, freely available online and continuously updated, is the standard reference for the whole field and the source this page follows most closely¹.

Normalisation and N-Gram Models

BeforeBefore any statistical method can run, raw text needs normalising: lower-casing, stripping punctuation, splitting into tokens, and often reducing words to a root form (stemming or lemmatisation) so that "running" and "runs" aren't treated as two unrelated vocabulary items. Regular expressions are the practical workhorse for a lot of this — pattern-matching to strip HTML tags, split sentences on punctuation, or find and normalise dates and numbers.

An n-gram language model answers a simple question — given the last n-1 words, what's the probability distribution over the next word? — estimated directly from counting how often each sequence of n words appears in a large corpus. A bigram model (n=2) predicts the next word from just the previous one; a trigram model (n=3) uses the previous two. The practical problem is data sparsity: most possible word sequences never appear in any training corpus no matter how large, giving them an estimated probability of exactly zero, which is fatal for a model that needs to assign some probability to sequences it hasn't seen — smoothing techniques (redistributing a small amount of probability mass from seen sequences to unseen ones) exist specifically to fix this.

Part-of-Speech Tagging and Hidden Markov Models

Part-of-speechPart-of-speech (POS) tagging assigns each word in a sentence a grammatical category — noun, verb, adjective, and so on — which matters because the same word form can be more than one part of speech ("book" a flight vs. read a "book"), and knowing which is essential for almost every downstream task. A Hidden Markov Model (HMM) frames this as inferring a hidden sequence (the tags) from an observed sequence (the words): it assumes each tag depends only on the previous tag (a Markov assumption on the hidden states) and each word depends only on its own tag (the emission probabilities), then asks which sequence of hidden tags best explains the observed sentence.

Checking every possible tag sequence directly is exponential in sentence length, which is exactly the kind of blow-up dynamic programming exists to avoid. The Viterbi algorithm solves it in time proportional to sentence length times the number of tags, by keeping, at each position, only the single best path to each possible tag so far — rather than every path — and building the next position's best paths from those, exactly the overlapping-subproblems structure dynamic programming is built to exploit.

Each column above is one word's possible tags (a "lattice" or trellis); each arrow is a transition probability between tags, and Viterbi keeps only the highest-scoring incoming arrow at every node (solid) rather than tracking every path (dashed) — after the last word, tracing back through the kept arrows from the best final-column score gives the single best tag sequence for the whole sentence, without ever having enumerated the others.

Word Embeddings

TreatingTreating each word as an arbitrary, unrelated symbol (as raw n-gram counting effectively does) throws away an obvious source of information: "cat" and "dog" are more similar to each other than either is to "bicycle," and a model that can't represent that similarity has to relearn every pattern separately for every word rather than generalising across related ones. The distributional hypothesis — a word is characterised by the company it keeps — motivates representing each word instead as a dense vector, positioned in a continuous space such that words appearing in similar contexts end up with similar vectors. Mikolov et al.'s word2vec popularised this at scale with two efficient training schemes: predict a word from its surrounding context (continuous bag-of-words) or predict the surrounding context from a word (skip-gram)². The famous consequence of training this way is that vector arithmetic starts tracking meaning — the vector for "king" minus "man" plus "woman" lands close to the vector for "queen" — purely as a side effect of words with similar roles appearing in similar contexts across a large corpus, with no explicit instruction to encode gender or royalty as a concept.

Applied Tasks: NER and Sentiment Analysis

NamedNamed entity recognition (NER) identifies and classifies spans of text into categories — person, organisation, location, date — and is usually framed as a sequence-labelling problem structurally similar to POS tagging (label each token, but this time with a category like "beginning of a person's name" rather than a part of speech). Sentiment analysis classifies a whole piece of text (a review, a tweet) by the attitude it expresses, and ranges from simple lexicon-based approaches (count positive versus negative words from a pre-built list) to full learned classifiers trained on labelled examples — the same supervised-learning machinery covered on Supervised Learning, just applied to text-derived features (word counts, or word embeddings averaged over a document) instead of numeric ones.

Sentiment analysis is also a real argument against blindly reusing the normalisation pipeline from earlier on this page. Stripping emoji and emoticons during cleaning — a default move if they're treated as noise the way HTML tags are — throws away a genuinely strong sentiment signal rather than an irrelevant one: Novak et al.'s emoji sentiment lexicon, built from 751 of the most frequent emoji across a large multilingual tweet corpus, found that most carry a clear positive or negative polarity of their own, that tweets containing emoji score measurably more positive on average than those without, and that human annotators agree with each other more often when an emoji is present, not less — evidence that emoji are clarifying the writer's intended tone, not decorating it³. The practical conclusion: a sentiment pipeline should treat emoji as a feature to extract (often via a lexicon mapping each one to a polarity score, exactly like a word), not a character class to delete during cleanup.

Sarcasm is the harder problem sentiment analysis doesn't have a clean answer for, and it attacks lexicon-based methods at their weakest point: a sarcastic sentence's individual words often carry the opposite polarity to the sentiment actually meant — "what a wonderful start to the day" after missing a train scores strongly positive on a word-count lexicon while meaning the reverse. González-Ibáñez, Muresan, and Wacholder's study of sarcasm detection on Twitter found that even human judges, given only the text of a tweet with no other context, frequently disagreed with each other about whether it was sarcastic, and that the lexical and pragmatic cues good automatic classifiers lean on (exaggeration, specific punctuation and interjection patterns) still fall well short of reliable detection⁴. This is a real, unresolved limitation to design around, not a solved sub-task: a production sentiment system needs some explicit answer for how much sarcasm-driven misclassification it can tolerate, rather than assuming the lexicon or classifier already handles it.

Where This Leads

EveryEvery method above — n-grams, HMMs, static word embeddings — predicts from a fixed, local window of context. The next step in the field's history was building models that could weigh the entire surrounding context, of any length, when deciding what a word means in a specific sentence — which is exactly the shift covered on Neural Network Architectures and, in full detail for the language-modelling case specifically, on Understanding LLMs. Modern large language models are, in one real sense, the direct descendants of the n-gram language model this page opened with — still predicting the next word from context — just with a vastly richer, learned notion of what "context" means.

References