Concept Mapping and Self-Organizing Maps

The Self-Organizing Map Pipeline

TheThe Self-Organizing Map (SOM) is a neural network architecture introduced by Teuvo Kohonen in 1982 as an unsupervised method for projecting high-dimensional input onto a low-dimensional (typically 2D) lattice while preserving neighbourhood relationships in the original data [1]. Training proceeds by competitive learning: each input vector is compared against every neuron's weight vector, the closest neuron (the "Best Matching Unit") and its lattice neighbours are pulled towards the input, and the neighbourhood radius shrinks over successive epochs until the grid settles into a stable topological ordering. The property that makes SOMs useful for a concept map specifically is that inputs which are similar in the original high-dimensional space end up near each other on the trained grid — so a 2D layout of document vectors can be read, approximately, as a spatial layout of document similarity.

Generating a cartographic topological map from a collection of text documents involves a multi-stage Machine Learning pipeline:

1. Feature Extraction: Document text is tokenised into word n-grams, filtered against standard stopword lists, and weighted using Term Frequency-Inverse Document Frequency (TF-IDF).
2. Pure Python Broken Stick Feature Selection: Selecting high-variance features using the Broken Stick distribution ($E_k = \frac{1}{p} \sum_{i=k}^p \frac{1}{i}$) without relying on native C-extension dependencies like NumPy or Scikit-Learn.
3. 2D Kohonen Map Training: Training a $40 \times 40$ toroidal or planar grid of neuron weight vectors using competitive unsupervised learning.
4. U-Matrix Terrain Generation: Computing Unified Distance Matrix (U-Matrix) values between neighbouring neuron weight vectors to form topographic elevation contours.

Pure Python Broken Stick Feature Selection

ToTo eliminate Windows C-extension runtime crashes and MINGW-W64 math library incompatibilities, feature selection is implemented in Pure Python using the Broken Stick Model. The expected proportion of variance for the $k$-th component in a random stick-breaking process of length $p$ is given by:

\[E_k = \frac{1}{p} \sum_{i=k}^{p} \frac{1}{i}\]

Features whose observed variance exceeds the broken stick threshold $E_k$ are retained for SOM training, automatically isolating high-variance topic discriminators while pruning noise.

U-Matrix Terrain Heatmaps & Mountain Ridges

TheThe U-Matrix (Unified Distance Matrix) measures the Euclidean distance between adjacent neuron weight vectors across the 2D grid:

• Low U-Matrix Values (Valleys & Plains): Indicate clusters of closely related documents with similar vocabulary. These areas are rendered as light parchment terrain where city nodes settle.
• High U-Matrix Values (Mountain Ridges): Indicate sharp semantic transitions or boundary zones between distinct topic areas. These regions are populated with mountain peak vectors (⛰️) and dense forest icons (🌲).
• Swamps and Marshes (🌾): Low-density or isolated transition zones marked with reeds and puddles.

Cartographic Export Schema

TheThe trained SOM topology is exported directly into assets/site-atlas.json as a lightweight JSON payload consumed by the frontend renderer. The schema includes grid dimensions, node coordinates, U-Matrix elevation arrays, Voronoi grid categories, terrain features, and precomputed highway paths with bridge concept metadata.

References

1. T. Kohonen, "Self-Organized Formation of Topologically Correct Feature Maps," Biological Cybernetics, vol. 43, no. 1, pp. 59–69, 1982. https://doi.org/10.1007/BF00337288