나는 컴퓨터 “Apple” (과일, 아이폰 및 벡터라고 생각했다) 을 가르치려고 노력했다.

Hello, DEV Community! 🙌 It’s my first time writing here.

As I was studying how models handle text today, I ran into a funny realization: computers don’t understand human words at all. They only understand numbers.

So how do we explain to a model that a cat is a pet, an apple can be a fruit or a tech company?

Here is what I learned today about Word Embedding, explained from a pure logic and geometry perspective!

1. The First step: Making a Massive List(One-Hot Encoding)

If you ask a beginner programmer how to turn words into numbers, the most obvious idea is to give every single word a simple code or an index in a giant list.

Imagine taking an entire dictionary of 100,000 words and marking 1 for the word you want and 0 for everything else.

• cat = [1, 0, 0, 0, ...]
• dog = [0, 1, 0, 0, ...]
• apple = [0, 0, 1, 0, ...]

Why this fails:

1. It Wastes Endless Space: You end up creating massive arrays full of useless zeros just to represent one tiny word. RIP RAM. 🪦
2. Computers Got Zero “Street Smarts”: In pure math, these simple list positions have nothing in common. To the computer, cat has as much in common with dog as it does with banana. It has no idea that two animals belong together.

2. The Solution: Giving Words Map Coordinates

Instead of a giant empty list, researchers like Mikolov et al. at Google introduced Word Embedding (such as Word2Vec and GloVe).

Think of it like plotting points on a map. In physics or math class, you plot points using X, Y, and Z coordinates. Word embedding does the exact same thing, except instead of 3 directions, they use 50 to 300 invisible dimensions!

These dimensions represent hidden concepts like is_animal, is_food, is_tech, or grammatical_type.

• “cat” gets coordinates near animal concepts.
• “dog” gets coordinates right next to “cat”.
• “apple” gets pulled into a coordinate spot sitting right in between the fruit neighborhood and the iPhone neighborhood!

How does the computer learn these coordinates? By sliding a “context window” across millions of sentences on the internet. It observes which words consistently live as neighbors in the same sentences so if words appear in similar contexts, they get pulled toward the same coordinates in vector space.

Because the computer reads millions of sentences, it notices which words “live” in the same neighborhoods.

3. Math Geometry Magic

Because every word becomes a point in space, we can measure how close two words are by calculating the angle or distance between their points.

If two words point in nearly the exact same direction, they share similar meanings!

Doing Algebra with Words

My discovery today was that you can actually do math equations with word coordinates:

King - Man + Woman = Queen

This famous vector arithmetic concept was demonstrated in the original Word2Vec paper by Mikolov et al. (2013).”

If you start at the coordinate for King, subtract the “male” direction, and add the “female” direction, your new coordinate in space lands right on Queen. 🤯

Final Thoughts

Learning that word embedding are just coordinate maps helped me bridge the gap between machine language (a.k.a math) and human language.

In any case, word embedding are nothing but high dimensional maps that make words street-smart. The idea of understanding text via coordinates really clicked with me!

References & Further Reading

1. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781
2. Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. Stanford NLP
3. Alammar, J. (2019). The Illustrated Word2Vec. jalammar.github.io