What are word embeddings' real origins?

Claude Shannon's 1948 info theory encoded words as probability vectors — neural nets just densified it later.

How does RoPE positional encoding work?

Rotates query/key vectors by position angle; scales to megatokens without param explosion.

Can Gemma 4 beat closed models?

Leaderboard says yes on raw smarts, but open evals expose safety gaps — edge wins await.

Word Embeddings: Shannon's 1948 Secret, Not Word2Vec Myth

Word embeddings didn't spring from Word2Vec. They trace to Shannon's 1948 brilliance. This week's AI digest reveals the deep history — and fixes for bland bots.

theAIcatchup Apr 09, 2026 4 min read

Vintage photo of Claude Shannon with modern word embedding vectors overlaid

⚡ Key Takeaways

Word embeddings originated with Shannon's 1948 information theory, predating neural networks. 𝕏
RoPE enables massive context windows via clever rotations, hand-computable elegance. 𝕏
Tune RAG chunk overlap to 10-20% for real recall gains; ignore at peril. 𝕏

Published by

theAIcatchup

AI news that actually matters.

#AI history #Claude Shannon #Gemma 4 #RAG chunking #RoPE positional encoding #Shannon information theory #information theory #word embeddings

Worth sharing?

Get the best AI stories of the week in your inbox — no noise, no spam.

Originally reported by Towards AI

⚡ Key Takeaways

The 60-Second TL;DR

theAIcatchup

Share this article

Worth sharing?

Related Stories

Foundation Models: AI's Hidden Engines Reshaping Everyday Tech

Gemma 4 Compresses Frontier AI into Everyday Code

Google's Gemma 4: The Open Model That Finally Makes Edge AI Less of a Joke

Gemma 4 Lands Hard: Google's Open-Weight Arsenal Fires Back at China

Stay in the loop