Hello, I’m Shrijith Venkatramana, and I’m building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Most people learn neural networks by staring at the model.
Weights. Attention. MLPs. LayerNorm. Tokenizers. Context windows.
But when you actually train an LLM, there is another piece of machinery making billions of decisions every second:
the optimizer.
A 70-billion-parameter model does not “learn” because gradient descent tells it which direction is better. It learns because an optimizer turns an enormous, noisy stream of gradients into parameter updates that are small enough not to explode, large enough to make progress, and adaptive enough that different parameters can move at radically different effective rates.
For the last decade, the dominant answer has largely been some form of Adam, and increasingly AdamW.
The interesting part is that Adam is not some mysterious LLM-specific invention. The original Adam paper was submitted in December 2014 by Diederik Kingma and Jimmy Ba, before the Transformer, before GPT, and before the modern LLM era. Kingma was working on scalable machine learning and generative models; Ba was then a PhD student working with Geoffrey Hinton at Toronto.
Three years later, the Transformer paper used Adam directly in its training recipe.
Then came AdamW, which fixed a subtle but important problem in how regularization interacted with adaptive optimization.
By 2025, Adam was sufficiently influential to receive an ICLR Test of Time award.
So what exactly is Adam doing?
And why is AdamW usually what you actually want when training a Transformer?
1. First, forget Adam: what problem is the optimizer solving?
Suppose your neural network has parameters
theta = [theta_1, theta_2, ..., theta_N]
Enter fullscreen mode Exit fullscreen mode
and your training batch produces a loss L.
Backpropagation gives you
g = dL/dtheta
Enter fullscreen mode Exit fullscreen mode
The simplest possible optimizer is gradient descent:
theta <- theta - alpha * g
Enter fullscreen mode Exit fullscreen mode
where alpha is the learning rate.
That looks almost embarrassingly simple.
And that is indeed roughly what people did before adaptive optimizers became dominant.
The problem is that the gradients of a neural network are not nicely behaved.
Imagine two parameters:
g_1 = 0.001
g_2 = 10
Enter fullscreen mode Exit fullscreen mode
A single learning rate has to deal with both.
If you choose alpha = 0.001, parameter 1 barely moves:
delta_1 = -0.001 * 0.001 = -0.000001
Enter fullscreen mode Exit fullscreen mode
while parameter 2 gets:
delta_2 = -0.001 * 10 = -0.01
Enter fullscreen mode Exit fullscreen mode
And this situation is not exotic.
Different parameters can have wildly different gradient scales. Some receive dense gradients every step. Others receive sparse or intermittent signals. Some directions in parameter space are noisy. Others are remarkably consistent.
So the fundamental problem is:
How do we turn a raw gradient into a sensible update for each individual parameter?
Momentum gives one answer.
Adaptive methods give another.
Adam essentially combines both.
2. Adam’s key idea: keep a memory of the gradient
Adam stands for Adaptive Moment Estimation.
The easiest way to understand it is to imagine that every parameter maintains two small pieces of memory.
The first remembers:
“What direction have gradients generally been pointing?”
The second remembers:
“How large have those gradients generally been?”
For every parameter, Adam maintains:
m = moving average of gradients
v = moving average of squared gradients
Enter fullscreen mode Exit fullscreen mode
More precisely:
m_t = beta1 * m_(t-1) + (1 - beta1) * g_t
v_t = beta2 * v_(t-1) + (1 - beta2) * g_t^2
Enter fullscreen mode Exit fullscreen mode
Typically:
beta1 = 0.9
beta2 = 0.999
Enter fullscreen mode Exit fullscreen mode
The interpretation is surprisingly intuitive.
m: momentum
Suppose gradients over five steps are:
+1
+1
+1
+1
+1
Enter fullscreen mode Exit fullscreen mode
Then the moving average also points strongly positive.
Now imagine:
+1
-1
+1
-1
+1
Enter fullscreen mode Exit fullscreen mode
The signs keep cancelling.
Adam therefore distinguishes:
consistent signal
Enter fullscreen mode Exit fullscreen mode
from
noisy oscillation
Enter fullscreen mode Exit fullscreen mode
This is momentum.
v: gradient scale
Now suppose a parameter frequently gets gradients around 10, while another gets gradients around 0.01.
Their squared gradients differ by a factor of:
10^2 / 0.01^2 = 100 / 0.0001 = 1,000,000
Enter fullscreen mode Exit fullscreen mode
Adam remembers this.
That allows it to normalize the effective update.
Ignoring some details for a moment, the update looks like:
delta_theta ~= -alpha * m / sqrt(v)
Enter fullscreen mode Exit fullscreen mode
So if a parameter has persistently large gradients, its denominator is large.
If its gradients are consistently tiny, its denominator is small.
Adam is therefore doing something qualitatively like:
move in the direction supported by recent gradients, but normalize the step according to how volatile/large those gradients have been.
That is the core idea.
It is not just “gradient descent with momentum.”
It is per-parameter adaptive step sizing.
3. The weird-looking bias correction is actually necessary
There is an immediately obvious problem with the equations above.
At initialization:
m_0 = 0
v_0 = 0
Enter fullscreen mode Exit fullscreen mode
Suppose the very first gradient is:
g_1 = 1
Enter fullscreen mode Exit fullscreen mode
Then:
m_1 = 0.1
Enter fullscreen mode Exit fullscreen mode
because:
m_1 = 0.9 * 0 + 0.1 * 1
Enter fullscreen mode Exit fullscreen mode
But the actual observed gradient was 1, not 0.1.
The exponential moving average starts biased toward zero because its history is artificially filled with zeros.
Adam therefore uses bias correction:
m_hat_t = m_t / (1 - beta1^t)
v_hat_t = v_t / (1 - beta2^t)
Enter fullscreen mode Exit fullscreen mode
and the actual update becomes:
theta <- theta - alpha * m_hat / (sqrt(v_hat) + epsilon)
Enter fullscreen mode Exit fullscreen mode
That little correction matters most early in training.
For example, with:
beta1 = 0.9
t = 1
Enter fullscreen mode Exit fullscreen mode
we have:
1 - beta1^t = 1 - 0.9 = 0.1
Enter fullscreen mode Exit fullscreen mode
so:
m_hat_1 = 0.1 / 0.1 = 1
Enter fullscreen mode Exit fullscreen mode
Exactly what we wanted.
The full Adam algorithm therefore has only a handful of moving pieces:
m_t = beta1 * m_(t-1) + (1-beta1) * g_t
v_t = beta2 * v_(t-1) + (1-beta2) * g_t^2
m_hat = m_t / (1-beta1^t)
v_hat = v_t / (1-beta2^t)
theta <- theta - alpha * m_hat / (sqrt(v_hat) + epsilon)
Enter fullscreen mode Exit fullscreen mode
That’s basically it.
A remarkable amount of modern deep learning sits on top of those few equations.
4. Why Adam was such a big deal for Transformers
The timing here is worth appreciating.
Kingma and Ba submitted the Adam paper in December 2014.
At that point, the dominant deep-learning world looked very different. Recurrent networks, convolutional networks, and SGD-style training were central. The Transformer did not yet exist.
Then, in 2017, Vaswani and colleagues published Attention Is All You Need.
The Transformer paper didn’t invent some new optimizer specially designed for attention. It simply used Adam:
beta1 = 0.9
beta2 = 0.98
epsilon = 1e-9
Enter fullscreen mode Exit fullscreen mode
with a warmup-and-decay learning-rate schedule.
That is historically significant because the Transformer went on to become the basic architecture underneath the modern LLM ecosystem.
In other words, one of the most consequential architecture papers in modern AI essentially plugged an existing adaptive optimizer into a radically different neural architecture.
And it worked spectacularly well.
There is a useful practical lesson here:
The optimizer does not have to understand the semantics of the architecture.
Adam has no idea whether a parameter belongs to:
Q projection
K projection
V projection
MLP
embedding table
layer normalization
Enter fullscreen mode Exit fullscreen mode
It simply sees gradients and maintains statistics about them.
That abstraction is part of its power.
A tiny numerical example
Suppose two parameters receive:
Parameter A:
gradients ≈ [0.1, 0.2, 0.15, 0.1]
Parameter B:
gradients ≈ [10, 20, 15, 10]
Enter fullscreen mode Exit fullscreen mode
Parameter B has gradients roughly 100x larger.
With vanilla SGD:
delta_B ≈ 100 * delta_A
Enter fullscreen mode Exit fullscreen mode
Adam partially cancels that scale difference because its denominator tracks gradient magnitude.
You can think of Adam as making the optimizer less sensitive to the arbitrary units in which different parts of the network happen to express their gradients.
That is especially attractive in giant heterogeneous models.
5. But Adam has an enormous hidden cost: memory
There is a catch.
Adam needs to store two additional tensors:
m
v
Enter fullscreen mode Exit fullscreen mode
for every parameter.
So if your model has N parameters, Adam needs roughly:
2N extra values
Enter fullscreen mode Exit fullscreen mode
If those optimizer states are stored in FP32:
4 bytes/value
Enter fullscreen mode Exit fullscreen mode
then optimizer state alone costs:
2 * 4 * N = 8N bytes
Enter fullscreen mode Exit fullscreen mode
Consider a 7B parameter model:
7,000,000,000 * 8 bytes
= 56,000,000,000 bytes
≈ 56 GB
Enter fullscreen mode Exit fullscreen mode
Just for the two Adam moment tensors.
Not model weights.
Not activations.
Not gradients.
Not KV cache.
Just:
m + v
Enter fullscreen mode Exit fullscreen mode
For a 70B model:
70B * 8 bytes ≈ 560 GB
Enter fullscreen mode Exit fullscreen mode
This is one reason optimizer engineering becomes a systems problem at LLM scale.
You can easily have a situation where the matrix multiplications themselves are perfectly GPU-friendly, but your optimizer state is forcing enormous distributed-memory and communication overhead.
There are several ways modern systems deal with this:
FSDP / ZeRO-style sharding
optimizer-state partitioning
CPU/NVMe offload
8-bit optimizer states
fused optimizer kernels
mixed precision
Enter fullscreen mode Exit fullscreen mode
But Adam’s conceptual simplicity hides a surprisingly expensive implementation reality.
For example, suppose your parameters are BF16:
2 bytes / parameter
Enter fullscreen mode Exit fullscreen mode
but your Adam moments are FP32:
8 bytes / parameter total for m and v
Enter fullscreen mode Exit fullscreen mode
The “small” optimizer logic now consumes roughly four times as much memory as the model parameters themselves.
That is why optimizer state can become a first-class architectural concern in large training systems.
6. Adam versus AdamW: the subtle problem with weight decay
This is probably the most important distinction to understand in practice.
People often use the terms:
L2 regularization
weight decay
Enter fullscreen mode Exit fullscreen mode
as though they are interchangeable.
For ordinary SGD, they can effectively be equivalent.
For Adam, they are not.
Suppose we add an L2 penalty to the loss:
L' = L + (lambda / 2) * ||theta||^2
Enter fullscreen mode Exit fullscreen mode
The gradient becomes:
g' = g + lambda * theta
Enter fullscreen mode Exit fullscreen mode
Now notice what Adam does to g'.
It doesn’t simply subtract:
alpha * lambda * theta
Enter fullscreen mode Exit fullscreen mode
from the weights.
The regularization term goes into the adaptive machinery:
g' -> m -> v -> normalization
Enter fullscreen mode Exit fullscreen mode
So the shrinkage of a parameter becomes entangled with Adam’s gradient statistics.
That produces a surprising effect.
Two parameters with the same weight magnitude can receive different effective regularization depending on their gradient history.
Suppose:
theta_1 = 1
theta_2 = 1
Enter fullscreen mode Exit fullscreen mode
and the only difference is that:
sqrt(v_1) = 0.1
sqrt(v_2) = 10
Enter fullscreen mode Exit fullscreen mode
The same regularization contribution gets normalized very differently.
So the thing you thought was:
“shrink every weight by some amount”
has turned into something closer to:
“shrink weights according to how the optimizer’s adaptive statistics happen to scale their gradients.”
That is not the same operation.
Enter AdamW
In 2017, Ilya Loshchilov and Frank Hutter proposed a simple fix.
Don’t put weight decay inside the gradient.
Do it separately.
Instead of conceptually doing:
g <- g + lambda * theta
Adam(g)
Enter fullscreen mode Exit fullscreen mode
AdamW does:
Adam(g)
theta <- theta - alpha * lambda * theta
Enter fullscreen mode Exit fullscreen mode
or, equivalently:
theta <- (1 - alpha * lambda) * theta
Enter fullscreen mode Exit fullscreen mode
Now the optimization step and the shrinkage step are decoupled.
That is the entire conceptual breakthrough.
It sounds tiny.
It isn’t.
This means the optimizer controls:
How should the model move to reduce the loss?
Enter fullscreen mode Exit fullscreen mode
while weight decay controls:
How strongly should parameters be pulled toward zero?
Enter fullscreen mode Exit fullscreen mode
Those are different jobs.
AdamW keeps them separate.
A concrete comparison
Suppose:
theta = 2
alpha = 0.001
lambda = 0.1
Enter fullscreen mode Exit fullscreen mode
Then AdamW’s direct decay contribution is:
alpha * lambda * theta
= 0.001 * 0.1 * 2
= 0.0002
Enter fullscreen mode Exit fullscreen mode
So the weight gets multiplied by:
1 - 0.0001
= 0.9999
Enter fullscreen mode Exit fullscreen mode
per optimization step, ignoring the gradient update for illustration.
After 10,000 steps, that multiplicative factor becomes approximately:
0.9999^10000 ≈ e^(-1) ≈ 0.368
Enter fullscreen mode Exit fullscreen mode
So repeated tiny decay can become very substantial.
This is a useful way to think about weight decay:
It is not a tiny penalty applied occasionally. It is a multiplicative force acting at every optimization step.
And that is why seemingly boring hyperparameters like weight_decay=0.1 can have a large effect over a long training run.
7. What Adam/AdamW actually means when training an LLM
At this point, the practical picture looks something like this:
forward pass
|
v
compute loss
|
v
backprop
|
v
gradient g_t
|
+----> Adam exponential moving averages
| |
| v
| m_t, v_t
| |
| v
| adaptive update
|
+----> AdamW weight decay
|
v
parameters
Enter fullscreen mode Exit fullscreen mode
There are several consequences worth keeping in your head.
Learning rate is still incredibly important
Adam does not eliminate the need to tune the learning rate.
The optimizer normalizes gradients, but alpha still determines the global scale of movement.
A useful mental model is:
Adam decides:
"How large should this parameter's step be relative to its gradient history?"
Learning rate decides:
"How aggressive should the entire optimizer be?"
Enter fullscreen mode Exit fullscreen mode
That is why learning-rate schedules remain central in LLM training.
The Transformer paper, for example, used a warmup followed by inverse-square-root decay rather than holding the learning rate constant.
beta1 controls gradient-memory timescale
The moving average
m_t = beta1*m_(t-1) + (1-beta1)*g_t
Enter fullscreen mode Exit fullscreen mode
has an effective memory on the order of roughly:
1 / (1 - beta1)
Enter fullscreen mode Exit fullscreen mode
steps.
So:
beta1 = 0.9
Enter fullscreen mode Exit fullscreen mode
means roughly a ten-step memory scale.
That is not an exact cutoff; it is an intuition for the EMA timescale.
Likewise:
beta2 = 0.999
Enter fullscreen mode Exit fullscreen mode
corresponds to a much longer memory:
~1000 steps
Enter fullscreen mode Exit fullscreen mode
for the second-moment estimate.
This is why changing beta values is not just changing some arbitrary constants.
You’re changing the temporal horizon over which the optimizer interprets gradient behavior.
epsilon is mostly a numerical stabilizer
The denominator is:
sqrt(v_hat) + epsilon
Enter fullscreen mode Exit fullscreen mode
The epsilon prevents division by something vanishingly small.
In many practical regimes, it is not the dominant behavioral hyperparameter.
But in low-gradient or low-precision regimes, its interaction with numerical scale can matter.
Adam is not Newton’s method
A common misunderstanding is:
“Adam uses second-order information.”
Not really.
It tracks a second moment of gradients:
E[g^2]
Enter fullscreen mode Exit fullscreen mode
but it does not construct the Hessian:
H = d^2L/dtheta^2
Enter fullscreen mode Exit fullscreen mode
and does not estimate the full curvature matrix.
Adam is still a first-order optimizer.
Its sophistication comes from using historical statistics of first-order information.
8. Why developers should care beyond knowing the equations
If you are debugging LLM training, AdamW is not an implementation detail.
It can directly influence:
training stability
loss curves
sample efficiency
generalization
memory footprint
distributed-training architecture
hyperparameter sensitivity
Enter fullscreen mode Exit fullscreen mode
A few practical examples:
Training is unstable
You might immediately suspect:
bad initialization
bad normalization
bad data
exploding gradients
Enter fullscreen mode Exit fullscreen mode
But the optimizer configuration is also part of the system.
A learning rate that is perfectly reasonable under one optimizer can behave differently under another.
Loss decreases but validation quality stagnates
Weight decay becomes interesting.
Because AdamW separates optimization from regularization, you can reason about:
learning rate
Enter fullscreen mode Exit fullscreen mode
and
weight decay
Enter fullscreen mode Exit fullscreen mode
as two separate control knobs.
That conceptual separation is much cleaner than treating “L2 regularization” as something buried inside the gradient.
GPU memory is unexpectedly full
Check the optimizer state.
For a 7B model:
Adam moments ≈ 56 GB in FP32
Enter fullscreen mode Exit fullscreen mode
That number alone can explain a lot of apparently mysterious infrastructure decisions.
You scale the model and everything changes
This is an increasingly interesting research question.
The optimal AdamW weight decay is not necessarily a universal constant that you can blindly copy from a smaller model.
Recent work has explicitly studied how the optimal weight decay changes with model size, dataset size, and training dynamics.
In other words, once you’re operating at serious scale, “just set AdamW to 0.1” is more cargo cult than theory.
9. The big picture: Adam is really a control system for noisy learning
The cleanest mental model I know is this:
A neural network is trying to optimize an absurdly high-dimensional function using noisy measurements.
The raw gradient says:
"Here is what today's minibatch thinks you should do."
Enter fullscreen mode Exit fullscreen mode
Adam says:
"Fine. But I also remember what the gradients have been doing lately."
Enter fullscreen mode Exit fullscreen mode
It keeps track of:
direction -> m
scale -> v
Enter fullscreen mode Exit fullscreen mode
and uses those statistics to construct an adaptive update.
AdamW then says:
"And separately, I want the parameters to decay."
Enter fullscreen mode Exit fullscreen mode
That separation turns out to matter.
So the evolution is roughly:
SGD
|
+-- momentum
|
+-- adaptive scaling
|
v
Adam
|
+-- decoupled weight decay
|
v
AdamW
Enter fullscreen mode Exit fullscreen mode
And the reason this matters for LLMs is not that Adam is mathematically glamorous.
It is that training billion-parameter models is fundamentally an optimization-and-systems problem.
The model might contain 70 billion parameters, but every one of those parameters is being updated by a tiny piece of state maintained over the entire training trajectory.
That makes the optimizer part of the model’s computational machinery.
The next time you see:
optimizer = AdamW(...)
Enter fullscreen mode Exit fullscreen mode
you are not looking at five lines of boilerplate.
You are looking at a compact algorithm that is simultaneously doing:
momentum
adaptive normalization
bias correction
parameter updates
regularization
Enter fullscreen mode Exit fullscreen mode
for billions of variables, potentially millions of times.
That is a rather extraordinary amount of machinery hiding behind one constructor.
One question for you
When you train or fine-tune an LLM, how much attention do you actually pay to the optimizer compared with the model architecture and data?
Your team’s attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I’m building LiveReview, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. Try LiveReview on your codebase:
