How to Count Parameters in Artificial Neural Networks (ANNs)

When building neural networks, one of the first questions you should ask is:

How many trainable parameters does my model have?

The number of parameters determines:

• Model complexity
• Memory usage
• Training speed
• Risk of overfitting

In this article, we’ll learn how to calculate the number of parameters manually and verify the results using TensorFlow 2.x (Keras).

What Are Parameters?

Parameters are the values that the neural network learns during training.

There are two types:

• Weights
• Biases

Every neuron has:

• One weight for every input it receives
• One bias

Therefore, for a layer with:

• Input features = n
• Neurons = h

Example 1: Single Hidden Layer

Suppose we have:

• Input features = 4
• Hidden neurons = 5
• Output neurons = 1

Architecture:

Input(4)
      │
Hidden(5)
      │
Output(1)

Enter fullscreen mode Exit fullscreen mode

Hidden Layer

Each of the 5 neurons receives 4 inputs.

Weights:

4 × 5 = 20

Enter fullscreen mode Exit fullscreen mode

Biases:

5

Enter fullscreen mode Exit fullscreen mode

Total:

20 + 5 = 25

Enter fullscreen mode Exit fullscreen mode

Output Layer

Input = 5

Output neurons = 1

Weights:

5 × 1 = 5

Enter fullscreen mode Exit fullscreen mode

Biases:

1

Enter fullscreen mode Exit fullscreen mode

Total:

5 + 1 = 6

Enter fullscreen mode Exit fullscreen mode

Total Parameters

25 + 6 = 31

Enter fullscreen mode Exit fullscreen mode

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(4,)),
    tf.keras.layers.Dense(5, activation="relu"),
    tf.keras.layers.Dense(1)
])

model.summary()

Enter fullscreen mode Exit fullscreen mode

Output:

Layer (type)      Output Shape     Param #

dense             (None, 5)          25
dense_1           (None, 1)           6

Total params: 31

Enter fullscreen mode Exit fullscreen mode

Perfect match.

Example 2: Two Hidden Layers

Architecture:

Input(8)
      │
Hidden(16)
      │
Hidden(10)
      │
Output(3)

Enter fullscreen mode Exit fullscreen mode

First Hidden Layer

Input = 8

Neurons = 16

Weights = 8 × 16 = 128
Biases = 16

Total = 144

Enter fullscreen mode Exit fullscreen mode

Second Hidden Layer

Input = 16

Neurons = 10

Weights = 16 × 10 = 160
Biases = 10

Total = 170

Enter fullscreen mode Exit fullscreen mode

Output Layer

Input = 10

Output neurons = 3

Weights = 10 × 3 = 30
Biases = 3

Total = 33

Enter fullscreen mode Exit fullscreen mode

Total Parameters

144 + 170 + 33 = 347

Enter fullscreen mode Exit fullscreen mode

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(8,)),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(10, activation="relu"),
    tf.keras.layers.Dense(3, activation="softmax")
])

model.summary()

Enter fullscreen mode Exit fullscreen mode

Output:

Layer (type)      Param #

dense                 144
dense_1               170
dense_2                33

Total params: 347

Enter fullscreen mode Exit fullscreen mode

Again, the manual calculation matches TensorFlow exactly.

Example 3: Deep Neural Network

Architecture:

Input(20)
      │
Hidden(64)
      │
Hidden(32)
      │
Hidden(16)
      │
Output(5)

Enter fullscreen mode Exit fullscreen mode

Layer 1

(20 × 64) + 64
= 1280 + 64
= 1344

Enter fullscreen mode Exit fullscreen mode

Layer 2

(64 × 32) + 32
= 2048 + 32
= 2080

Enter fullscreen mode Exit fullscreen mode

Layer 3

(32 × 16) + 16
= 512 + 16
= 528

Enter fullscreen mode Exit fullscreen mode

Output Layer

(16 × 5) + 5
= 80 + 5
= 85

Enter fullscreen mode Exit fullscreen mode

Total Parameters

1344
+2080
+528
+85
------
4037

Enter fullscreen mode Exit fullscreen mode

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(20,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(5, activation="softmax")
])

model.summary()

Enter fullscreen mode Exit fullscreen mode

Expected output:

Layer (type)      Param #

dense              1344
dense_1            2080
dense_2             528
dense_3              85

Total params: 4037

Enter fullscreen mode Exit fullscreen mode

Why Don’t Activation Functions Add Parameters?

Layers such as:

Dense(32, activation="relu")

Enter fullscreen mode Exit fullscreen mode

or

Dense(10, activation="sigmoid")

Enter fullscreen mode Exit fullscreen mode

have exactly the same number of parameters.

Activation functions like:

• ReLU
• Sigmoid
• Tanh
• Softmax

perform mathematical operations but do not learn any weights or biases, so they contribute zero trainable parameters.

Quick Reference

Layer Formula Dense (inputs × neurons) + neurons Dense (alternative form) (inputs + 1) × neurons Biases One per neuron Total Model Parameters Sum of all layer parameters

Key Takeaways

• Every Dense layer learns weights and biases.

• The parameter count for a Dense layer is:
  

  Parameters = (Input Units × Output Units) + Output Units

Enter fullscreen mode Exit fullscreen mode

• Equivalently:

  Parameters = (Input Units + 1) × Output Units

Enter fullscreen mode Exit fullscreen mode

• The output size of one layer becomes the input size of the next layer.

• The total number of trainable parameters is the sum of the parameters across all trainable layers.

• You can always verify your manual calculations using model.summary() in TensorFlow 2.x with Keras.