Glossary Entry

Learning Rate

A training hyperparameter that controls how large each parameter update is during optimization.

Training Optimization

Also called: learning rates

Seed source: Google ML Glossary

The learning rate determines how aggressively training updates the model after seeing new information. If it is too high, optimization can become unstable; if it is too low, training can become painfully slow or stall.

This tradeoff appears in posts on transfer learning, policy gradients, and neural nets more broadly. Small changes to the learning rate can materially change whether training converges well or falls apart.