Home >  Term: precision
precision

A quantitative measure of the density of representable numbers. For example, in a binary floating point format that has a precision of 53 significant bits, there are 253 representable numbers between any two adjacent powers of two (within the range of normal numbers). Do not confuse precision with accuracy, which expresses how closely one number approximates another.

0 0

Creator

  • Shemayah
  •  (Bronze) 432 points
  • 100% positive feedback
© 2024 CSOFT International, Ltd.