'Need to' overstates how to be successful in digital photography. Yes, your 'best' results are likely at the base ISO, but your composition may not allow for this limitation. It may be too dark to work at ISO-100. Your subject may be moving too fast to allow for a shutterspeed that exposes correctly for ISO-100. Your selection of an aperture may be too wide or too small to expose correctly at ISO-100.

So, if you have an option between say ISO-400 and ISO-100, say whether to use a tripod or a stabilized lens, if these tools let you drop the shutterspeed and the ISO down to ISO-100, go with the lower ISO. If not, fear not, in 99% of the situations, it's immaterial.

Bill - all digital cameras capture at their own 'base ISO'. Most cameras this is ISO-100, some might be higher, such as ISO-125, some all the way to ISO-200. The image is then amplified (or downgraded) to the ISO specified by the exposure setting, either the human or the camera's calculation. The numbers below the base ISO are just more trickery of digital photography, not the 'base ISO' of the camera. How each camera does this is proprietary. The only issue is the result maps to the ISO international standard -- International Organization for Standardization, a nongovernmental organization that publishes standards.

So, the base ISO is the unamplified sensitivity of your camera’s sensor. Your reference to the lower values, below ISO-100 are not the base, unless confirmed by the manufacturer. ISO-080, 60, 50 are 'expanded' values. These expanded values reduce the dynamic range capability of the camera, just a values higher than the camera's normal range, such as higher than ISO-3200, check your individual user manual. When you use a lower ISO setting than your base (i.e. ISO 50 instead of ISO 100) such as to achieve a slower shutter speed, your camera sensor will digitally expose your image at ISO 100, and then the processor will reduce that exposure to simulate what it would look like at ISO-050.

The result? You can potentially lose contrast and detail in the highlights and reduce your overall dynamic range since the image is purposefully overexposed. Highlights that are on the cusp of being clipped at ISO-100, when you meter the scene, will likely be blown-out from the exposure being brought down.