I am reading through the responses and there seems to be lots of well-meaning but sometimes misunderstood information about the cause of moisture/condensation getting into a camera. To understand the risks of getting moisture into a camera while changing lenses it will help to understand how/why the air holds water. The colder air is, the less water it can hold. Warmer air can hold more water than colder air.
Relative humidity is normally expressed as a percentage. A higher percentage means that the air–water mixture is more humid. At 100% relative humidity, the air is saturated and is at its dew point. It is when air is cooled below its dew point ( for ex: by touching a camera body) that you will get water in/on your camera.
The practical effect to you and your camera is that you may have an issue if your camera body is cooler than the outside air temperature. For example, this might happen if your camera was in an air-conditioned house, and you go outside where it is much hotter. When that outside air touches the (colder) camera body, it may cool the air below the dew point and cause water to appear on the camera.
A specific example is: air at 80 degrees and relative humidity 50%. From a chart I see that the dew point is 60 degrees. This means that as long as the camera is OVER 60 degrees, I should be OK. In case you are interested, here is a site to calculate the actual numbers.
http://www.dpcalc.org/If your camera body is below the outside dew point , I would NEVER take off the lens. Exposure to the warmer will cause moisture almost immediately. I would also be careful about "pumping" lens or doing anything that will bring in the outside warmer air into the camera.
In your case, it seems like it was cool out and your camera was probably much warmer than the outside air, so you would have been OK.
Having said that, it is always good to use a quick and careful process to change lenses. Point the camera down. Change lenses as fast as possible. Leave the camera open for under 1 sec, if you can. This will minimize dust and air contaminants from the outside air from entering your camera.
The reverse is also true. If you have been outside in the cold weather and bring a camera into a warm and humid house, keep the camera in an airtight container (camera bag etc) until it warms up to the inside temp. If you do not do this, you may get moisture inside the camera, even if you do not open up the camera. Of course most times your air-conditioned house will have very low humidity, so this may often not be an issue.
Hope this helps.