Physicist: Beautiful question!
The short answer is that if you have a warm bubble of air it will rise, but as it does it’ll expand in the lower pressure environment, which causes it to cool.

Weather balloons are intentionally under-filled on the ground so that when they’re at altitude they’ll be the right size. Shown here are professional weather scientists (left) and other (right).
If you’re not familiar with the ideal gas law (compression heats, expanding cools), then here’s a better way to think about it. Gases (which includes air) can be though of as a hell of a lot of perfectly bouncy rubber balls. Each molecule is ricocheting very fast between other molecules and the ground. But, like anything that moves, when they move upward they slow down.

Air molecules are like rubber balls. They move faster when lower and slower when higher. Air not shown.
This random movement is what we in the biz’ call “heat energy”. Heat is literally just the random motion of atoms. When an air molecule is close to the ground it’s moving pretty fast, but if it has a chance to bounce higher, then it will slow down during the trip. When it’s a little higher, all of the other air molecules around will have had similar experiences and will also be moving a little slower. Slower motion = lower temperature.
You can extend this a bit and say that air near the surface of the Earth is moving about as fast as it would be if it had been dropped from the top of the atmosphere (ignoring air resistance, ironically), which is about half a kilometer per second at room temperature. This is a little ballpark, because the atmosphere is more complicated than air just being stacked up.
When a ball or air molecules are bouncing, they exchange “kinetic energy” for “gravitational potential energy“, but overall the total energy stays the same. So air actually maintains the same amount of total energy regardless of height, it’s just a question of how much of that energy is tied up in gravitational potential (being up high) or kinetic energy (moving fast). When a vertical column of air has the same amount of energy, then “bubbles” of air are free to move up and down, and those bubbles will have the same temperature of the surrounding air when they arrive.

An inversion layer, where the temperature suddenly increases as you go up, stops air from mixing properly.
However, sometimes this isn’t the case. Temperature drops as you go up, but hot air still rises through cold air. If you’re moving upward and find that suddenly the temperature increases (instead of slowly decreasing), then you’ve hit an “inversion layer“. This happens a lot in valleys and canyons. The warmer layer has more energy than the colder layer just beneath it (so it’s hotter and lighter), and this stops the air layers from mixing. The cooler air doesn’t have the extra energy needed to rise through the warmer air above it, so the air layers stack up like oil on water. Since the layers are just sitting on top of each other, they’re free to behave very differently. If you’ve ever been outside and seen clouds at different altitudes moving at different speeds or different directions, then you’re probably looking at clouds separated by an inversion layer or two.
The rubber ball picture is from an over-the-top ad, that can be found here.
10 Responses to Q: If hot air rises, why is it generally colder at higher elevations?