For your education and entertainment Sir!
In the heyday of sailing ships, all warships and many freighters carried iron cannons. Those cannons fired round iron cannon balls. It was necessary to keep a good supply near the cannon but how to prevent them from rolling about the deck?
The best storage method devised was a square-based pyramid with one ball on top, resting on four resting on nine, which rested on sixteen. Thus, a supply of 30 cannon balls could be stacked in a small area right next to the cannon. There was only one problem; how to prevent the bottom layer from sliding or rolling from under the others.
The solution was a metal plate called a 'Monkey' with 16 round indentations. However, if this plate were made of iron, the iron balls would quickly rust to it. The solution to the rusting problem was to make 'Brass Monkeys.' Since brass contracts more profoundly and much faster than iron when it's chilled, if the temperature dropped too far the brass indentations would shrink so much that the iron cannonballs would roll right off the monkey. Thus, it was quite literally, 'Cold enough to freeze the balls off a brass monkey.'
And now you know "The Rest of the Story"!