On the PC, you holster your weapon by holding your reload button (default 'r') for a few moments.
Its the same button you use to quickdraw your equipped weapon. You hold down the button for about as long as you would hold the PipBoy button (default 'tab') to turn on the flashlight.
On the XBOX360 and PS3, you hold down the button that you use to reload. Holding this button will also make you pull out you pull out your weapon once it is holstered.