The main thing with balancing on your hands is to find your sweet spot, which means to center your body's weight as directly over your hands as possible. You'll get better at finding this spot with practice.

Basically, you always have to feel as though you're barely not falling over (to the back). Don't be afraid to take that swing with your legs, because you'll probably never find your equilibrium otherwise.