I take mine off if I put hand lotion on in the middle of the day, but if I put body lotion on after a shower, I don't bother taking it off. The only time I've taken it off are when I've used very harsh cleaning chemicals and even then it was mostly because I didn't want to scratch up the gold, not because of the stones. Otherwise, I wear it all the time-- handwashing, dishwashing, gardening, swimming, everything. I don't think you have any reason to fear the stone would fall out and the only reason not to wear it while you put hand lotion on is that it will dull the setting until the next time you wash it. I find that just having mine on when I wash my hair every day keeps it clean enough.

What sort of metal is the setting?