I use Google Wallet and actually think it's less risky then using your card just about anywhere.
What the most unsafe place to use a credit card.... a restaurant. You willingly give your card to someone you probably don't even know, they walk away and charge your account for the meal. Did they write the card number, expiration date and security code down? You may never know, months from now they can use that information and you will never know who got the number
I never load more than $50 on the Google prepaid card so if some lucky person does steal my phone they aren't getting much