In a strictly legal sense, in the United States you don't have to ask permission to take a picture of a person in a public place. Some photographers prefer to ask permission anyway, which is the most polite way to go about taking pictures, but it will change the behavior of the subject in the photo. It's up to the photographer which he or she prefers.
In any case, it's best to not be the creepy guy with a long lens hiding behind a potted plant taking pictures of people as they window shop. Also, if you follow the above tips and stick with one camera attached to your wrist and one spare lens in your pocket, there's no reason to sport a fishing vest full of equipment, which is going to make people wary of your intentions.
Instead, be the guy or girl with a smile and a camera and most people won't think twice about you or your hobby -- which is how you'll get those fantastic photos of people living their lives in public spaces.
There's just one more tip on being unobtrusive on the next page.