I want to quickly address also the advice that you should simply walk away. Sciffydof, whilst I don't agree with you per se, you brought up an interesting point earlier. What if you're in a professional gig?
Walking away I feel is an approach that should be exercised with great and cautious judgment. It would be hypocritical of me not to point out that, if you walk away, then you're also giving up on entertaining the majority. In a general street magic scenario, I would only exercise this option if I felt it highly likely I would be asked to come back. In other words, rapport with your audience is key. IF you have a strong enough rapport, they will do all the work for you. Actually, if you have a strong enough rapport with your audience, you shouldn't need to take any action at all. I've performed in many situations over the past few years for a group of around two dozen people with a heckler involved - and the entire crowd inevitably tells the heckler to stop, without me needing to say a word. This sort of thing is much more effective coming from their friends because, whilst a generalisation, it is often true that hecklers do care about their image. And coming from people who don't have the spotlight, and who they're familiar with, it is all the more effective. You'd do something more readily if told by a dozen best friends than by a random intruder.
In a professional situation is interesting. This also applies to the above, but I would simply mention that you are being paid to entertain people here, but quite obviously, people are not obligated to watch if they don't want to. I would then simply explain that I will be performing at this event for the next so and so time period, walking around to different groups, so please feel free to follow, or engage me at another time or place. Some people just don't want to be entertained. That's fair enough, and you have to be a professional in the sense that if they don't want to, you can't force it. But at the same time, you let the majority know that you are available whenever they want to, and since they want to, they are basically given every opportunity to still be entertained and enjoy your magic. If the booker asked you about a situation, you would simply explain that one spectator did not feel like entertainment, so you moved on to other people. If you're good, and haven't screwed every single performance up, that will be more than adequate an explanation - especially when everyone else sends the praise in.
In case anyone hasn't realised, and is wondering what approach I personally take, to the greatest extent possible, I let the spectators do the work. I work very heavily on my communication and rapport building skills everywhere I go. It has gotten me acting jobs, magic jobs, interviews, elected official positions, and all sorts of things. And it is one of the most important aspects of the performance of magic, intrinsically tied in with persona and presentation, which, as most of you will know, I champion very strongly.
On the rare occasion this doesn't work (I don't think this has ever not worked for me), then perhaps a more direct approach such as romeo's is applicable. Whilst it does confront the heckler, it is done as a warning rather than as a challenge, and I like that again, it uses the idea of peer pressure more than anything else. Again, avoid direct challenges is the key. The heckler needs to know what is right and wrong within the context of a performance. A quiet, polite line should be enough as well. Following a polite, but firm and direct request to stop what we would call heckling, you just have an extreme case of someone without manners, and they will come across as d*cks, and everyone will know it. I would either default to romeo's approach, or default to my approach in professional gigs detailed above. At an actual gig, I would perform to this group later, in a different place, minus the heckler, or have them removed if appropriate (bar, etc.). Suffice to say that I have never had this happen - a continuous heckler despite a firm request to stop from you and other spectators. This is such a remote case, that it would almost constitute an extreme situation - see below. But no heckler enjoys being disliked by his peers. And if you've just got some pathological demon, well, see below. But short of having a pathological demon, everything detailed above will prevent this situation from happening. However you do need to establish your authority as the performer, after you meet everyone and start performing. Note, of course, that this only works if you're doing a good job of performing. If no-one really cares, then you leaving won't be a loss, so they won't be moved to move in your favour. Don't neglect your job.
In very extreme cases, this obviously may change - but then again, all the advice I've given above should fit into everything but the most extreme of cases. And when I say extreme, I will refer to the earlier example of the owner who pulled the literal plugs of the band performing. Or, the pathological demon mentioned above. When you get to that extreme, the only real possibility (in magic terms, it would be someone knocking cards out of your hand physically, or something similar) is indeed to confront them and end the performance. This is slightly contradictory to what I've said earlier, but there's really not much you can do. If you're at a gig, speak to the booker or security. If you're on the street, stand up for yourself foremost, think about entertainment afterwards. Stay safe, because they've forcibly removed the entertainment factor in an irrevocable way. You can perform to other people later - however high your goal is, you can cop one performance in your lifetime. There's not much you can do here. The only way you can win from this situation is to do well despite the guy, and the only way you can do that is to move on.
One last thing. Someone mentioned taking them aside, and asking them why they did it. I recall someone on here a long time ago providing the line, something along the lines of, "I see you're knowledgeable in card magic too. You know what, let me entertain these guys since they're obviously having fun, and we can have a more in depth discussion about the technical side of prestidigitation and subterfuge later, alright?" And then continue. Which is really quite interesting. The problem is though, given the generally young demographic of the guys here... Don't tell me that doesn't sound cheesy! Can you really imagine walking up to a heckling kid after and asking, "Why did you do that?" And good luck getting an answer! It's just not a practical response in my opinion because too many lack the maturity and age to do so. Richard Osterlind could get away with it. A kid with a backwards baseball cap, not so much.
My conclusion regarding a solution? Rapport works best. And you should be aiming for that anyway. Be more direct only if necessary, but never in a challenging way. Warn without challenging. And think about how you warn - It's the difference between saying "Please don't do this" and "Please don't do this, or else." The former is workable, the latter, bad.
I hope this has been mildly useful to someone - and even if you don't agree, please think about it anyway.