Definitely some very good advice and thoughts have been posted, and they are applicable almost all of the time. However, I think that it is unavoidable that from time to time, there is going to be that person who cares about nothing but trying to show up the performer and prove how clever and smart they are - no matter how well the presentation is structured. In some cases, this may have nothing to do with a deficiency on the part of the performer, but a character trait of the onlooker.
I'm not going to say these people don't exist - but I can say that I never seem to encounter them.
Part of this may be age. If you are older than 25 (or 45), people tend to challenge you less than if you are under 18. Part of this may be how and where you perform. Friends and family are more likely to challenge you than strangers. People you perform for more frequently are more likely to challenge you than someone who has never seen you perform before. People you approach "on the street" or in a bar are more likely to challenge you than the audience in a paid show. People in smaller groups (table hopping) are more likely to challenge you than people in larger groups. Most of those variables are beyond our control.
HOWEVER, the one thing we have control over is the selection, the design and the presentation of our magic.
To break out of the "understood" game of I try to trick you (or "Fool Us") and you try to guess how it is done, you need to convey the sense that what you are about to do is something completely different than the spectator has ever seen and that this person will remember for the rest of their lives. That is the difference between a performance piece and a trick.