Yeah this sort of happened to me too at lunch, just not as bad. The guy was quietly revealing my tricks to the guy next to him. Then he did a trick with a slip force, trying to steal my audience. Lunch ended, so nothing else happened. I guess just some people get jealous of your audience, and try to be a jerk and take over. I would try to ignore him, but if he comes again, get some older magicians on your side to make him get a life (and an audience).