I'm not entirely sure. I think it's a mixture of a few things really. The first is that, no matter how good it looks and how flawlessly you execute it, a colour change is almost always going to look like a sleight. It doesn't matter if it's a Bertram, a Shapeshifter, an Erdnase or whatever, they all look like you did something clever. The main exception to this is a double lift/top change, where the change can be done away from any other cards, in the spectator's hands, completely fairly.
Another thing is that I don't approve of magic that is about the props over the magician/audience. An oil and water routine, a sandwich routine, a spellbound or coins across, for example. Now it's debatable that this is down to patter and presentation, which is true to a certain extent, but something like Out of this World is always going to be more about the spectator than an Invisible Palm, regardless of how well you dress it up. A colour change is one of those things that is very much about the cards and not at all about the spectator or the magician. This isn't always the case - Derren Brown has a brilliant colour change sequence, for example, that is absolutely about the spectator rather than the cards, but I feel that this is the exception rather than the rule.
Also (and before anybody slates me about this - I know it's a very stupid reason to hate a certain set of moves or whatever, but it does bug me) THEY ARE NOT COLOUR CHANGES! A colour change would be, for example, taking a standard seven of spades and turning it into a seven of spades with red pips. Turning one card into another is not a colour change. It is changing one card into another. There is a difference. Don't call me out on this being a rubbish argument, because I know, but it gets on my nerves quite a lot,
This is a personal choice, colour changes just don't suit me as a performer in the slightest, and I'm not telling anybody else to cut them out of their repertoire at all. I do think that a well placed colour change can be very powerful, but I think that magicians in general are taking the wrong approach - trying to find new and more visual ways of changing a card into another, when by far the best method in my eyes, and the only one I ever use, is a double lift or a top change. If I ever learn a different one (and I do occasionally) it will be purely for my personal satisfaction and not for performances. I can still appreciate a good colour change, and enjoy watching them occasionally and being horribly fooled, but that is purely as a magician, and not something I particularly want my spectators to remember me by.