I often remove them when practicing, but if I know I will be in a situation where I may be asked to perform, I will keep them in the deck. They're actually a great excuse to (secretly) cull multiple cards to the back of the deck whilst in the disguise of up-jogging and removing them. Before setting them down, you can also plant an indifferent card face-down between them and set them to the side. Any time you wish to go into a sandwich routine a few effects down the line, they'll swear you never went near the Jokers - yet there's one card in between.
I keep them in the deck so I can remove them when culling force cards after a spectator shuffle, I also use them for a deck switch technique from card college.