You'll need to remove the colors you don't want from every size folder, otherwise you'll get Sims in dresses that look like they're made of skin, and that's just awkward. It's a little inconvenient, but I figured a little inconvenience was worth saving hundreds of outfits' worth of space.