If color wasn't that important how do you explain lakes that have particular colors that work best. Colors and contrast that show best on bright days, dark days, and up and down the water colume really do make a difference. After reading thousands of kokanee reports over the last 15 years it's obvious color matters.
I've had days where both sides of the boat had the same lure, speed, set-back, scent, dodger, leader length, corn, ball trolls, depth, etc. One side gets hammmered... 5 fish one right after the other. The other rod got absolutly zero action. The only difference was the lure color. Once that rod got changed to the same color we couldn't keep the fish off. This was over a large school during a HOT bite... color absolutly mattered that day.
To answer the question "if it was truly the color, wouldn't we all have the same favorite color?"... look at the success of pink.