The Trichromatic Theory proposes that humans have three different types of cones (photoreceptors) that each respond at maximum levels to different wavelengths. These photoreceptors are S-cones that respond best to short wavelengths, M-cones that respond to middle wavelengths, and L-cones that respond to long wavelengths. Now one argument for the the idea that people see color differently is because of the phenomenon we know as color blindness.
Because of the three different cones, when one cone is affected respectively, it creates a different spectrum for the viewer. The three different types of cones affected produces Tritanope or absence of S-cones, Deuteranope or absence of M-cones, and lastly Protanope or absence of L-cones. The picture below depicts what each of these types of color blindness appears like to the observer.
Unfortunately for my ex-boyfriend, he was color blind and had an absence of M-cones. This made it hard for him to differentiate between some shades of orange, red, green and brown. Also unfortunate for him is that I am mischievous at heart and used to always attempt to fool him in figuring out what colors things actually were. When choosing outfits I would convince him that those green pants absolutely went with that red shirt, so on and so forth. For him, everything looked what I would describe as murky shades of earth colors. In more serious instances sometimes he would have a hard time when looking at stop lights, I obviously helped him when he struggled there.
Colorblindness varies among individuals and while some struggle with stop lights, difficulties are different between each type.