Lil, I suspect that this saying is about as old as the invention of spectacles. And the first spectacles were quite ugly.
Also, notice the expression refers to "Boys" and "Girls", not men and women. What the hell do little boys know anyhow???
I don't think there is any reason for women to fear being rejected for wearing glasses. I'd hit on you in a heartbeat Lil, with or w/o the glasses.

Joe