When fragrances make a person physically ill - YES. Its the same as smoking. Its also called consideration. If I didn't have this proven medical problem and worked with someone who couldn't handle fragrances, I wouldn't think twice about not wearing it.
I'm sure if you asked your co-workers not to wear perfume because you are deathly allergic to it, they would refrain, out of common decency. HOWEVER, if you are indeed deathly allergic to fragrances, you'd better be taking some kind of medication, because the world will not stop for you. YOU are the odd man out, not the other way around, dearie. It's up to YOU to protect your health..not me. Of course, if we worked together, I would never wear perfume...but I wouldn't be happy.