The exact Bible verse your pastor is referred to, in regard to women only allowed to wear dresses and not pants, is Deuteronomy 22:5. It states, "Anything that pertains to a man shall not be worn by a woman, neither shall a man put a woman’s garment on. For all that do so are abominable to the Lord . . ." (Deuteronomy 22:5, HBFV)
As a woman, I understand how you feel. I was also quite confused a while ago when the church I attended also taught that women should only wear dresses and not pants. This teaching forced me to do some research on the subject.
Ask yourself the following. What were ladies wearing in the days that the Levitical priesthood was set up and the Law of God was written? Obviously, there had to be some differences between a man's robe and that of a woman. Do most females want to look like men? I don't think so. It is therefore safe to assume that some robes would look a bit more feminine than other ones.
Applying sound principles
One of the basic principles of Bible study is to note what a verse says and what it does not say. Deuteronomy 22:5 does not give us any details regarding how clothes should cover the male or female body in Moses' time or throughout history. It does not say men should wear short sleeves and women longer ones. It does not state how long clothes for each sex should be or even if patterns on the cloth are acceptable for one sex and not another.
Implicit in God's Deuteronomy command is that the sexes should be distinguished from each other in regard to how they dress. The details of how this is manifested are based on the commonly accepted norms of the society and culture a person lives in.
It should be noted that the difference between the dress of men and women, in Moses' day, did not revolve around whether cloth was used to distinguish each person's legs (pants) or not (dresses). It centers on something a bit different. The book "Manners and Customs of Bible Lands" by Fred Wight states this distinction.
"The dress of women was different in detail rather in kind. They (women) also (like men) wore a tunic and cloak . . . Doubtless they wore longer tunics, larger mantles than their menfolk."
Women were also sometimes distinguished in that they, at times, wore veils (see Genesis 24:65) while men did not. There were also other visible distinctions of sex such as females wearing makeup and generally placing on their bodies more ornamentation than males.
Women, in most western societies, have been wearing slacks (pants, jeans, etc.) for many years. This type of clothing is commonly accepted for both sexes. While some ladies may prefer to wear dresses, not doing so is not prohibited in the Bible.