The solemn day of Yom Kippur is always kept in the seventh Hebrew sacred calendar month (Tishri) from sunset on the ninth day to sunset on the tenth (Leviticus 23:32). It was a time when an annual ritual, performed by the High Priest, symbolized the taking away of the the sins of the people (Leviticus 23, Hebrews 9:22).
The first place that the word "atonement" is used in the KJV Bible is in the book of Exodus.
And Aaron and his sons shall eat the flesh of the ram and the bread that is in the basket by the door of the tabernacle of the congregation. And they shall eat those things with which the atonement was made to consecrate and to sanctify them. But a stranger shall not eat of it because they are holy (Exodus 29:32 - 33).
Aaron (the brother of Moses), in the symbolism of Yom Kippur, represents Christ (Leviticus 16:4 - 6). The Most Holy Place, also known as the 'Holy of Holies," represents God's throne in heaven. Since Aaron, as a human being, is of himself unqualified to stand before God he must be ceremonially cleansed and have his sins forgiven.
A most unique event
One of the most unique events in the Old Testament occurs on Yom Kippur. Aaron (and those High Priests which came after him) were commanded to randomly select (via lots) one goat to represent "the Lord" and another one to represent "Azazel" (Satan the devil, the being chiefly responsible for our sins). Casting lots is an ancient way of allowing God to decide something rather than imperfect man (Leviticus 16:8, 10, 26).
The centerpiece of Yom Kippur begins when the Azazel goat has the sins of the people placed on it by Aaron. It was then released into the wilderness. The goat that represented Jesus Christ was killed as a sin offering. The blood from the sacrifice was then taken by the High Priest into the tabernacle's (later, the temple's) Holy of Holies.
The High Priest was allowed, only once a year during the Day of Atonement (Yom Kippur), to enter the most holy section of the entire tabernacle (temple). Once in the Holy of Holies, the priest sprinkled the blood from the sacrifice on the mercy seat of the Ark of the Covenant. This represented Christ, on the Sunday after his resurrection, presenting his blood as a sacrifice for all sin to God in heaven.
The primary reason true Christians fast on Yom Kippur is that it is commanded by God. The day focuses the mind on what the Eternal is doing and reminds believers about his great mercy and compassion for all people. On the day of Atonement believers fast for a full 24 hour period.