Before we discuss guilt, it is interesting to note that the first time the Bible uses the word adultery is in the Ten Commandments (Exodus 20:14). It is used a total of forty times in the KJV translation.
It is amazing how human we are and how we all generally think the same. As you know, the apostle Paul clearly said we all delight in the law of God after the inward man (Romans 7:22). The problem is we also have the law of sin, the same law that led to adultery, in our members as well.
Paul knew quite well the struggles all humans have against sin (Romans 7:14 - 15, 18), including adultery. We ALL sin according to the Bible (Romans 7:23). Even though God does not grade sin, we humans tend to do so. Christ said He forgives ALL manner of sin! The answer to Apostle Paul's question as to WHO will save him (and us) from a body that wants to lead us to death is Jesus Christ our Savior (Romans 7:25)!
Overcoming guilt is a matter of faith in Christ. The reality is that your guilt means you doubt God forgives you. Intellectually, of course, you know he does. The problem is that in your heart you believe that the sin of adultery is too evil! You are grading sin which God does not do.
If you believe we all continue to sin (besides the one you mention) even after we ask for forgiveness then you should feel guilty about them all! Since only this sin, adultery, leads you to guilt you have the common human problem of self-righteousness.
Paul said the law of God was given that we might realize how much sin we commit (Romans 5:20). Either this is your only sin that is bad enough for you to feel guilty over or you do not understand how sinful you really are (which is self-righteous). All sin is equal, and therefore, we should continually see our guilt and sinful nature and know we look to Christ to forgive us.
Pray for a deeper repentance and have the faith that God forgives you, and all of us, regardless of the type of sin. Until you forgive yourself for the adultery, you really do not believe Christ's blood has paid for that sin and your guilt will remain.