Real Estate contract?

We got into contract to buy house. House inspection showed that there is Asbestos in the basement. Owner agreed to hire the company and remove Asbestos. Instead of hiring professionals the owner did remediation by himself. We were worried that Asbestos is still in the house and hired Asbestos abatement company to perform tests. Test came back positive on two spots on the carpet and wall. Air quality test came inconclusive. Naturally we do not want to buy the house with Asbestos all over especially because we have 1 yo baby. The owner does not want to cancel and does not want to give us money back. If we will walk away from this point, will our down payment be returned to us (money in their lawyer's escrow account). I am ready to leave earnest money to them. Can I get out without going to court?