How dangerous is asbestos?

If asbestos was removed by non professional, who did not know how to handle it, how dangerous is it for people who will move into the house? Should any special cleanup need to be done? We are buying house and owner removed asbestos himself, instead of hiring professionals. Should we just walk out of the contract? I am afraid that no amount of cleaning will guarantee that asbestos is gone , and we have a little baby.