Should I be concerned about asbestos exposure?

After a recent house fire, I discovered that the house my family was living in had asbestos. I have noticed that since we have moved out, many of our respiratory issues don't seem so bad. But is this something that builds up in your system?