Worried about asbestos ?

Basically im a landscapers apprentice and today i was digging a hole in someones back garden for a plant and as ive dug down ive started to hit bits of hardcore, some big, some small, but all of the same stuff. I started pulling it out and it was all bits of roof tiles, but i still wasnt entirely sure what it was, so i said to my boss as he walked past whats this and because he was doing something didnt really pay that much attention and said 'i dunno' or something and carried on, so i was like okay then, just carry on. So i carry on digging but as i am, it seemed the stuff was breaking in the ground, not like getting crushed up with loads of dust coming out every where, just like snapping in half where the spade was hitting it. I then said again to my boss, what actually is this stuff, and he came and looked and said ohh concrete asbestos (well i think thats what he said, he said something along them lines) roof tiles, yeah they used to use it on shed roofs and stuff. So i was like, what ! asbestos ?! F*** this i aint doing that no more and he was like nah dont worry about it, that'll be fine, trust me, im a landscape gardener, ive seen it all before blah blah. But you know, im a little uneasy as soon as i hear the word 'asbestos' but still i thought hmmm, might aswell just carry on then. Now what i want to know is, was my boss right to say dont worry about it ? Also should i really be worried about this ? Thanks !