How come when I'm talking toy friends on xbox live their voice comes out of the speakers not my headset?

I can talk through my headset but not hear my friends voices out of it.