I have had the same problem, mostly with Halo 3, although I have had the same problem with CoD4. It does tend to be the same friend. My guess is that it may have something to do with his Internet connection.
Does anyone know if different types of Internet connection will not allow you to join other parties on Live?