Well, the answer to your final question is... In the Avengers movie: Because there are no X-Men or Fantastic Four in this universe. In the comics: Sometimes you do see that. The Avengers will show up to help out the FF, and Reed will say, "We've got this one, guys." (And for some reason the Avengers back off.) Or sometimes the two teams team up. And sometimes the excuse is that the FF are off on some mission in outer space. (Even if in their own comic that month they're fighting the Ringmaster and his Circus of Crime in Queens.) And sometimes they just ignore this question. There's a certain amount of disbelief that you have to suspend for this kind of stuff. Me, sometimes I can do that and sometimes I can't. For example, in Superman, I was okay with the flying and the x-ray vision and the space capsule from Krypton, but not with him reversing time by flying backwards around the planet.