My own personal take on this would be and if I was learning English from somebody from one of the seven English speaking countries, I wouldn't mind which one of these countries the speaker came from as long as that person's dialect was not overly strong to their particular region. Every country has regions where the accent is much stronger than in other areas. In other words, a much more important factor would be whether the person in question has a neutral accent or not.
Also, in terms of Americans saying they can't understand other native English speakers. Well a small part of that may be due to ignorance (Sorry US folk but it's actually not all that difficult). For the most part though, it maybe down to the fact that Americans are exposed to other accents on a much less frequent basis. For instance, on American TV, the US accent dominates the airwaves whereas in other English speaking countries you are regularly exposed to a much much more even keel of accents. Also, the fact that US folk don't tend to travel outside of America all that much in comparison to their native English speaking counterparts may also be another factor. I don't know how reliable this figure is but it's often cited that 75% of Americans don't own a passport. Whereas in places like the UK, this figure is as low as 29%.