It depends on what you're doing. Lots of providers tend to throttle their internet traffic on specific users if they use a lot of bandwidth. Stuff like p2p filesharing or using torrents (especially for movies) will easily cause your bandwidth to get throttled, especially if you're using a provider like Comcast. They're currently under fire for doing this, but until the outcome it looks as though they might continue to do this.
There are other things to consider as well, such as the use of a hub instead of a switch, multiple wireless connections on one wireless router/a, or multiple continuous sessions accessing the same outbound connection (aka, one computer streaming music and another downloading files at the same time).
