FTP Upload - Big Problem!

[REngels]

Ok. I've been very impressed with the quality of the responses to other people's questions, so here goes...

I have a small network, running behind a DSL router. The network consists of some Windows 2000 PCs, and OS9.1 Mac and some OSX Macs.

The router is performing NAT, so there is only 1 real IP address. Because of this, the computers behind the router obviously need to use passive FTP.

All of the machines work fine except for the OSX machines! The FTP works fine for downloads but hangs on an upload! Usually, the resulting file is of 0 bytes. The hanging symptoms are similar to if I had passive FTP turned off.

I have tried several FTP clients on the OSX machine: Transmit, Fetch, and the command line FTP client that comes with OSX. All exhibit the same behavior.

<B>Here's the rub... if I run a OS9.X FTP program under the Mac Classic mode, it works fine... (as long as I use passive FTP)</B> For example, Fetch 3.0.5 works fine.

A few last notes... this problem only appears when accessing FTP sites beyong the router (i.e on the Internet). If I access an FTP server on the local network (one of the Windows PCs) from the OSX machine, everything works fine. Also, I've tried accessing a variety of extenal FTP servers (Unix, Windows) from the OSX machines, and they yield the same upload problem.

Under OSX, I've tried setting the passive mode in both the client application and in the network preferences. Nothing seems to help.

Also, I am running 10.1. but I think the same problem was there under 10.0.x. My OSX machines are pretty generic... Powerbook G3, and Titanium G4.

I am really at my wits end, and any help would be greatly appreciated.

Thanks,
Robert Engels

[Soapvox]

I don't have any idea of what I am talking about so if you feel like trying it cool, if not I understand.

By educated guess is that the port for ftp is not set up to allow passive connections, I would suggest downloading brickhouse from version trackers and fooling around with it to try to setup ftp port to work with passive, the setup tool is very easy to use, but also getting intothe guts is pretty easy as well, I hope this helps, if I am full of crap please someone tell me so I can learn thanks!

[REngels]

Soapvox, thanks for the input, but I'm pretty sure that's not it.

The FTP Server (located on the Internet) does support passive connections, since that is how the other non-OSX machines connect.

Also, I've used 'ipfw' to make sure there were no ports beng blocked. I have no other firewall software loaded.

Robert

[Klink]

Hi REngles,

I think your right, this passive ftp upload problem has been around pre 10.1. I remember running into this some time back and quickly placed blamed on a third party app for this error. And in some respects it turns out to be somewhat true.

I'm able to reproduce this passive ftp upload problem with Transmit and using ftp from within Terminal. Using tcpflow, I was able to find where and possibly why this upload error occurs. The freeze is caused by a server error response to the TYPE command that's initiated after the user issues the PUT command to start a file transfer to the server. It seems uploading in ftp passive mode will only work if the file TYPE is set to binary image. Set it to any thing else and the server will throw a 5xx type error and pause (freeze occurs on the client side). In an active ftp session, in which both tcp ports 21 (commands) and 20 (data) are used, the default file type for data transfer is acsii. And this works as expected. In a passive ftp session the file transfer type for Mac OS X should be binary image (contrary to RFC959). This switching from acsii to binary is where things go doo doo for some ftp clients. In my case Transmit and Terminal ftp if not properly set.

I'll try to explain how this affected some of your findings mentioned in your first post.

-"The router is performing NAT, so there is only 1 real IP address. Because of this, the computers behind the router obviously need to use passive FTP"

This is not necessarily true. NAT'ing will not hamper the proper flow of an active ftp session initiated by a client behind the router. It will hamper the flow If say you had an ftp server setup behind this router on your LAN using NAT. Then the hiding of the servers private address would be an issue if your router did not perform some sort of reverse proxy for your LAN ftp server. If your ingress ports 20 and 21 are open, active ftp session should be no problem.

-"The FTP works fine for downloads but hangs on an upload! Usually, the resulting file is of 0 bytes"

The resulting file can sometimes be seen because the LIST command is given from the client to the server before the TYPE command which freezes the session. The server will retain the file name of the yet to happen transfer regardless if it has failed or succeeded. The remnants of this is what you see on the server's target directory. Very similar to what happens with tftp and the touch file, except without the errors.

-"I have tried several FTP clients on the OSX machine: Transmit, Fetch, and the command line FTP client that comes with OSX. All exhibit the same behavior."

I've noticed this with Transmit. I can say that Transmit doesn't seem to give this file TYPE binary command when set for a passive session.
The command line ftp program you run in terminal defaults to file TYPE ascii and session mode active. You have to physically set these parameters to passive and file TYPE binary to have successful uploads in passive mode.

-"this problem only appears when accessing FTP sites beyond the router (i.e on the Internet). If I access an FTP server on the local network (one of the Windows PCs) from the OSX machine, everything works fine. Also, I've tried accessing a variety of external FTP servers (Unix, Windows) from the OSX machines, and they yield the same upload problem"

I can see your situation being this way if your router is set to block ingress port 20 (not egress 20. Your egress ports should be somewhere in the 4k's). Perhaps the ftp session to your winbox is not passive after all but active, or the Win box doesn't mind the TYPE I command and accepts the acsii transfer without error. You would need to sniff around ports 20, 21 for the answer if all of the above is not doing it for you.


Hope this helps. And please let us know of your outcomes.

[REngels]

Klink,

Thanks for the suggestions but I think you might be mistaken on a few issues...

In general, I do not believe it has anything to do with binary versus ascii. I have manually set the mode to binary in a variety of clients with no difference in the result. See below for what I think it might be...

As for the other issues...

Machines using NAT must use passive FTP, unless the Router does some advanced FTP packet sniffing. Without passive FTP, the client machine sends the FTP the IP address and PORT the server should contact the client on (using the PORT command) for the data channel. Obviously, the client machine has a local non-routable address, so the information contained in the PORT command is invalid - that is why it will hang - the FTP server cannot contact the client machine. An advanced router can intercept the outgoing PORT command and change the IP and PORT to a NAT address, and then it will work. Most modern routers have this feature, so passive mode is not required. With passive FTP, the client issues a PASV command, to which the FTP server responds with a IP and PORT to be used for the data channel (that the server listens on) - and the client initiates the connection, thereby avoiding the NAT problems. To see if your router is doing PORT command intercepting, turn on the debug option with your FTP program - you will see the outgoing IP is a local one... (the router then MUST change the packet, and remember the original IP/PORT to map to later). Note, some FTP programs detect that passive mode should be used, and will do so regardless of the program setting (IE is this way I believe).

Since NAT is enabled, there is no need to block any incoming ports - there can be no inbound connections to any machine on the local network, since they do not have routable IP addresses. Think about it... even with only one real IP address, there can still be several machines on the local network listening on port 80 (HTTP) for a connection... when an inbound connection attempt packet arrives at the router for the real IP address and port, how would the router know which local machine to send it to??? This is how NAT works! As stated above with the FTP, the router can do some remembering of IP/PORTS when it knows to (intercepted the FTP PORT command). Also, our router also allows for a DMZ machine, for which all inbound connection attempts on any port (ports can be filtered if you desire) are routed to. This is how you can set up a local FTP or Web server behind the router.

Now back to the problem and the possible solution...

I can only think of two possible causes.

The first is that it has something to do with the MTU. Since the local machines are connected to the ethernet network, they have an MTU of 1500. The problem is that the PPPoE connection to the Internet requires an MTU of 1492. Normally the router should fragment the packets... but maybe this is not occuring in this case. This is why the downloads work okay, but the uploads fail... The 1500 byte packets are not being fragmented, and so they are discarded by the PPPoE connection. This is also why files < 1k will upload properly. I've seen other messages concerning this problem on the Apple message boards for OSX. Maybe OSX is setting the DNF bit for all packets? I will see if I can set up a sniffer on the network to determine this...

The only other possibility I can think of is that the underlying Internet services library is using a format for the outbound PASV and PORT commands that is confusing the router. I do not know much about Mac programming, so I do not know if there is a standard library for FTP services, or if each application implements their own version (which would make this being the cause highly unlikely).

Maybe we can figure this out. As I said, at least one other person on the Apple OSX message board has reported a this problem, so I'm pretty sure the problem is real... just rare.

Robert

[brodie90]

my 192...ip is dynamic, my true ip is static, so everytime my 192... ip changes i have to change forwarding to my192... Ip, a pain in the ass. is there a way to ftp direct to a machine on a LAN without router forwarding?

[REngels]

Your message isn't really related to my problem, but here's your answer.

There is usually a few solutions:

1) Just give yourself a static IP address in the 192 range. The other machines (and DHCP servers) will detect that it is being used, and will not use it, yet the NAT and other services should still work as expected. This works well for server machines (inluding print servers) that never move. Then you can always setup IP forwarding to that machine.

2) If your DHCP server supports DHCP Client ID, then you're set. Assign yourself a Client ID, and set up the DHCP server to always set that particular client to a certain IP address. If your machine is a laptop that moves between networks, then this is usually the only viable solution, if your want to support a web/ftp server on your machine. This works especially well if each network you attach to supports DHCP Client ID.

Robert

[REngels]

No problem, thanks for the input, but you really do not want to change the MTU of the Ethernet interface - that is a hack. The TCP/IP should find the MSS it can use to talk to the destintion machine, and then limit its TCP segents to that size. The MTU for standard Ethernet interfaces should always be 1500.

Now, the PPPoE interface is another story. There the MTU should be 1492.

When using a DSL router (or any other WAN router for that matter), the sending machine needs to detect the MSS for TCP transmissions. It will always send over its Ethernet interface with an MTU of 1500, but it will not necessarily send a packet of length 1500.

See MTU Size - Big Problem for additional information.

Robert