FTP Upload - Big Problem!

R

REngels

Registered
Ok. I've been very impressed with the quality of the responses to other people's questions, so here goes...

I have a small network, running behind a DSL router. The network consists of some Windows 2000 PCs, and OS9.1 Mac and some OSX Macs.

The router is performing NAT, so there is only 1 real IP address. Because of this, the computers behind the router obviously need to use passive FTP.

All of the machines work fine except for the OSX machines! The FTP works fine for downloads but hangs on an upload! Usually, the resulting file is of 0 bytes. The hanging symptoms are similar to if I had passive FTP turned off.

I have tried several FTP clients on the OSX machine: Transmit, Fetch, and the command line FTP client that comes with OSX. All exhibit the same behavior.

<B>Here's the rub... if I run a OS9.X FTP program under the Mac Classic mode, it works fine... (as long as I use passive FTP)</B> For example, Fetch 3.0.5 works fine.

A few last notes... this problem only appears when accessing FTP sites beyong the router (i.e on the Internet). If I access an FTP server on the local network (one of the Windows PCs) from the OSX machine, everything works fine. Also, I've tried accessing a variety of extenal FTP servers (Unix, Windows) from the OSX machines, and they yield the same upload problem.

Under OSX, I've tried setting the passive mode in both the client application and in the network preferences. Nothing seems to help.

Also, I am running 10.1. but I think the same problem was there under 10.0.x. My OSX machines are pretty generic... Powerbook G3, and Titanium G4.

I am really at my wits end, and any help would be greatly appreciated.

Thanks,
Robert Engels
 
I don't have any idea of what I am talking about so if you feel like trying it cool, if not I understand.

By educated guess is that the port for ftp is not set up to allow passive connections, I would suggest downloading brickhouse from version trackers and fooling around with it to try to setup ftp port to work with passive, the setup tool is very easy to use, but also getting intothe guts is pretty easy as well, I hope this helps, if I am full of crap please someone tell me so I can learn thanks!
 
Soapvox, thanks for the input, but I'm pretty sure that's not it.

The FTP Server (located on the Internet) does support passive connections, since that is how the other non-OSX machines connect.

Also, I've used 'ipfw' to make sure there were no ports beng blocked. I have no other firewall software loaded.

Robert
 
Hi REngles,

I think your right, this passive ftp upload problem has been around pre 10.1. I remember running into this some time back and quickly placed blamed on a third party app for this error. And in some respects it turns out to be somewhat true.

I'm able to reproduce this passive ftp upload problem with Transmit and using ftp from within Terminal. Using tcpflow, I was able to find where and possibly why this upload error occurs. The freeze is caused by a server error response to the TYPE command that's initiated after the user issues the PUT command to start a file transfer to the server. It seems uploading in ftp passive mode will only work if the file TYPE is set to binary image. Set it to any thing else and the server will throw a 5xx type error and pause (freeze occurs on the client side). In an active ftp session, in which both tcp ports 21 (commands) and 20 (data) are used, the default file type for data transfer is acsii. And this works as expected. In a passive ftp session the file transfer type for Mac OS X should be binary image (contrary to RFC959). This switching from acsii to binary is where things go doo doo for some ftp clients. In my case Transmit and Terminal ftp if not properly set.

I'll try to explain how this affected some of your findings mentioned in your first post.

-"The router is performing NAT, so there is only 1 real IP address. Because of this, the computers behind the router obviously need to use passive FTP"

This is not necessarily true. NAT'ing will not hamper the proper flow of an active ftp session initiated by a client behind the router. It will hamper the flow If say you had an ftp server setup behind this router on your LAN using NAT. Then the hiding of the servers private address would be an issue if your router did not perform some sort of reverse proxy for your LAN ftp server. If your ingress ports 20 and 21 are open, active ftp session should be no problem.

-"The FTP works fine for downloads but hangs on an upload! Usually, the resulting file is of 0 bytes"

The resulting file can sometimes be seen because the LIST command is given from the client to the server before the TYPE command which freezes the session. The server will retain the file name of the yet to happen transfer regardless if it has failed or succeeded. The remnants of this is what you see on the server's target directory. Very similar to what happens with tftp and the touch file, except without the errors.

-"I have tried several FTP clients on the OSX machine: Transmit, Fetch, and the command line FTP client that comes with OSX. All exhibit the same behavior."

I've noticed this with Transmit. I can say that Transmit doesn't seem to give this file TYPE binary command when set for a passive session.
The command line ftp program you run in terminal defaults to file TYPE ascii and session mode active. You have to physically set these parameters to passive and file TYPE binary to have successful uploads in passive mode.

-"this problem only appears when accessing FTP sites beyond the router (i.e on the Internet). If I access an FTP server on the local network (one of the Windows PCs) from the OSX machine, everything works fine. Also, I've tried accessing a variety of external FTP servers (Unix, Windows) from the OSX machines, and they yield the same upload problem"

I can see your situation being this way if your router is set to block ingress port 20 (not egress 20. Your egress ports should be somewhere in the 4k's). Perhaps the ftp session to your winbox is not passive after all but active, or the Win box doesn't mind the TYPE I command and accepts the acsii transfer without error. You would need to sniff around ports 20, 21 for the answer if all of the above is not doing it for you.


Hope this helps. And please let us know of your outcomes.
 
Klink,

Thanks for the suggestions but I think you might be mistaken on a few issues...

In general, I do not believe it has anything to do with binary versus ascii. I have manually set the mode to binary in a variety of clients with no difference in the result. See below for what I think it might be...

As for the other issues...

Machines using NAT must use passive FTP, unless the Router does some advanced FTP packet sniffing. Without passive FTP, the client machine sends the FTP the IP address and PORT the server should contact the client on (using the PORT command) for the data channel. Obviously, the client machine has a local non-routable address, so the information contained in the PORT command is invalid - that is why it will hang - the FTP server cannot contact the client machine. An advanced router can intercept the outgoing PORT command and change the IP and PORT to a NAT address, and then it will work. Most modern routers have this feature, so passive mode is not required. With passive FTP, the client issues a PASV command, to which the FTP server responds with a IP and PORT to be used for the data channel (that the server listens on) - and the client initiates the connection, thereby avoiding the NAT problems. To see if your router is doing PORT command intercepting, turn on the debug option with your FTP program - you will see the outgoing IP is a local one... (the router then MUST change the packet, and remember the original IP/PORT to map to later). Note, some FTP programs detect that passive mode should be used, and will do so regardless of the program setting (IE is this way I believe).

Since NAT is enabled, there is no need to block any incoming ports - there can be no inbound connections to any machine on the local network, since they do not have routable IP addresses. Think about it... even with only one real IP address, there can still be several machines on the local network listening on port 80 (HTTP) for a connection... when an inbound connection attempt packet arrives at the router for the real IP address and port, how would the router know which local machine to send it to??? This is how NAT works! As stated above with the FTP, the router can do some remembering of IP/PORTS when it knows to (intercepted the FTP PORT command). Also, our router also allows for a DMZ machine, for which all inbound connection attempts on any port (ports can be filtered if you desire) are routed to. This is how you can set up a local FTP or Web server behind the router.

Now back to the problem and the possible solution...

I can only think of two possible causes.

The first is that it has something to do with the MTU. Since the local machines are connected to the ethernet network, they have an MTU of 1500. The problem is that the PPPoE connection to the Internet requires an MTU of 1492. Normally the router should fragment the packets... but maybe this is not occuring in this case. This is why the downloads work okay, but the uploads fail... The 1500 byte packets are not being fragmented, and so they are discarded by the PPPoE connection. This is also why files < 1k will upload properly. I've seen other messages concerning this problem on the Apple message boards for OSX. Maybe OSX is setting the DNF bit for all packets? I will see if I can set up a sniffer on the network to determine this...

The only other possibility I can think of is that the underlying Internet services library is using a format for the outbound PASV and PORT commands that is confusing the router. I do not know much about Mac programming, so I do not know if there is a standard library for FTP services, or if each application implements their own version (which would make this being the cause highly unlikely).

Maybe we can figure this out. As I said, at least one other person on the Apple OSX message board has reported a this problem, so I'm pretty sure the problem is real... just rare.

Robert
 
my 192...ip is dynamic, my true ip is static, so everytime my 192... ip changes i have to change forwarding to my192... Ip, a pain in the ass. is there a way to ftp direct to a machine on a LAN without router forwarding?
 
Your message isn't really related to my problem, but here's your answer.

There is usually a few solutions:

1) Just give yourself a static IP address in the 192 range. The other machines (and DHCP servers) will detect that it is being used, and will not use it, yet the NAT and other services should still work as expected. This works well for server machines (inluding print servers) that never move. Then you can always setup IP forwarding to that machine.

2) If your DHCP server supports DHCP Client ID, then you're set. Assign yourself a Client ID, and set up the DHCP server to always set that particular client to a certain IP address. If your machine is a laptop that moves between networks, then this is usually the only viable solution, if your want to support a web/ftp server on your machine. This works especially well if each network you attach to supports DHCP Client ID.

Robert
 
No problem, thanks for the input, but you really do not want to change the MTU of the Ethernet interface - that is a hack. The TCP/IP should find the MSS it can use to talk to the destintion machine, and then limit its TCP segents to that size. The MTU for standard Ethernet interfaces should always be 1500.

Now, the PPPoE interface is another story. There the MTU should be 1492.

When using a DSL router (or any other WAN router for that matter), the sending machine needs to detect the MSS for TCP transmissions. It will always send over its Ethernet interface with an MTU of 1500, but it will not necessarily send a packet of length 1500.

See MTU Size - Big Problem for additional information.

Robert
 
REngles,

I'm titillated to hear your continued interest in this issue, and the few hours I spent researching and conducting my previous tests have not gone in vein.

I'm a big believer in simplifying a test scenario to troubleshoot a problem. This ftp issue gets a bit complicated when the introduction of router filters, and proxies are introduced into the equation (let alone changing the network layer as in PPPoE). I propose we look at this with a bit more focus on the one issue, the ftp protocol and how Mac OS X does or doesn't work with this in a controlled test bed.

I have at my disposal an ftp server in which I have full control (Using AppleShareIP 6.x. Don't laugh please. Ostensibly seen as a unix ftp server from the outside.) I also have the ability to run my OSX 10.1 machine on the servers local net (on the same switch) with the only addition of my ISP supplying routing for this net. If I conduct another test, using only Terminal and it's ftp application as the client (only because I have utmost control of the protocol settings and there validity), use tcpflow as my monitoring source and capture the results of both active and passive ftp sessions, perhaps we can look at the results and come to an answer that pleases us both.

I'll await your response before starting if you have other parameters you would like added to the test scenario I may have overlooked.

testuser you're being too modest. You have an excellent brain for this. It would only take little effort to read through RFC959 to get up to speed. Your input is always welcome.
 
Klink,

I think that is a fine way to go, but the only FTP problems I have is when I go out to the Internet. I don't have a firewall per se, but a DSL/Ethernet/Wireless Bridge/ Router, that does NAT. It is the NetGear MR314 for reference.

I have done extensive testing using the LAN Sniffer here. As previously noted, there are no problems when using FTP on the local LAN - since all machines can accept the 1514 byte Ethernet Frames.

The problem only occurs when going out to the Internet. Simply, the other Windows and OS9.x machines use an Ethernet packet size of 1454. For some reason the OSX machine still sends Ethernet packets of length 1514 - too big to be sent on the PPPoE WAN connection.

Robert
 
Rob at this point you seem to be very close to a solution or at the least on the right track. It's clear to me any results of my proposed test wouldn't shed any light on your situation.
Your thread "MTU size - Big problem" raises further interesting needed research on my part. Unfortunately I lack a proper analyzer for Ethernet framing or a PPPoE connection which makes things a tad more difficult. I'll try reading up on TCP/IP transmissions and see if something comes to light.
 
You must log in or register to reply here.
Back
Top