I installed 10.4.3 a couple days after it was released, and only this morning have I been unable to ssh from one computer to another. Yesterday, I was able to ssh from system to system, and even used sftp and scp to copy files. By the way, I truly do mean that. I used sftp to move a song file from one computer to another, logged out, put the mac to sleep. I woke up this morning, and decided to move another file - no go.
This seems to only occur when directly connected network nodes are involved, for example, I cannot ssh from 172.16.1.1 to 172.16.1.2. If, however, my router at 192.168.2.1 is set up to port forward to 172.16.1.2, I can ssh from 172.16.1.1 to 192.168.2.1 successfully, thus bypassing the error.
What's odd, to me at least, is that in the broken scenario the ssh client is not sending a "Client: Key Exchange Init" immediately after the client and server protocols are established between 172.16.1.1 to 172.16.1.2.
Here's a crude example, from the point of view of the server, of it working...
172.16.1.1 -> 172.16.1.2 SYN
172.16.1.2 -> 172.16.1.1 SYN ACK
172.16.1.1 -> 172.16.1.2 ACK
172.16.1.2 -> 172.16.1.1 Server Protocol: SSH-1.99-OpenSSH_3.8.1p1
172.16.1.1 -> 172.16.1.2 Client Protocol: SSH-2.0-OpenSSH_3.8.1p1
172.16.1.2 -> 172.16.1.1 ACK
172.16.1.1 -> 172.16.1.2 Client: Key Exchange Init
172.16.1.2 -> 172.16.1.1 Server: Key Exchange Init
172.16.1.1 -> 172.16.1.2 Client: Diffie-Hellman GEX Request
172.16.1.2 -> 172.16.1.1 Server: Diffie-Hellman Key Exchange Reply
172.16.1.1 -> 172.16.1.2 Client: Diffie-Hellman GEX Init
172.16.1.2 -> 172.16.1.1 Server: Diffie-Hellman GEX Reply
...and so on...
And from there everything negotiates out and I'm connected.
Meanwhile, here's a broken connection
172.16.1.3 -> 172.16.1.2 SYN
172.16.1.2 -> 172.16.1.3 SYN ACK
172.16.1.3 -> 172.16.1.2 ACK
172.16.1.2 -> 172.16.1.3 Server Protocol: SSH-1.99-OpenSSH_3.8.1p1
172.16.1.3 -> 172.16.1.2 Client Protocol: SSH-2.0-OpenSSH_3.8.1p1
172.16.1.2 -> 172.16.1.3 ACK
172.16.1.2 -> 172.16.1.3 Server: Key Exchange Init
172.16.1.3 -> 172.16.1.2 ACK
... two minutes pass ...
172.16.1.2 -> 172.16.1.3 FIN ACK
172.16.1.3 -> 172.16.1.2 ACK
172.16.1.3 -> 172.16.1.2 Client: Key Exchange Init
172.16.1.2 -> 172.16.1.3 RST
172.16.1.3 -> 172.16.1.2 Client: Diffie-Hellman GEX Request
172.16.1.2 -> 172.16.1.3 RST
That's just a filtered view of the ssh problem... there's more to the problem.
host_1:~ user$ ssh -vvv 172.16.1.2
OpenSSH_3.8.1p1, OpenSSL 0.9.7i 14 Oct 2005
debug1: Reading configuration data /Users/user/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 172.16.1.2 [172.16.1.2] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /Users/user/.ssh/rsa1 type 0
debug1: identity file /Users/user/.ssh/rsa2 type 1
debug3: Not a RSA1 key file /Users/user/.ssh/dsa2.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /Users/user/.ssh/dsa2 type 2
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.8.1p1
debug1: match: OpenSSH_3.8.1p1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_3.8.1p1
debug3: Trying to reverse map address 172.16.1.2.
debug1: An invalid name was supplied
Cannot determine realm for numeric host address
debug1: An invalid name was supplied
A parameter was malformed
Validation error
debug1: An invalid name was supplied
Cannot determine realm for numeric host address
debug1: An invalid name was supplied
A parameter was malformed
Validation error
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Write failed: Broken pipe
host_1:~ user$
During the time that it's "trying to reverse map address", the system has sent out a DNS query regarding PTR record 172.16.1.2, one every 5 seconds, to the DNS servers configured on my system. No replies are ever received.
When I perform either "nslookup -type=PTR 172.16.1.2", I can observe the same results. Queries are sent, but no answers are received.
However, when I use "dig -t PTR 172.16.1.2", I, unsurprised, see this:
; <<>> DiG 9.2.2 <<>> -t PTR 172.16.1.2
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21026
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;172.16.1.2. IN PTR
;; Query time: 15 msec
;; SERVER: 172.16.73.10#53(172.16.73.10)
;; WHEN: Sat Jan 7 20:38:06 2006
;; MSG SIZE rcvd: 30
So there's no answer. Fine, but that's my diagnostic tool doing it, and apparently no longer how OS X resolves, or attempts to resolve, private IP space.
My question is this: What changed overnight to cause this behavior?
This seems to only occur when directly connected network nodes are involved, for example, I cannot ssh from 172.16.1.1 to 172.16.1.2. If, however, my router at 192.168.2.1 is set up to port forward to 172.16.1.2, I can ssh from 172.16.1.1 to 192.168.2.1 successfully, thus bypassing the error.
What's odd, to me at least, is that in the broken scenario the ssh client is not sending a "Client: Key Exchange Init" immediately after the client and server protocols are established between 172.16.1.1 to 172.16.1.2.
Here's a crude example, from the point of view of the server, of it working...
172.16.1.1 -> 172.16.1.2 SYN
172.16.1.2 -> 172.16.1.1 SYN ACK
172.16.1.1 -> 172.16.1.2 ACK
172.16.1.2 -> 172.16.1.1 Server Protocol: SSH-1.99-OpenSSH_3.8.1p1
172.16.1.1 -> 172.16.1.2 Client Protocol: SSH-2.0-OpenSSH_3.8.1p1
172.16.1.2 -> 172.16.1.1 ACK
172.16.1.1 -> 172.16.1.2 Client: Key Exchange Init
172.16.1.2 -> 172.16.1.1 Server: Key Exchange Init
172.16.1.1 -> 172.16.1.2 Client: Diffie-Hellman GEX Request
172.16.1.2 -> 172.16.1.1 Server: Diffie-Hellman Key Exchange Reply
172.16.1.1 -> 172.16.1.2 Client: Diffie-Hellman GEX Init
172.16.1.2 -> 172.16.1.1 Server: Diffie-Hellman GEX Reply
...and so on...
And from there everything negotiates out and I'm connected.
Meanwhile, here's a broken connection
172.16.1.3 -> 172.16.1.2 SYN
172.16.1.2 -> 172.16.1.3 SYN ACK
172.16.1.3 -> 172.16.1.2 ACK
172.16.1.2 -> 172.16.1.3 Server Protocol: SSH-1.99-OpenSSH_3.8.1p1
172.16.1.3 -> 172.16.1.2 Client Protocol: SSH-2.0-OpenSSH_3.8.1p1
172.16.1.2 -> 172.16.1.3 ACK
172.16.1.2 -> 172.16.1.3 Server: Key Exchange Init
172.16.1.3 -> 172.16.1.2 ACK
... two minutes pass ...
172.16.1.2 -> 172.16.1.3 FIN ACK
172.16.1.3 -> 172.16.1.2 ACK
172.16.1.3 -> 172.16.1.2 Client: Key Exchange Init
172.16.1.2 -> 172.16.1.3 RST
172.16.1.3 -> 172.16.1.2 Client: Diffie-Hellman GEX Request
172.16.1.2 -> 172.16.1.3 RST
That's just a filtered view of the ssh problem... there's more to the problem.
host_1:~ user$ ssh -vvv 172.16.1.2
OpenSSH_3.8.1p1, OpenSSL 0.9.7i 14 Oct 2005
debug1: Reading configuration data /Users/user/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 172.16.1.2 [172.16.1.2] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /Users/user/.ssh/rsa1 type 0
debug1: identity file /Users/user/.ssh/rsa2 type 1
debug3: Not a RSA1 key file /Users/user/.ssh/dsa2.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /Users/user/.ssh/dsa2 type 2
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.8.1p1
debug1: match: OpenSSH_3.8.1p1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_3.8.1p1
debug3: Trying to reverse map address 172.16.1.2.
debug1: An invalid name was supplied
Cannot determine realm for numeric host address
debug1: An invalid name was supplied
A parameter was malformed
Validation error
debug1: An invalid name was supplied
Cannot determine realm for numeric host address
debug1: An invalid name was supplied
A parameter was malformed
Validation error
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Write failed: Broken pipe
host_1:~ user$
During the time that it's "trying to reverse map address", the system has sent out a DNS query regarding PTR record 172.16.1.2, one every 5 seconds, to the DNS servers configured on my system. No replies are ever received.
When I perform either "nslookup -type=PTR 172.16.1.2", I can observe the same results. Queries are sent, but no answers are received.
However, when I use "dig -t PTR 172.16.1.2", I, unsurprised, see this:
; <<>> DiG 9.2.2 <<>> -t PTR 172.16.1.2
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21026
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;172.16.1.2. IN PTR
;; Query time: 15 msec
;; SERVER: 172.16.73.10#53(172.16.73.10)
;; WHEN: Sat Jan 7 20:38:06 2006
;; MSG SIZE rcvd: 30
So there's no answer. Fine, but that's my diagnostic tool doing it, and apparently no longer how OS X resolves, or attempts to resolve, private IP space.
My question is this: What changed overnight to cause this behavior?