Problem with UDP Port forwarding under Vista

This problem is related to openswan, VMware and Vista. I asked the question
in the mailing lists of Openswan and VMware but did not receive any response.
In short, UDP port forwarding for ports 500 and 4500 worked under XP but not
under Vista. Under Vista, the firewall and the IKEEXT services were disabled
for testing, and the problem persisted. I was wondering if there is any
"secret" registry key I need to use :-)

The remote openswan server is a Linux box with a public IP
(abc.def.45.22) and ports 500 and 4500 open. It is NOT behind a NAT
router.

The client is a guest Linux virtual machine (192.168.117.128) running
in NAT mode inside VMware under Windows.

Incoming UDP port forwarding was enabled for ports 500 and 4500 for
NAT in VMware. Under Windows, IPSec/IKEEXT services were disabled to
release UDP ports 500 and 4500, Otherwise the VMware NAT service would
not start after port forwarding was set up. I tried both VMware Server
2.0 Beta and VMware Workstation 6.0.2 on Windows XP and Vista.

The host machine was connected to a broadband router or to a cable
modem directly. In both cases, if the host OS is XP, the VPN worked on
the VM.

However, the VPN did not work if the host OS was Vista, no matter the
machine was directly connected to the cable modem or not. Using
Wireshark, I could see the responses from the server were received by
the host machine, The VM even received a few packets from the server
through ports 500 and 4500 at the beginning. But then the packets were
not forwarded to the VM any more.

The capture below shows the initialization process of VPN. Frames
10-12 were three pings from the client, the responses were received by
the Vista host but not forwarded to the VM. When the host was XP,
there was no problem.

No. Time Source Destination Src
Port Dest Port Protocol Info
1 0.000000 192.168.117.128 abc.def.45.22 500
500 ISAKMP Identity Protection (Main Mode)
2 0.032642 abc.def.45.22 192.168.117.128 500
500 ISAKMP Identity Protection (Main Mode)
3 0.040566 192.168.117.128 abc.def.45.22 500
500 ISAKMP Identity Protection (Main Mode)
4 0.086090 abc.def.45.22 192.168.117.128 500
500 ISAKMP Identity Protection (Main Mode)
5 0.127976 192.168.117.128 abc.def.45.22 4500
4500 ISAKMP Identity Protection (Main Mode)
6 0.187260 abc.def.45.22 192.168.117.128 4500
4500 ISAKMP Identity Protection (Main Mode)
7 0.195701 192.168.117.128 abc.def.45.22 4500
4500 ISAKMP Quick Mode
8 0.275112 abc.def.45.22 192.168.117.128 4500
4500 ISAKMP Quick Mode
9 0.312014 192.168.117.128 abc.def.45.22 4500
4500 ISAKMP Quick Mode
10 4.460707 192.168.117.128 abc.def.45.22 4500
4500 ESP ESP (SPI=0x494bd498)
11 5.402257 192.168.117.128 abc.def.45.22 4500
4500 ESP ESP (SPI=0x494bd498)
12 6.414442 192.168.117.128 abc.def.45.22 4500
4500 ESP ESP (SPI=0x494bd498)
13 19.289616 192.168.117.128 abc.def.45.22 4500
4500 UDPENCAP
14 19.289959 192.168.117.128 abc.def.45.22 4500
4500 UDPENCAP


The VPN worked when the VM was in the bridged mode. But I need to make
it work with NAT. The confusing part was that there was no problem
under Windows XP. It seemed Vista blocked the port forwarding.

In summary, under Windows XP, IPSec services were disabled to release ports
500 and 4500. The Windows firewall was not disabled and ports 500 and 4500
were not open as exception. And the VPN was initiated from inside and it
worked.

Under Windows Vista, IKEEXT services were disabled to release ports 500 and
4500. The Windows firewall was disabled. The VPN was initiated from the
client and it did not work. It seemed the Vista host stopped forwarding the
UDP packets to the virtual machine.

I did not have any third-party proxy/firewall programs on Vista. I disabled
the Windows Firewall (It might not be necessary to disable
a stateful firewall but I just wanted to try) and UAC, and the
problem persisted.

Here is ipsec.conf. Any suggestions? Thanks. Philip.

version 2.0

config setup
# NAT-TRAVERSAL support, see README.NAT-Traversal
nat_traversal=yes
conn testuser
type=tunnel
left=%defaultroute
leftid=@testuser
leftsubnet=10.0.0.12/32
leftrsasigkey= (deleted)
right=abc.def.45.22
rightid=@vpnserver
rightsubnet=10.28.0.254/24
rightrsasigkey= (deleted)
authby=rsasig
auto=start

#Disable Opportunistic
include /etc/ipsec.d/examples/no_oe.conf

BTW. I also disabled UAC under Vista. The process which handles port
forwarding was vmnet.exe. I tested TCP port forwarding and it worked. Since
UDP ports 500 and 4500 are used by ipsec, I wonder if there is any hidden
rule/policy which does not allow these ports to be forwarded under Vista.

"Philip888" wrote:

> This problem is related to openswan, VMware and Vista. I asked the question
> in the mailing lists of Openswan and VMware but did not receive any response.
> In short, UDP port forwarding for ports 500 and 4500 worked under XP but not
> under Vista. Under Vista, the firewall and the IKEEXT services were disabled
> for testing, and the problem persisted. I was wondering if there is any
> "secret" registry key I need to use :-)
>
> The remote openswan server is a Linux box with a public IP
> (abc.def.45.22) and ports 500 and 4500 open. It is NOT behind a NAT
> router.
>
> The client is a guest Linux virtual machine (192.168.117.128) running
> in NAT mode inside VMware under Windows.
>
> Incoming UDP port forwarding was enabled for ports 500 and 4500 for
> NAT in VMware. Under Windows, IPSec/IKEEXT services were disabled to
> release UDP ports 500 and 4500, Otherwise the VMware NAT service would
> not start after port forwarding was set up. I tried both VMware Server
> 2.0 Beta and VMware Workstation 6.0.2 on Windows XP and Vista.
>
> The host machine was connected to a broadband router or to a cable
> modem directly. In both cases, if the host OS is XP, the VPN worked on
> the VM.
>
> However, the VPN did not work if the host OS was Vista, no matter the
> machine was directly connected to the cable modem or not. Using
> Wireshark, I could see the responses from the server were received by
> the host machine, The VM even received a few packets from the server
> through ports 500 and 4500 at the beginning. But then the packets were
> not forwarded to the VM any more.
>
> The capture below shows the initialization process of VPN. Frames
> 10-12 were three pings from the client, the responses were received by
> the Vista host but not forwarded to the VM. When the host was XP,
> there was no problem.
>
> No. Time Source Destination Src
> Port Dest Port Protocol Info
> 1 0.000000 192.168.117.128 abc.def.45.22 500
> 500 ISAKMP Identity Protection (Main Mode)
> 2 0.032642 abc.def.45.22 192.168.117.128 500
> 500 ISAKMP Identity Protection (Main Mode)
> 3 0.040566 192.168.117.128 abc.def.45.22 500
> 500 ISAKMP Identity Protection (Main Mode)
> 4 0.086090 abc.def.45.22 192.168.117.128 500
> 500 ISAKMP Identity Protection (Main Mode)
> 5 0.127976 192.168.117.128 abc.def.45.22 4500
> 4500 ISAKMP Identity Protection (Main Mode)
> 6 0.187260 abc.def.45.22 192.168.117.128 4500
> 4500 ISAKMP Identity Protection (Main Mode)
> 7 0.195701 192.168.117.128 abc.def.45.22 4500
> 4500 ISAKMP Quick Mode
> 8 0.275112 abc.def.45.22 192.168.117.128 4500
> 4500 ISAKMP Quick Mode
> 9 0.312014 192.168.117.128 abc.def.45.22 4500
> 4500 ISAKMP Quick Mode
> 10 4.460707 192.168.117.128 abc.def.45.22 4500
> 4500 ESP ESP (SPI=0x494bd498)
> 11 5.402257 192.168.117.128 abc.def.45.22 4500
> 4500 ESP ESP (SPI=0x494bd498)
> 12 6.414442 192.168.117.128 abc.def.45.22 4500
> 4500 ESP ESP (SPI=0x494bd498)
> 13 19.289616 192.168.117.128 abc.def.45.22 4500
> 4500 UDPENCAP
> 14 19.289959 192.168.117.128 abc.def.45.22 4500
> 4500 UDPENCAP
>
>
> The VPN worked when the VM was in the bridged mode. But I need to make
> it work with NAT. The confusing part was that there was no problem
> under Windows XP. It seemed Vista blocked the port forwarding.
>
> In summary, under Windows XP, IPSec services were disabled to release ports
> 500 and 4500. The Windows firewall was not disabled and ports 500 and 4500
> were not open as exception. And the VPN was initiated from inside and it
> worked.
>
> Under Windows Vista, IKEEXT services were disabled to release ports 500 and
> 4500. The Windows firewall was disabled. The VPN was initiated from the
> client and it did not work. It seemed the Vista host stopped forwarding the
> UDP packets to the virtual machine.
>
> I did not have any third-party proxy/firewall programs on Vista. I disabled
> the Windows Firewall (It might not be necessary to disable
> a stateful firewall but I just wanted to try) and UAC, and the
> problem persisted.
>
> Here is ipsec.conf. Any suggestions? Thanks. Philip.
>
> version 2.0
>
> config setup
> # NAT-TRAVERSAL support, see README.NAT-Traversal
> nat_traversal=yes
> conn testuser
> type=tunnel
> left=%defaultroute
> leftid=@testuser
> leftsubnet=10.0.0.12/32
> leftrsasigkey= (deleted)
> right=abc.def.45.22
> rightid=@vpnserver
> rightsubnet=10.28.0.254/24
> rightrsasigkey= (deleted)
> authby=rsasig
> auto=start
>
> #Disable Opportunistic
> include /etc/ipsec.d/examples/no_oe.conf
>
>