RD session SLOW to connect using RD Gateway on Server 2008R2

I have a new W2008 R2 Server running both RD Gateway and RD Host Services on the same box.  When starting a RD client session locally , sans the gateway,  the time to connect is quick as would be expected.  Using the gateway locally, the connection time is a tad slower.   BUT when we connect to the gateway via the internet, the time to negotiate a connection up is painfully slow. 

Any configuration hints that might make the connection time quicker?

Thanks
Brian