Win 7 client waits a long time at "initializing remote connection" when connecting to RDS Collection through Connection Broker

I have fully patched windows 7 clients with the RDP 8.0 client (Shell Version 6.2.9200) that take over a minute to connect to an RDS session host through a connection broker.

All servers are 2012 R2.

On the RDS Session Host it eventually gets connected to I get this warning

ID 20499
Remote Desktop Services has taken too long to load the user configuration from server \\myserver for user myuser

edit:

My windows 8.1 machine running RDP 8.1 in the same environment connects with no delay. That session connection also gets the warning ID 20499 so that seems to be another issue