vagrant setup not work as expected...

[mcarbonneaux]

i've configured as https://github.com/github/glb-director/blob/master/docs/setup/example-setup-vagrant.md.

i've shutdowned director2 that use xdp and use dpdk.

when i use curl from user machine... i go in timeout...

when i use curl from router machine they work but... the ip source that are seen front proxy are 192.168.50.1, in place of 192.168.50.2 why ?

and when try from user machine they use the same ip (192.168.50.1) as ip source in place of 192.168.40.x source ip, and cannot reply to user machine beceause that machine not have the 192.168.50 network...

what are missing ? why glb-director not correctly encapsulate the source ip ?

this ip are not configured any where (no default route any where! but exist on 192.168.50 network because of virtual network as host ip interface)....

vagrant@proxy1:~$ tshark -ni any port not 22 and not arp and port not 547 and not stp
Capturing on 'any'
    1 0.000000000 192.168.50.1 ? 10.10.10.10  TCP 116 37924 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1513029862 TSecr=0 WS=64
    2 0.000000000 192.168.50.1 ? 10.10.10.10  TCP 76 [TCP Out-Of-Order] 37924 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1513029862 TSecr=0 WS=64
    3 0.000051246  10.10.10.10 ? 192.168.50.1 TCP 76 80 ? 37924 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=1575348520 TSecr=1513029862 WS=64
    4 0.000255767 192.168.50.1 ? 10.10.10.10  TCP 108 37924 ? 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1513029862 TSecr=1575348520
    5 0.000255767 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Dup ACK 4#1] 37924 ? 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1513029862 TSecr=1575348520
    6 0.000373384 192.168.50.1 ? 10.10.10.10  HTTP 183 GET / HTTP/1.1
    7 0.000373384 192.168.50.1 ? 10.10.10.10  TCP 143 [TCP Retransmission] 37924 ? 80 [PSH, ACK] Seq=1 Ack=1 Win=64256 Len=75 TSval=1513029862 TSecr=1575348520
    8 0.000391393  10.10.10.10 ? 192.168.50.1 TCP 68 80 ? 37924 [ACK] Seq=1 Ack=76 Win=65088 Len=0 TSval=1575348520 TSecr=1513029862
    9 0.000574828  10.10.10.10 ? 192.168.50.1 HTTP 336 HTTP/1.1 200 OK  (text/html)
   10 0.000756007 192.168.50.1 ? 10.10.10.10  TCP 108 37924 ? 80 [ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1513029863 TSecr=1575348521
   11 0.000756007 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Dup ACK 10#1] 37924 ? 80 [ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1513029863 TSecr=1575348521
   12 0.001064636 192.168.50.1 ? 10.10.10.10  TCP 108 37924 ? 80 [FIN, ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1513029863 TSecr=1575348521
   13 0.001064636 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Out-Of-Order] 37924 ? 80 [FIN, ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1513029863 TSecr=1575348521
   14 0.001100476  10.10.10.10 ? 192.168.50.1 TCP 68 80 ? 37924 [FIN, ACK] Seq=269 Ack=77 Win=65088 Len=0 TSval=1575348521 TSecr=1513029863
   15 0.001311342 192.168.50.1 ? 10.10.10.10  TCP 108 37924 ? 80 [ACK] Seq=77 Ack=270 Win=64128 Len=0 TSval=1513029863 TSecr=1575348521
   16 0.001311342 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Dup ACK 15#1] 37924 ? 80 [ACK] Seq=77 Ack=270 Win=64128 Len=0 TSval=1513029863 TSecr=1575348521

[mcarbonneaux]

with director2 (xdp) up and director1 (dpdk) down...

from router machine they use the correct ip source (192.168.50.2)... but i see gue encapsulation arriving on router...

root@router:/var/log# tshark -ni ens7  port not 22 and not arp and port not 547 and not stp  and not icmp
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens7'
    1 0.000000000 192.168.50.2 ? 10.10.10.10  TCP 74 38552 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1517516318 TSecr=0 WS=64
    2 0.000301919 192.168.50.7 ? 192.168.50.10 UDP 114 53035 ? 19523 Len=72
    3 0.000333580 192.168.50.7 ? 192.168.50.10 UDP 114 53035 ? 19523 Len=72
    4 0.000513269  10.10.10.10 ? 192.168.50.2 TCP 74 80 ? 38552 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20308140 TSecr=1517516318 WS=64
    5 0.000531631 192.168.50.2 ? 10.10.10.10  TCP 66 38552 ? 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1517516318 TSecr=20308140
    6 0.000579569 192.168.50.2 ? 10.10.10.10  HTTP 141 GET / HTTP/1.1
    7 0.000698798 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64
    8 0.000715795 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64
    9 0.000720665 192.168.50.7 ? 192.168.50.10 UDP 181 53035 ? 19523 Len=139
   10 0.000732276 192.168.50.7 ? 192.168.50.10 UDP 181 53035 ? 19523 Len=139
   11 0.000883367  10.10.10.10 ? 192.168.50.2 TCP 66 80 ? 38552 [ACK] Seq=1 Ack=76 Win=65088 Len=0 TSval=20308140 TSecr=1517516318
   12 0.001098714  10.10.10.10 ? 192.168.50.2 HTTP 334 HTTP/1.1 200 OK  (text/html)
   13 0.001106750 192.168.50.2 ? 10.10.10.10  TCP 66 38552 ? 80 [ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1517516319 TSecr=20308140
   14 0.001234437 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64
   15 0.001257957 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64
   16 0.001370792 192.168.50.2 ? 10.10.10.10  TCP 66 38552 ? 80 [FIN, ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1517516319 TSecr=20308140
   17 0.001512882 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64
   18 0.001532522 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64
   19 0.001649547  10.10.10.10 ? 192.168.50.2 TCP 66 80 ? 38552 [FIN, ACK] Seq=269 Ack=77 Win=65088 Len=0 TSval=20308141 TSecr=1517516319
   20 0.001662357 192.168.50.2 ? 10.10.10.10  TCP 66 38552 ? 80 [ACK] Seq=77 Ack=270 Win=64128 Len=0 TSval=1517516319 TSecr=20308141
   21 0.001777110 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64
   22 0.001797798 192.168.50.7 ? 192.168.50.10 UDP 106 53035 ? 19523 Len=64

but from proxy1 they use 195.168.50.1...

vagrant@proxy1:~$ tshark -ni any  port not 22 and not arp and port not 547 and not stp  and not icmp
Capturing on 'any'
    1 0.000000000 192.168.50.1 ? 10.10.10.10  TCP 116 38530 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1517455449 TSecr=0 WS=64
    2 0.000000000 192.168.50.1 ? 10.10.10.10  TCP 76 [TCP Out-Of-Order] 38530 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1517455449 TSecr=0 WS=64
    3 0.000051547  10.10.10.10 ? 192.168.50.1 TCP 76 80 ? 38530 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20247272 TSecr=1517455449 WS=64
    4 0.000334203 192.168.50.1 ? 10.10.10.10  TCP 108 38530 ? 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1517455450 TSecr=20247272
    5 0.000334203 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Dup ACK 4#1] 38530 ? 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1517455450 TSecr=20247272
    6 0.000460081 192.168.50.1 ? 10.10.10.10  HTTP 183 GET / HTTP/1.1
    7 0.000460081 192.168.50.1 ? 10.10.10.10  TCP 143 [TCP Retransmission] 38530 ? 80 [PSH, ACK] Seq=1 Ack=1 Win=64256 Len=75 TSval=1517455450 TSecr=20247272
    8 0.000477217  10.10.10.10 ? 192.168.50.1 TCP 68 80 ? 38530 [ACK] Seq=1 Ack=76 Win=65088 Len=0 TSval=20247272 TSecr=1517455450
    9 0.000662137  10.10.10.10 ? 192.168.50.1 HTTP 336 HTTP/1.1 200 OK  (text/html)
   10 0.000919116 192.168.50.1 ? 10.10.10.10  TCP 108 38530 ? 80 [ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1517455450 TSecr=20247272
   11 0.000919116 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Dup ACK 10#1] 38530 ? 80 [ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1517455450 TSecr=20247272
   12 0.001168016 192.168.50.1 ? 10.10.10.10  TCP 108 38530 ? 80 [FIN, ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1517455451 TSecr=20247272
   13 0.001168016 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Out-Of-Order] 38530 ? 80 [FIN, ACK] Seq=76 Ack=269 Win=64128 Len=0 TSval=1517455451 TSecr=20247272
   14 0.001200109  10.10.10.10 ? 192.168.50.1 TCP 68 80 ? 38530 [FIN, ACK] Seq=269 Ack=77 Win=65088 Len=0 TSval=20247273 TSecr=1517455451
   15 0.001412720 192.168.50.1 ? 10.10.10.10  TCP 108 38530 ? 80 [ACK] Seq=77 Ack=270 Win=64128 Len=0 TSval=1517455451 TSecr=20247273
   16 0.001412720 192.168.50.1 ? 10.10.10.10  TCP 68 [TCP Dup ACK 15#1] 38530 ? 80 [ACK] Seq=77 Ack=270 Win=64128 Len=0 TSval=1517455451 TSecr=20247273

from users machine they use 192.168.40.1 on router machine in place of the 192.168.40 interface ip...

vagrant@router:~$ tshark -ni ens6  port not 22 and not arp and port not 547 and not stp  and not icmp
Capturing on 'ens6'
    1 0.000000000 192.168.40.1 ? 10.10.10.10  TCP 74 50602 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879245733 TSecr=0 WS=64
    2 1.020433452 192.168.40.1 ? 10.10.10.10  TCP 74 [TCP Retransmission] 50602 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879246753 TSecr=0 WS=64
    3 3.036442907 192.168.40.1 ? 10.10.10.10  TCP 74 [TCP Retransmission] 50602 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879248769 TSecr=0 WS=64
    4 7.164480600 192.168.40.1 ? 10.10.10.10  TCP 74 [TCP Retransmission] 50602 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879252897 TSecr=0 WS=64
    5 8.305017046 192.168.40.2 ? 192.168.40.3 BGP 85 KEEPALIVE Message
    6 8.305036204 192.168.40.3 ? 192.168.40.2 TCP 66 179 ? 36129 [ACK] Seq=1 Ack=20 Win=1018 Len=0 TSval=3732229081 TSecr=1705863005

but on the proxy1 machine they use 192.168.50.1...

vagrant@proxy1:~$ tshark -ni any  port not 22 and not arp and port not 547 and not stp  and not icmp
Capturing on 'any'
    1 0.000000000  10.10.10.10 ? 192.168.50.1 TCP 76 80 ? 50596 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20503811 TSecr=2879092176 WS=64
    2 10.062101543 192.168.50.1 ? 10.10.10.10  TCP 116 50598 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879141860 TSecr=0 WS=64
    3 10.062101543 192.168.50.1 ? 10.10.10.10  TCP 76 [TCP Out-Of-Order] 50598 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879141860 TSecr=0 WS=64
    4 10.062229733  10.10.10.10 ? 192.168.50.1 TCP 76 80 ? 50598 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20513873 TSecr=2879141860 WS=64
    5 11.072004083  10.10.10.10 ? 192.168.50.1 TCP 76 [TCP Retransmission] 80 ? 50598 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20514883 TSecr=2879141860 WS=64
    6 11.083299095 192.168.50.1 ? 10.10.10.10  TCP 116 [TCP Retransmission] 50598 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879142882 TSecr=0 WS=64
    7 11.083299095 192.168.50.1 ? 10.10.10.10  TCP 76 [TCP Retransmission] 50598 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879142882 TSecr=0 WS=64
    8 11.083329749  10.10.10.10 ? 192.168.50.1 TCP 76 [TCP Retransmission] 80 ? 50598 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20514894 TSecr=2879141860 WS=64
    9 13.088012404  10.10.10.10 ? 192.168.50.1 TCP 76 [TCP Retransmission] 80 ? 50598 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20516899 TSecr=2879141860 WS=64
   10 13.099311633 192.168.50.1 ? 10.10.10.10  TCP 116 [TCP Retransmission] 50598 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879144898 TSecr=0 WS=64
   11 13.099311633 192.168.50.1 ? 10.10.10.10  TCP 76 [TCP Retransmission] 50598 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2879144898 TSecr=0 WS=64
   12 13.099348309  10.10.10.10 ? 192.168.50.1 TCP 76 [TCP Retransmission] 80 ? 50598 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=20516910 TSecr=2879141860 WS=64

[mcarbonneaux]

i've found... is because libvirt create there vm with default gateway on the ssh part of vagrant, and all network are nat network with dhcp... the first ip of each network can route traffic...
with my pull request her #105 they resolve the problem.