机器部署信息
lvs ：
10.0.0.200 vip
10.0.0.19 外网IP ，
172.168.1.19 内网IP
dr rs:
10.0.0.200 vip
10.0.0.18 rip
nat rs:
172.168.1.17 rip
客户端：
10.0.0.14 cip
lvs机器：
ip addr add 10.0.0.200/24 dev ens33:0
IP：
[root@mcw09 ~]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback00:00:00:00:00:00brd00:00:00:00:00:00inet127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:f0:dd:56 brd ff:ff:ff:ff:ff:ff inet 10.0.0.19/24brd10.0.0.255scopeglobal ens33 valid_lft forever preferred_lft forever inet 10.0.0.200/24scopeglobal secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::495b:ff7:d185:f95d/64 scope link valid_lft forever preferred_lft forever inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed valid_lft forever preferred_lft forever3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:f0:dd:60 brd ff:ff:ff:ff:ff:ff inet 172.168.1.19/24brd172.168.1.255scopeglobal ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever[root@mcw09 ~]#
路由没有啥变动，
[root@mcw09 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.2540.0.0.0UG10000ens340.0.0.010.0.0.2540.0.0.0UG10100ens3310.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw09~]#
dr的rs和lvs用同一个网段，nat的转发和lvs可以是两个网段
[root@mcw09 ~]# ipvsadm -LnIP Virtual Server version 1.2.1(size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 10.0.0.200:80rr->172.168.1.17:80Masq100->10.0.0.18:80Route100[root@mcw09~]#
dr机器：
route add -host 10.0.0.200 dev lo
[root@mcw08 ~]# cat /etc/sysctl.confnet.ipv4.conf.all.arp_ignore=1net.ipv4.conf.all.arp_announce=2net.ipv4.conf.lo.arp_ignore=1net.ipv4.conf.lo.arp_announce=2
lo添加了VIP
[root@mcw08 ~]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback00:00:00:00:00:00brd00:00:00:00:00:00inet127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 10.0.0.200/32scopegloballo:0 valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:26:33:3f brd ff:ff:ff:ff:ff:ff inet 10.0.0.18/24brd10.0.0.255scopeglobal ens33 valid_lft forever preferred_lft forever inet6 fe80::f32c:166d:40de:8f2e/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::495b:ff7:d185:f95d/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed valid_lft forever preferred_lft forever3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:26:33:49 brd ff:ff:ff:ff:ff:ff inet 172.168.1.18/24brd172.168.1.255scopeglobal ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever15: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100link/none inet 10.8.0.1/24brd10.8.0.255scopeglobal tun0 valid_lft forever preferred_lft forever inet6 fe80::923d:6caf:c22:c8a5/64 scope link flags 800 valid_lft forever preferred_lft forever[root@mcw08 ~]#
nat机器：
[root@mcw07 ~]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback00:00:00:00:00:00brd00:00:00:00:00:00inet127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:5d:df:62 brd ff:ff:ff:ff:ff:ff3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:5d:df:6c brd ff:ff:ff:ff:ff:ff inet 172.168.1.17/24brd172.168.1.255scopeglobal ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever[root@mcw07 ~]#
只开启了一个内网IP，并且默认网关指向dip。这里的dip和rip是同一个网段的。不在同一个网段的不清楚咋弄
[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG000ens34172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]#
测试：
mcw04上测试，访问lvs上VIP 10.0.0.200,从而访问到两个rs,一个rs是nat，一个是dr
添加其他网络测试
当把nat的另外一个网卡起起来之后
[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG000ens34172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]# ifup ens33Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/45)[root@mcw07~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG000ens340.0.0.010.0.0.2540.0.0.0UG10000ens3310.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]#
nat的这个rs机器就不能访问到了
[root@mcw04 ~]# curl 10.0.0.200:80curl: (7) Failed connect to 10.0.0.200:80; Connection timed out[root@mcw04~]# curl 10.0.0.200:80rs1 mcw08 ^_^10.0.0.18[root@mcw04~]# curl 10.0.0.200:80curl: (7) Failed connect to 10.0.0.200:80; Connection timed out[root@mcw04~]# curl 10.0.0.200:80rs1 mcw08 ^_^10.0.0.18[root@mcw04~]#
删掉一条nat的rs的默认路由之后还是无法访问到nat的
[root@mcw07 ~]# [root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG000ens340.0.0.010.0.0.2540.0.0.0UG10000ens3310.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]# ip route del defaultvia10.0.0.254 dev ens33[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG000ens3410.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]#
[root@mcw04 ~]# curl 10.0.0.200:80rs1 mcw08 ^_^10.0.0.18[root@mcw04~]# curl 10.0.0.200:80curl: (7) Failed connect to 10.0.0.200:80; Connection timed out[root@mcw04~]#
因为内网dip做路由
内网网卡卡做默认路由时，如何去通外网。需要添加外网网段指向外网网卡(lvs nat rs中有多网卡处理案例)
dip和rip是同一个内网，因为将nat模型的lvs的dip作为默认网关后，这样ens33的能通223.5.5.5的网卡，现在不通了
[root@mcw07 ~]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback00:00:00:00:00:00brd00:00:00:00:00:00inet127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:5d:df:62 brd ff:ff:ff:ff:ff:ff inet 10.0.0.17/24brd10.0.0.255scopeglobal ens33 valid_lft forever preferred_lft forever inet6 fe80::f32c:166d:40de:8f2e/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::495b:ff7:d185:f95d/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed valid_lft forever preferred_lft forever3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether00:0c:29:5d:df:6c brd ff:ff:ff:ff:ff:ff inet 172.168.1.17/24brd172.168.1.255scopeglobal ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens3410.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]# [root@mcw07 ~]# [root@mcw07 ~]# [root@mcw07 ~]# ping 223.5.5.5PING223.5.5.5(223.5.5.5)56(84) bytes of data.From 172.168.1.19icmp_seq=1 Redirect Host(New nexthop: 172.168.1.254)From172.168.1.19: icmp_seq=1 Redirect Host(New nexthop: 172.168.1.254)^C---223.5.5.5 ping statistics ---3 packets transmitted, 0 received, +1errors,100% packet loss, time 2003ms[root@mcw07 ~]#
添加走223.5.5.5的，还是用ens33网卡，这个网卡之前默认网关是10.0.0.254.现在直接指定走这个网段的，都指定网关和网卡。这样就能重新通223.5.5.5这个网络了。
ip route add 223.0.0.0/8 via 10.0.0.254 dev ens33
[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens3410.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]# ip route add 223.0.0.0/8via10.0.0.254 dev ens33[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens3410.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34223.0.0.010.0.0.254255.0.0.0UG000ens33[root@mcw07~]# ping 223.5.5.5PING223.5.5.5(223.5.5.5)56(84) bytes of data.64bytesfrom223.5.5.5: icmp_seq=1ttl=128time=9.00ms64bytesfrom223.5.5.5: icmp_seq=2ttl=128time=7.96ms^C---223.5.5.5 ping statistics ---2 packets transmitted, 2received,0% packet loss, time 1001msrtt min/avg/max/mdev = 7.969/8.485/9.002/0.524ms[root@mcw07~]#
将下面跟ens33网卡的路由删除。最后只保留ens34的两条路由。这样情况下，两个网卡内的网段，发现网络都是互通的，也就是路由上没有这个网卡的配置，这个网卡相关的网段好像也是可以直接通的。再添加一个走223.0.0.0网段的路由，走ens33网卡接口和它对应的网关，这样就能通223.5.5.5了。
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens3410.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34[root@mcw07~]# ip route add 223.0.0.0/8via10.0.0.254 dev ens33[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens3410.0.0.00.0.0.0255.255.255.0U10000ens33172.168.1.00.0.0.0255.255.255.0U10000ens34223.0.0.010.0.0.254255.0.0.0UG000ens33[root@mcw07~]# ip route del 10.0.0.0/24 dev ens33[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens34172.168.1.00.0.0.0255.255.255.0U10000ens34223.0.0.010.0.0.254255.0.0.0UG000ens33[root@mcw07~]# ping 223.5.5.5PING223.5.5.5(223.5.5.5)56(84) bytes of data.64bytesfrom223.5.5.5: icmp_seq=1ttl=128time=25.8ms64bytesfrom223.5.5.5: icmp_seq=2ttl=128time=8.13ms^C---223.5.5.5 ping statistics ---2 packets transmitted, 2received,0% packet loss, time 1002msrtt min/avg/max/mdev = 8.133/16.975/25.818/8.843ms[root@mcw07~]# ping 10.0.0.18PING10.0.0.18(10.0.0.18)56(84) bytes of data.64bytesfrom10.0.0.18: icmp_seq=1ttl=64time=6.44ms64bytesfrom10.0.0.18: icmp_seq=2ttl=64time=0.690ms^C---10.0.0.18 ping statistics ---2 packets transmitted, 2received,0% packet loss, time 1001msrtt min/avg/max/mdev = 0.690/3.569/6.449/2.880ms[root@mcw07~]# ping 10.0.0.19PING10.0.0.19(10.0.0.19)56(84) bytes of data.64bytesfrom10.0.0.19: icmp_seq=1ttl=64time=0.919ms^C---10.0.0.19 ping statistics ---1 packets transmitted, 1received,0% packet loss, time 0msrtt min/avg/max/mdev = 0.919/0.919/0.919/0.000ms[root@mcw07~]#
现在下面ens33的跟10.0.0.0/24 10.0.0.254 相关的路由都已经删除，只保留了ens34的172.168.1.0/24 172.168.1.254这个两条路由，然后新增一个走向223.5.5.5这个外网IP时，走ens33网卡接口的路由，指定ens33的网关，这样223.5.5.5在mcw07上由不通变为通。并且不会影响lvs nat 下rs的功能。也就是lvs那里正常访问到mcw07.。如果加上ens33的路由，比如只加上10.0.0.0/24 ，网关是0.0.0.0时就无法让lvs nat正常响应数据了。
[root@mcw07 ~]# [root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens34172.168.1.00.0.0.0255.255.255.0U10000ens34223.0.0.010.0.0.254255.0.0.0UG000ens33[root@mcw07~]#
客户端正常响应数据，返回nat模型的数据，也就是mcw07的数据。不知道是否在某种情况下，可以添加ens33网卡原本有的路由条目，不过即使没加，但是10.0.0.0/24网段的IP也是通的，但是其它网段想通，需要指定走ens33，也就是我们这里这个案例的这种配置，不然可能是走的内网网卡ens34,这个本来就是默认不通外网的。也就是你想要通的网段，如果现在的默认网关ens34不通，但是用ens33通的话，需要添加该网段路由，指向ens33，这样它就知道怎么走，可以通网了，不然就是走的默认的ens34，这个不通的了
[root@mcw04 ~]# curl 10.0.0.200:80rs1 mcw08 ^_^10.0.0.18[root@mcw04~]# curl 10.0.0.200:80rs2 mcw07 ^_^10.0.0.17[root@mcw04~]#
下面是lvs规则
[root@mcw09 ~]# ipvsadm -LnIP Virtual Server version 1.2.1(size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 10.0.0.200:80rr->172.168.1.17:80Masq100->10.0.0.18:80Route100[root@mcw09~]#
给lvs nat 的rs添加arp抑制，不影响nat的正常
[root@mcw07 ~]# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0172.168.1.190.0.0.0UG10000ens34172.168.1.00.0.0.0255.255.255.0U10000ens34223.0.0.010.0.0.254255.0.0.0UG000ens33[root@mcw07~]# vim /etc/sysctl.conf [root@mcw07 ~]# tail -5/etc/sysctl.confnet.ipv4.conf.all.arp_ignore=1net.ipv4.conf.all.arp_announce=2net.ipv4.conf.lo.arp_ignore=1net.ipv4.conf.lo.arp_announce=2[root@mcw07~]# sysctl -p
mcw07这个nat模型的rs,正常被访问到
[root@mcw04 ~]# curl 10.0.0.200:80rs1 mcw08 ^_^10.0.0.18[root@mcw04~]# curl 10.0.0.200:80rs2 mcw07 ^_^10.0.0.17[root@mcw04~]#