Aggregation fails when using modems with IPs in same subnet

[SriramScorp]

When I use 3 USB modems with the same IP (192.168.42.1) and giving DHCP leases in the subnet 192.168.42.0/24, traffic flows only through one of the modems, based on the interface metric. The IP and DHCP range seems hardcoded in the modems and I'm unable to change them. Does aggregation work with this setup as long the interface IPs are unique? Or is it mandatory that each WAN interface should have IPs in unique subnet?

[matttbe]

Hello,

It might depend on the path-manager you use. Also, do you get a different IP each time?

If not, it is possible it gets confused if they all have the same IPs. A workaround is maybe to create a net namespace with veth devices having different IPs, where the traffic would go to one specific USB modem. There are probably other solutions but that's not common to have multiple times the same IP :)

If you have different IPs, it should work I think. It depends on the routing rules you use. http://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting

[SriramScorp]

The IPs are different. And the path-manager is 'fullmesh'. Does 'fullmesh' work or should I be using a different path-manager?

root@raspi-mptcp:~# 
root@raspi-mptcp:~# 
root@raspi-mptcp:~# ip rule list table 10
0:from 192.168.42.180 lookup 10
1:from all fwmark 0x53910 lookup 10
root@raspi-mptcp:~# 
root@raspi-mptcp:~# 
root@raspi-mptcp:~# ip rule list table 11
0:from 192.168.42.33 lookup 11
1:from all fwmark 0x53911 lookup 11
root@raspi-mptcp:~# 
root@raspi-mptcp:~# 
root@raspi-mptcp:~# ip route list table 10
default via 192.168.42.1 dev usb1 
192.168.42.0/24 dev usb1 scope link 
root@raspi-mptcp:~# 
root@raspi-mptcp:~# 
root@raspi-mptcp:~# ip route list table 11
default via 192.168.42.1 dev usb0 
192.168.42.0/24 dev usb0 scope link 
root@raspi-mptcp:~#

[matttbe]

Fullmesh should work.

How many subflows are created when you establish an MPTCP connection? cat /proc/net/mptcp_net/mptcp

How many IPs are listed by the FM PM? cat /proc/net/mptcp_fullmesh

[SriramScorp]

The num of subflows was set to 1. There are no statistics displayed except for the column headers when running cat /proc/net/mptcp_net/mptcp. Setting the number of subflows to a higher value (tried both '2' and '3') didn't change the cat output.

root@raspi-mptcp:~#
root@raspi-mptcp:~# 
root@raspi-mptcp:~# cat /proc/net/mptcp_net/mptcp 
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue in
ode
root@raspi-mptcp:~# 
root@raspi-mptcp:~#
root@raspi-mptcp:~# cat /proc/net/mptcp_fullmesh 
Index, Address-ID, Backup, IP-address, if-idx
IPv4, next v4-index: 5
3, 4, 0, 192.168.42.33 7
4, 5, 0, 192.168.42.180 8
IPv6, next v6-index: 1
root@raspi-mptcp:~# 
root@raspi-mptcp:~#
root@raspi-mptcp:~# cat /sys/module/mptcp_fullmesh/parameters/num_subflows 
1
root@raspi-mptcp:~# 

[VenkateswaranJ]

There are no statistics displayed except for the column headers when running cat /proc/net/mptcp_net/mptcp.

Are there any MPTCP connections that exist in your machine when you are running this command? Also, it has to be run with root privilege.

[matttbe]

There are no statistics displayed except for the column headers when running cat /proc/net/mptcp_net/mptcp.

Did you have active MPTCP connections running when you ran the cat command?
If yes, either the connection was not created with MPTCP options (sysctl net.mptcp.mptcp_enabled), the server doesn't accept MPTCP or a middlebox on the way is stripping MPTCP option (tracebox can help to detect such a thing, do not hesitate to look at other issues mentioning it)

[matttbe]

Also, it has to be run with root privilege.

Only to see the tokens but here we don't care about the tokens ;-)

[SriramScorp]

Yes, the commands were run with root privileges (please check the shell prompts). I initiated a file download from the LAN side of the mptcp-based router when running cat /proc/net/mptcp_net/mptcp. MPTCP is enabled on both server and client side.

root@raspi-mptcp:~# 
root@raspi-mptcp:~# 
root@raspi-mptcp:~# sysctl -a 2>/dev/null | grep mptcp
net.mptcp.mptcp_binder_gateways = 
net.mptcp.mptcp_checksum = 0
net.mptcp.mptcp_debug = 0
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = default
net.mptcp.mptcp_syn_retries = 1
net.mptcp.mptcp_version = 0
root@raspi-mptcp:~# 
root@raspi-mptcp:~# 
root@raspi-mptcp:~# curl http://www.multipath-tcp.org
Yay, you are MPTCP-capable! You can now rest in peace.
root@raspi-mptcp:~#

However, the ISP seem to be blocking mptcp, as suggested by tracebox. I guess I have to try with a different provider.

tracebox to 130.104.228.140 (multipath-tcp.org): 64 hops max
1: 192.168.42.1 0ms
2: *
3: *
4: *
5: *
6: *
7: *
8: *
9: *
10: *
11: *
12: *
13: *
14: *
15: *
16: *
17: 130.104.228.140 9195ms TCP::SrcPort (51931 -> 80) TCP::DstPort (80 -> 51931) TCP::SeqNumber (1117944487 -> 4091690734) TCP::AckNumber (0 -> 1117944488) TCP::DataOffset (10 -> 7) TCP::Flags (( SYN ) -> ( SYN ACK )) TCP::WindowsSize (5840 -> 29200) TCP::CheckSum (0xc145 -> 0x592) IP::DiffServicesCP (0 -> 10) IP::TotalLength (60 -> 48) IP::Identification (0x1085 -> 0x0) IP::TTL (17 -> 47) IP::CheckSum (0x779 -> 0xf9e1) IP::SourceIP (192.168.42.33 -> 130.104.228.140) IP::DestinationIP (130.104.228.140 -> 192.168.42.33) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300) -TCPOptionMPTCPCapable < TCPOptionMPTCPCapable (12 bytes) :: Kind = 30 , Length = 12 , Subtype = 0 , Version = 0 , Checksum = 1 (Checksum Enabled) , Flags = 0 , Crypto = 1 (HMAC-SHA1) , Sender's Key = Sender's Key = 5066262781819997082 , > TCPOptionWindowScale::Shift (14 -> 7)

[matttbe]

However, the ISP seem to be blocking mptcp, as suggested by tracebox. I guess I have to try with a different provider.

No it looks like the MPTCP options are received by the multipath-tcp.org server. Can you try with the IP/DNS name of your server?
Or look at packet traces on both the client and server if you control both ends?

[SriramScorp]

Asking out of curiosity, but doesn't -TCPOptionMPTCPCapable mean the headers are stripped? I do notice now that it is not happening at a middlebox but at the destination itself (multipath-tcp.org -> 130.104.228.140).

[matttbe]

In the output you provided, it is quite confusing because you only have the view of the end server. In fact, you can see the SYN+ACK (+MPCAPABLE) here, not the SYN (+MPCAPABLE). It tells you that the bytes in the MPCAPABLE option are no longer the same.

[SriramScorp]

Update: After using a different ISP in one of the modems, there is some output from cat /proc/net/mptcp_net/mptcp. But now the traffic seem to be flowing only via the new modem.

root@raspi-mptcp:~#
root@raspi-mptcp:~# 
root@raspi-mptcp:~# while sleep 2 ; do cat /proc/net/mptcp_net/mptcp ; done
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue in
ode
   0: 62D45125 6DFD62D  0 4F2AA8C0:58CD                         14D716A5:FE4D                         01 04 00000000:00000000 422
100
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue in
ode
   0: 62D45125 6DFD62D  0 4F2AA8C0:58CD                         14D716A5:FE4D                         01 04 00000000:00000000 422
100
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue in
ode
   0: 62D45125 6DFD62D  0 4F2AA8C0:58CD                         14D716A5:FE4D                         01 04 00000000:00002CF9 422
100
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue in
ode
   0: 62D45125 6DFD62D  0 4F2AA8C0:58CD                         14D716A5:FE4D                         01 04 00000000:00000000 422
100
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue in
ode
   0: 62D45125 6DFD62D  0 4F2AA8C0:58CD                         14D716A5:FE4D                         01 04 00000000:00000000 422
100
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue in
ode
   0: 62D45125 6DFD62D  0 4F2AA8C0:58CD                         14D716A5:FE4D                         01 04 00000000:00000000 422
100
^C
root@raspi-mptcp:~#

[matttbe]

ns seems to say you have 4 subflows. Is it not what you expected to get with 2*2 IPs?
Easier to check that with tcpdump (or netcat//proc/net/tcp) to check the different subflows that are being used.

[SriramScorp]

@matttbe, could you please clarify about TCPOptionMPTCPCapable ? When I ran tracebox with a different ISP modem (which I gather doesn't block mptcp), I don't see the minus sign. I have been relying on tracebox to check whether my ISP supports mptcp or not. Please tell me I've not been reading it wrong all this time.

tracebox to 130.104.228.140 (multipath-tcp.org): 64 hops max
1: 192.168.42.1 0ms
2: *
3: 10.0.244.125 9024ms [PARTIAL] +PartialTCP < PartialTCP (8 bytes) :: SrcPort = 10823 , DstPort = 80 , SeqNumber = 1383231730 , >
IP::DiffServicesCP (0 -> 26) IP::TTL (3 -> 1) IP::CheckSum (0xfbd8 -> 0xfd70)
4: 10.0.66.149 35ms [PARTIAL] +PartialTCP < PartialTCP (8 bytes) :: SrcPort = 10823 , DstPort = 80 , SeqNumber = 1383231730 , >
IP::DiffServicesCP (0 -> 26) IP::TTL (4 -> 1) IP::CheckSum (0xfad8 -> 0xfd70)
5: *
6: *
7: 149.14.224.161 315ms TCP::CheckSum (0x70c6 -> 0x7166) IP::DiffServicesCP (0 -> 26) IP::TTL (7 -> 1) IP::CheckSum (0xf7d8 -> 0xfd70) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300)
8: 130.117.48.205 204ms TCP::CheckSum (0x70c6 -> 0x7166) IP::DiffServicesCP (0 -> 10) IP::TTL (8 -> 1) IP::CheckSum (0xf6d8 -> 0xfdb0) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300)
9: 130.117.51.42 184ms TCP::CheckSum (0x70c6 -> 0x7166) IP::DiffServicesCP (0 -> 10) IP::TTL (9 -> 1) IP::CheckSum (0xf5d8 -> 0xfdb0) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300)
10: 154.54.60.134 393ms TCP::CheckSum (0x70c6 -> 0x7166) IP::DiffServicesCP (0 -> 10) IP::TTL (10 -> 1) IP::CheckSum (0xf4d8 -> 0xfdb0) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300)
11: 154.25.12.246 455ms TCP::CheckSum (0x70c6 -> 0x7166) IP::DiffServicesCP (0 -> 10) IP::TTL (11 -> 1) IP::CheckSum (0xf3d8 -> 0xfdb0) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300)
12: *
13: *
14: 193.191.3.86 343ms TCP::CheckSum (0x70c6 -> 0x7166) IP::TTL (14 -> 1) IP::CheckSum (0xf0d8 -> 0xfdd8) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300)
15: *
16: *
17: 130.104.230.56 304ms TCP::CheckSum (0x70c6 -> 0x7166) IP::TTL (17 -> 1) IP::CheckSum (0xedd8 -> 0xfdd8) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300)
18: 130.104.228.140 9375ms TCP::SrcPort (10823 -> 80) TCP::DstPort (80 -> 10823) TCP::SeqNumber (1383231730 -> 960686325) TCP::AckNumber (0 -> 1383231731) TCP::Flags (( SYN ) -> ( SYN ACK )) TCP::WindowsSize (5840 -> 28800) TCP::CheckSum (0x70c6 -> 0xfaa) IP::Identification (0x2992 -> 0x0) IP::TTL (18 -> 36) IP::CheckSum (0xecd8 -> 0x46b) IP::SourceIP (192.168.42.180 -> 130.104.228.140) IP::DestinationIP (130.104.228.140 -> 192.168.42.180) TCPOptionMaxSegSize::MaxSegSize (1460 -> 1300) TCPOptionMPTCPCapable::Sender's Key (Sender's Key = 5284479190991138687 -> Sender's Key = 11161987870401695007) TCPOptionWindowScale::Shift (14 -> 7)

[matttbe]

@SriramScorp oh yes, my bad, I thought it was saying the option was still there but the bytes were different (the json format is clearer :) -j | jq .).

So yes, good point, MPTCP options are certainly stripped in the middle. Can you check with another port number?

sudo tracebox -p 'IP/tcp{dst=443}/MPCAPABLE' multipath-tcp.org

Sometime there is an "HTTP optimiser" on port 80.
But we also saw in the past that it helps to contact the ISP and tell them MPTCP is blocked and it is used by million of devices now (mainly Apple).

[SriramScorp]

I tried aggregating network bandwidth with 2 modems using a different ISP which was not blocking MPTCP. Now, traffic seems to flow through both interfaces and aggregation seems to work without any issues.

Also, do you get a different IP each time?

While MPTCP seems to work with different interface IPs, I haven't tested if it would still work if all modems were to lease the same DHCP IP. Maybe, I could configure the interfaces with identical static IPs instead and post my findings later.

Can you check with another port number?

I haven't been able to test this either. But, assuming there is an "HTTP optimiser" as you suggested, will that help circumvent the MPTCP filtering by the ISP?

[matttbe]

I tried aggregating network bandwidth with 2 modems using a different ISP which was not blocking MPTCP. Now, traffic seems to flow through both interfaces and aggregation seems to work without any issues.

👍

Also, do you get a different IP each time?

While MPTCP seems to work with different interface IPs, I haven't tested if it would still work if all modems were to lease the same DHCP IP. Maybe, I could configure the interfaces with identical static IPs instead and post my findings later.

I initially thought your issue was that each modem was giving the same IP but you correctly identified the cause was different.

Can you check with another port number?

I haven't been able to test this either. But, assuming there is an "HTTP optimiser" as you suggested, will that help in somehow circumventing the MPTCP filtering by the ISP?

Good question. These optimisers are usually quite old and don't really make sense anymore now that a large part of the traffic is in HTTPS. I would not be surprised if they only look at the traffic on port 80.
But there are other "optimisers" on the market. Some don't support MPTCP, some support it but not by default, some firewall blocks MPTCP by default, etc. In other words, it is possible your ISP is blocking MPTCP not on purpose and telling them already helped unblocking MPTCP. It also helps to tell them Apple devices use it.