SIP 200 OK message send to local IP address for UDP protocol - not connecting

[kwaternion]

I am setting up a LiveKit and SIP servers on remote machine and having a connection problem. The latest source codes were fetched from GitHub, compiled with mage.
In a local environment (Linux, Ubuntu), it works fine.

Now, I want to deploy it to a Cloud environment (Compute Engine, GCP). All TCP/UDP ports are open on GCP Firewall.
Here are the configurations I used:

sip_config.yaml

api_key: devkey
api_secret: secret
redis:
 address: "localhost:6379"
ws_url: "ws://localhost:7880"
sip_port: 5060
rtp_port: 10000-20000
use_external_ip: true
# use_external_ip: false
# nat_1_to_1_ip: "34.73.104.205" 
# local_net: "10.132.0.0/24"
# listen_ip: "10.132.0.18" 
logging:
 level: 'debug'

livekit.yaml

port: 7880
rtc:
    tcp_port: 7881
    port_range_start: 50000
    port_range_end: 60000
    use_external_ip: true
    # use_external_ip: false
    # use_ice_lite: false
    enable_loopback_candidate: false
redis:
    address: localhost:6379
    db: 0
    use_tls: false
keys:
    devkey: secret

I am running the servers via command lines:

livekit-server --dev  --config=livekit.yaml
sip –config=sip_config.yaml

When I connect via a SIP client (Zoiper) using TCP protocol, everything works correctly, and I can establish voice communication.
However, when I switch to a more applicable SIP transport protocol, i.e., UDP, I am unable to establish a successful connection.

When I checked the logs on the client side, I see the SIP INVITE (outgoing message), then receiving 100 and 180 messages from the remote machine. However, the 200 OK message is not arriving at my SIP client - thus, the connection cannot be established.

I checked the logs on the Compute Engine side, and it appears that the 100 and 180 messages are sent to the correct public IP of my router, while the 200 message is being sent to my private IP address (inside NAT): 10.132.0.18.5060 > 192.168.0.22.64230. Please see the logs below.

I also tried a SIP client using a public IP, still the 200 message seems to be sent to the remote machine's local IP address range (10.132.0.18). When TCP protocol is used, then the destination IP address for 200 OK message is correct (public IP of my router).

I also tried using LiveKit's production deployment configuration (using Caddy, +STUN/TURN servers) - but the issue persists.
When I set up a simple SIP server on the remote machine (no STUN, TURN, IP discovery) based on emiago/sipgo, )to test firewalls, ports etc.), I am able to establish a successful connection with UDP transport (regardless if the Client is public or within NAT).

I'm not sure if this issue is related to the SIP server configuration or the LiveKit server RTP configuration or some internal bug. I've tried many configurations, but haven't had any success so far. I do not see any problems/errors in SIP server logs, while livekit server logs are quite hard to analyse.
Could you please take a look and possibly provide advice or fix the issue if it exists?

If helpful, I can also provide SIP logs and livekit logs.

Additional info

ifconfig:

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.132.0.18  netmask 255.255.255.255  broadcast 0.0.0.0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)

Logs from Remote Compute Engine:

23:22:37.804734 IP (tos 0x0, ttl 124, id 16038, offset 0, flags [none], proto UDP (17), length 900)
    176.41.132.68.64230 > 10.132.0.18.5060: [udp sum ok] SIP, length: 872
        INVITE sip:[email protected];transport=UDP SIP/2.0
        Via: SIP/2.0/UDP 192.168.0.22:64230;branch=z9hG4bK-524287-1---907a37c1204f5ea9;rport
        Max-Forwards: 70
        Contact: <sip:[email protected]:64230;transport=UDP>
        To: <sip:[email protected]>
        From: <sip:[email protected];transport=UDP>;tag=7257e67f
        Call-ID: yeN76BrsvXhFHulDQNEx1w..
        CSeq: 1 INVITE
        Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE
        Content-Type: application/sdp
        Supported: replaces, norefersub, extended-refer, timer, sec-agree, outbound, path, X-cisco-serviceuri
        User-Agent: Z 5.6.4 v2.10.20.4
        Allow-Events: presence, kpml, talk, as-feature-event
        Content-Length: 184

        v=0
        o=Z 0 133439700 IN IP4 176.41.132.68
        s=Z
        c=IN IP4 176.41.132.68
        t=0 0
        m=audio 61185 RTP/AVP 0 101
        a=rtpmap:101 telephone-event/8000
        a=fmtp:101 0-16
        a=sendrecv
        a=rtcp-mux

23:22:37.805382 IP (tos 0x0, ttl 64, id 14084, offset 0, flags [DF], proto UDP (17), length 387)
    10.132.0.18.5060 > 176.41.132.68.64230: [bad udp cksum 0x568f -> 0x5579!] SIP, length: 359
        SIP/2.0 100 Processing
        Via: SIP/2.0/UDP 192.168.0.22:64230;branch=z9hG4bK-524287-1---907a37c1204f5ea9;rport
        From: <sip:[email protected];transport=UDP>;tag=7257e67f
        To: <sip:[email protected]>
        Call-ID: yeN76BrsvXhFHulDQNEx1w..
        CSeq: 1 INVITE
        Content-Length: 0
        Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE

23:22:37.809419 IP (tos 0x0, ttl 64, id 14087, offset 0, flags [DF], proto UDP (17), length 405)
    10.132.0.18.5060 > 176.41.132.68.64230: [bad udp cksum 0x56a1 -> 0x6f57!] SIP, length: 377
        SIP/2.0 180 Ringing
        Via: SIP/2.0/UDP 192.168.0.22:64230;rport;branch=z9hG4bK-524287-1---907a37c1204f5ea9
        From: <sip:[email protected];transport=UDP>;tag=7257e67f
        To: <sip:[email protected]>;tag=SCL_mf6pUarMFKAp
        Call-ID: yeN76BrsvXhFHulDQNEx1w..
        CSeq: 1 INVITE
        Content-Length: 0
        Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE

23:22:37.843548 IP (tos 0x0, ttl 64, id 32405, offset 0, flags [DF], proto UDP (17), length 589)
    **10.132.0.18.5060 > 192.168.0.22.64230:** [bad udp cksum 0xcdc0 -> 0x0323!] SIP, length: 561
        SIP/2.0 200 OK
        Via: SIP/2.0/UDP 192.168.0.22:64230;branch=z9hG4bK-524287-1---907a37c1204f5ea9;rport
        From: <sip:[email protected];transport=UDP>;tag=7257e67f
        To: <sip:[email protected]>;tag=SCL_mf6pUarMFKAp
        Call-ID: yeN76BrsvXhFHulDQNEx1w..
        CSeq: 1 INVITE
        Content-Length: 204
        Contact: <sip:34.73.104.205:5060>
        Content-Type: application/sdp

        v=0
        o=- 0 2 IN IP4 34.73.104.205
        s=LiveKit
        c=IN IP4 34.73.104.205
        t=0 0
        m=audio 15584 RTP/AVP 0 101
        a=rtpmap:0 PCMU/8000
        a=rtpmap:101 telephone-event/8000
        a=fmtp:101 0-16
        a=ptime:20
        a=sendrecv

[dennwc]

Thank you for a detailed description of the issue! What version of LiveKit SIP are you using?

[kwaternion]

I've build the executables directly from the latest codes pulled yesterday (21.10.2024) from github (livekit/livekit and livekit/sip) - to have the latest codes. But also experiencing same issue with older codes.