Video support

[Hollymingyi]

dear author,
could u help to telll me how to pull video meida from sip server and push video media to sip server? or do u have the plan to support video with sip module?

[Sean-Der]

Hi @Hollymingyi

would you mind explaining what you are doing more?

what SIP client are you using?

Do you need the SIP server to mix video, or can it be delivered as separate media sections?

[Hollymingyi]

hi sean,
what i want is that when i call into livekit room via freeswitch , i can communicate with livekit endpoint with video and audio together ,but now only audio support. Then i read the livekit/sip source code, i found that sdp message was maked like below：
[]*sdp.MediaDescription{
{
MediaName: sdp.MediaName{
Media: "audio",
Port: sdp.RangedPort{Value: rtpListenerPort},
Protos: []string{"RTP", "AVP"},
Formats: []string{"0", "101"},
},
Attributes: []sdp.Attribute{
{Key: "rtpmap", Value: "0 PCMU/8000"},
{Key: "rtpmap", Value: "101 telephone-event/8000"},
{Key: "fmtp", Value: "101 0-16"},
{Key: "ptime", Value: "20"},
{Key: "maxptime", Value: "150"},
{Key: "sendrecv"},
},
},
}
l
it means only audio support。

after all ,i want sip server to mix video into likekit room ,and separate media back to freeswitch.

[Sean-Der]

You are trying to connect Freeswitch + LiveKit?

Why are you running both servers? Would it be possible to only run one? What features are missing from either?

It would probably be more effective to use egress. That has composition and has more flexibility. If you can describe exactly what you are trying to build we would love to help!

[Hollymingyi]

ah ,if use ingress and egress to connect freeswitch and livekit , for example ,rtmp protocol ,the latency is more than 9s,its not suitable for our applications.    
right now , we can use sip softphone to connect livekit/sip and communicate with audio good.
but my hope is that we can use sip softphone to connect livekit/sip ,then we can communicate each other with audio and video. so if livekit/sip support video communication will be helpful for us.

[Hollymingyi]

[dennwc]

The most important detail is missing from the diagram - who is making SIP video call? How it produces and consumes video streams?

Sean is right to point out that passing m=video won't be enough. If device that sends SIP request can only handle one incoming video stream, LiveKit server would need to compose videos from multiple participants from the room. Thus, you would still need Egress for that room. Or active speaker detection at least (to select one stream).

[Hollymingyi]

hi dennwc，
yes,you r right. only m=video is not enough,LiveKit server would need to compose videos from multiple participants from the room,that's what i desire to do. As your suggestion，if we use egress to handle our problem, how should i to get egress and push to sip server, i would be happy to have a try， anyway thanks

[dennwc]

OK, got it. To emphasize, we don't have any plans to implement video for SIP.

However, if this turns out to be simple enough (e.g. pick one participant/egress, stream from it), then I might experiment with it when I have some free time.

There might be additional constraints here. For audio, we convert Ulaw <-> Opus in SIP server. If there's similar mismatch between video formats in SIP and LK, that will be a blocker. We definitely won't convert video in SIP server.

[Hollymingyi]

ah ,thanks anyway, our application wanna have video communication between sip and livekit, so i would prefer to get egress media from lk and then push it back to sip server (the sip invite have sdp,) via mediaconn. could u help to give me a plan or workflow to coding them?

[davidzhao]

Closing as we do not plan to support video with SIP.

[Hollymingyi]

hi david，
thanks anyway, when i publish h264 video into livekit ,the video is very fuzzy ,and the log like below:

2023-12-22T14:45:02.275+0800 INFO livekit.pub.sfu buffer/rtpstats_receiver.go:306 received sender report, out-of-order, resetting{"room": "91233445", "roomID": "RM_QCzP286EekLz", "participant": "Phone 89650050", "pID": "PA_L7PCC44n6mYa", "remote": false, "trackID": "TR_VCeqyecBzUPLHs", "relayed": false, "mime": "video/h264", "layer": 0, "last": "ntp: 2023-12-22 06:45:01.273902416 +0000 UTC, rtp: 1954390734, extRtp: 1954390734, at: 2023-12-22 14:45:01.274211609 +0800 CST m=+2591.952357079", "current": "ntp: 2023-12-22 06:45:02.274554252 +0000 UTC, rtp: 1954390045, extRtp: 1954390045, at: 2023-12-22 14:45:02.275287018 +0800 CST m=+2592.953432502", "count": 300}

is it something wrong?

[davidzhao]

Hey @Hollymingyi, if it's a general enquiry, please ask folks in our Slack community. I'm not able to tell what's happening based on limited logs.

[Hollymingyi]

hi david，
i have send pull request for my fork ,please have a check ,thank u very much

[Hollymingyi]

hi david，
could u tell me ，if i send h264 into livekit room，which payload type and profile-level-id should i set ?
it worked only when i set them like below:
m=video 17024 RTP/AVP 102
a=rtpmap:102 profile-level-id=42001f

the other payload type or profile-level-id cant work.

[Nitschi]

+1
As more and more Infrastructure supports SIP Video Clients, Video Support in Sip LiveKit would be extremely helpful for some of our upcoming usecases. If anyone has success with this, we would be very interested :)
In the meantime we will try our luck with a custom solution using SIPSorcery.

[dennwc]

This comes up quite frequently, so we would like to clarify the current state of the issue.

Current state

Video support for LiveKit SIP was considered, but we don't have a specific plan for it yet. LiveKit team is currently focused on other issues with higher priority.

Blockers

The reasons why we think it couldn't be implemented fast enough is:

Multiple participants

LiveKit rooms typically contain multiple participants, while SIP usually support only one video stream per call. This means that LiveKit SIP will have to mix video frames from the participants somehow. This is not trivial.

• One option would be to lock SIP video to the video track of the first participant. But I assume we will quickly get request for selecting which track is selected, for switching them via API or when participant leaves, etc. See below regarding switching.
• We could use active speaker detection to switch video track automatically. This is tricky, however, because video tracks may have different codecs, resolutions, etc. This effectively requires transcoding, since we cannot switch codec parameters on the flight after SDP.
• SIP could lock to a video track of an existing room composite egress. This should work well enough, but wouldn't be very intuitive. Also, we would have to switch to audio track from that egress to avoid A/V sync issues. This will increase the call latency for SIP participant.

Supported codecs

LiveKit currently supports the following codecs. We will have to either enforce selected codecs or transcode video in LK SIP.

• We don't have any data on SIP-native devices that support video to say if enforcing our set of codecs is reasonable. We assume that anything PC/tablet-like device can run either LiveKit Web or LiveKit Android client, thus we only consider other devices with native SIP video support.
• Video transcoding is expensive. We would like to avoid it in SIP, if possible.

Implementation

For anyone planning to contribute SIP video support, please consider the following:

• Video support should be optional. Disabling it should switch SDP to audio-only and lead to no additional performance impact.
• Similarly, if SIP peer only indicates audio support, video pipeline should be completely disabled for that call.
• First version may enforce LiveKit-supported video codecs and lock to a single participant's video track.
• We will not advertise video support officially, unless there's a solid plan for solving the issues mentioned in previous sections.

There's already a PR for an older LK SIP version which could act as a starting point: #35

Hope this resolves some of the questions regarding this issue. We are open to a discussion and/or contributions!