编程知识 cdmana.com

Talk about webrtc video call

WebRTC Provides a set of standards API, send Web The application can directly provide real-time audio and video communication function . Most browsers and operating systems support WebRTC, You can directly initiate real-time audio and video calls on the browser , This article takes WebRTC A beginner's perspective to complete a 1V1 Web version of real-time audio and video calls .

To complete audio and video calls, you need to understand four modules : Audio and video collection 、STUN/TURN The server 、 Signaling server 、 End to end P2P Connect . Use WebRTC Of API Complete audio and video acquisition , Cooperate with signaling server and WebRTC Of RTCPeerConnection Method can achieve 1V1 conversation , The simple flow chart is as follows :

Next, explain their functions and core API.

Audio and video collection

Collect and play the portrait and sound of the call

WebRTC Use getUserMedia Get the media stream object corresponding to the camera and microphone MediaStream, Media streaming can be done through WebRTC transmitted , And share among multiple peers . Assign the stream object to the of the video element srcObject, Realize local playback of audio and video

attribute

meaning

width

The width of the video

height

The height of the video

aspectRatio

The proportion

frameRate

Frame rate

facingMode

Mirror mode

resizeMode

Size mode

API:navigator.mediaDevices.getUserMedia
Parameters :constraints
return :promise, Method call succeeded MediaStream object .

const localVideo = document.querySelector("video");

function gotLocalMediaStream(mediaStream{
  localVideo.srcObject = mediaStream;
}

navigator.mediaDevices
  .getUserMedia({
      video: {
        width640,
        height480,
        frameRate:15,
        facingMode'enviroment'//  Set to rear camera
        deviceId : deviceId ? {exact:deviceId} : undefined
      },
      audiofalse
   })
  .then(gotLocalMediaStream)
  .catch((error) => console.log("navigator.getUserMedia error: ", error));

Connection management

Know how to capture local audio and video , Next, learn how to establish a connection with the other end to transmit audio and video data .

RTCPeerConnection yes WebRTC Network connection 、 Media management 、 Unified interface for data management . establish P2P Connection required RTCPeerConnection Several important classes in :SDPICESTUN/TURN.

  1. Session description information RTCSessionDescription(SDP)

SDP Is the capability of each end , Include audio codec types 、 Transmission protocol, etc . This information is necessary to establish a connection , Both parties know whether the video supports audio 、 What is the encoding method , Can pass. SDP get .

For example, video transmission , My code is H264 The other party can only solve H265, There's no way to communicate .

SDP The description is divided into two parts , They are session level descriptions (session level) And media level descriptions (media level), Its specific composition can be referred to RFC4566[1], With an asterisk (*) It's optional . The common contents are as follows :

Session description( Session level description )
v= (protocol version)
o= (originator and session identifier)
s= (session name)
c=* (connection information -- not required if included in all media) One or more Time descriptions ("t=" and "r=" lines; see below)
a=* (zero or more session attribute lines) Zero or more Media descriptions
Time description
t= (time the session is active)

Media description( Media level description ), if present
m= (media name and transport address)
c=* (connection information -- optional if included at session level)
a=* (zero or more media attribute lines)

SDP When parsing , Every SDP Line It's all about key=... form , It is concluded that key yes a after , There may be two ways , May refer to RFC4566[2]

a=<attribute>
a=<attribute>:<value>

Sometimes it's not a colon (:) It must be <attribute>:<value>, actually value There will also be colons in it , such as :

a=fingerprint:sha-256 7C:93:85:40:01:07:91:BE
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=ssrc:2527104241 msid:gLzQPGuagv3xXolwPiiGAULOwOLNItvl8LyS

Look at specific examples :

alert(pc.remoteDescription.sdp);

 v=0
 o=alice 2890844526 2890844526 IN IP4 host.anywhere.com
 s=
 c=IN IP4 host.anywhere.com
 t=0 0
 // The following media description , The media description part includes audio and video media
 m=audio 49170 RTP/AVP 0
 a=fmtp:111 minptime=10;useinbandfec=1 // Description of format parameters
 a=rtpmap:0 PCMU/8000 // Yes RTP Description of the data
...
 // Above is the audio media description , The following is the video media description
 m=video 51372 RTP/AVP 31
 a=rtpmap:31 H261/90000
 ...
 m=video 53000 RTP/AVP 32
 a=rtpmap:32 MPV/90000
  1. ICE candidates RTCIceCandidate

WebRTC The most convenient way of point-to-point connection is both sides IP Direct connection , But in practical applications , The two sides will be separated NAT The device caused trouble getting the address .

WebRTC adopt ICE The framework determines the best path to establish network connection at both ends , It shields developers from complex technical details .

(NAT And ICE Framework for use WebRTC The developer is a black box , In order to optimize the reading experience, this part is put at the end as supplementary knowledge )

Developers need to understand

  1. principle

Two node switching ICE Candidates to negotiate how they connect , Once both sides agree on a mutually compatible candidate , The candidate SDP Is used to create and open a connection , The media stream starts running through this connection .

  1. Two API

onicecandidate: Local agent creation SDP Offer And call setLocalDescription(offer) Post trigger , stay eventHandler The candidate information is transmitted to the remote end through the signaling server .

addIceCandidate: After receiving the candidate information sent by the signaling server, the call is sent. , Add for native ICE agent .

API:pc.onicecandidate = eventHandler
pc.onicecandidate = function(event{
  if (event.candidate) {
    // Send the candidate to the remote peer
  } else {
    // All ICE candidates have been sent
  }
}


API:pc.addIceCandidate
pc.addIceCandidate(candidate).then(_=>{
  // Do stuff when the candidate is successfully passed to the ICE agent
}).catch(e=>{
  console.log("Error: Failure during addIceCandidate()");
});

Signaling server

WebRTC Of SDP and ICE Information needs to depend on Signaling server Message transmission and exchange 、 establish P2P Connect , Then you can make audio and video calls 、 Transmitting text messages . If there is no signaling server ,WebRTC Unable to communicate .

Usually use socket.io The ability of real-time communication to build a signaling server .socket.io Cross platform 、 Across the terminal 、 Cross language , It is convenient for us to realize each end of signaling at each end , To connect with our server .

This figure shows the role of the signaling server in the whole call process .

Use the code to see how to build socket.io Signaling server

var express = require("express");
var app = express();
var http = require("http");
const { Server } = require("socket.io");
const httpServer = http.createServer(app);
const io = new Server(httpServer);

io.on("connection", (socket) => {
    console.log("a user connected");
    socket.on("message", (room, data) => {
      logger.debug("message, room: " + room + ", data, type:" + data.type);
      socket.to(room).emit("message", room, data);
    })
    socket.on("join", (room) => {
      socket.join(room);
    })
});

End to end P2P Connect

  1. Connection process

A and B The process of establishing network connection is shown in the figure :

  • A towards B launch WebRTC call
  • establish peerConnection object , Specify... In the parameter Turn/Stun The address of
var pcConfig = {
  iceServers: [
    {
      urls"turn:stun.al.learningrtc.cn:3478",
      credential"mypasswd",
      username"garrylea",
    },
    {
      urls:[
        "stun:stun.example.com",
        "stun:stun-1.example.com"
      ]
    }
  ],
};

pc = new RTCPeerConnection(pcConfig);
  • A call createOffer Method to create a local session description (SDP offer),SDP offer Contains information that has been attached to WebRTC conversation , All codecs and options supported by the browser MediaStreamTrack Information , as well as ICE agent , The purpose is to send to potential remote endpoints through signaling channels , To request a connection or update the configuration of an existing connection .
  • A call setLocalDescription Method to set the proposal to the local session description , And pass it on to ICE layer . Then, the session description is sent to the user through the signaling server B
API:pc.createOffer
Parameters : nothing
return :SDP Offer

API:pc. setLocalDescription
Parameters :offer
return :Promise<null>

function sendMessage(roomid, data{
  if (!socket) {
    console.log("socket is null");
  }
  socket.emit("message", roomid, data);
}

const offer = await pc.createOffer()
await pc.setLocalDescription(offer).catch(handleOfferError);
message.log(` Transmission initiator local SDP`);
sendMessage(roomid, offer);
  • A End pc.setLocalDescription(offer) After creation , One icecandidate The event is sent to RTCPeerConnection, onicecandidate Event will be triggered .B The client receives a new message sent by signaling from the remote page ICE Candidate address information , This machine can be called by RTCPeerConnection.addIceCandidate() To add a ICE agent .
//A End 
pc.onicecandidate = (event) => {
if (!event.candidate) return;
sendMessage(roomid, {
type: "candidate",
label: event.candidate.sdpMLineIndex,
id: event.candidate.sdpMid,
candidate: event.candidate.candidate,
});
};


//B End
socket.onmessage = e => {
if (e.data.hasOwnProperty("type") && e.data.type === "candidate") {
var candidate = new RTCIceCandidate({
sdpMLineIndex: data.label,
candidate: data.candidate,
});
pc.addIceCandidate(candidate)
.then(() => {
console.log("Successed to add ice candidate");
})
.catch((err) => {
console.error(err);
});
}
}
  • A Get local media stream as caller , call addtrack Method to add audio and video streams RTCPeerConnection Object to the other end , Triggered at the other end when joining ontrack event .
 Add media stream to media track 

API:stream.getTracks
Parameters : nothing
return : Media track object array

const pc = new RTCPeerConnection();
stream.getTracks().forEach((track) => {
  pc.addTrack(track, stream);
});

const remoteVideo = document.querySelector("#remote-video");
pc.ontrack = (e) => {
  if (e && e.streams) {
    message.log(" Receive each other's audio / Video streaming data ...");
    remoteVideo.srcObject = e.streams[0];
  }
};
  • B As caller , Received from signaling server A Session information sent , call setRemoteDescription Method to pass the proposal to ICE layer , call addTrack Methods add RTCPeerConnction
  • B call createAnswer Method to create a reply , call setLocalDeacription Method reply is set to the local session and passed to ICE layer .
socket.onmessage = e => {
    message.log(" Sender received SDP");
    await pc.setRemoteDescription(new RTCSessionDescription(e.data));
    message.log(" Create receiver ( The reply )SDP");
    const answer = await pc.createAnswer();
    message.log(` Transmission receiver ( The reply )SDP`);
    sendMessage(roomid, answer);
    await pc.setLocalDescription(answer);
}

  • AB Both have their own and each other's SDP, Agreement on media exchange , Collected ICE Complete connectivity detection and establish the latest connection mode ,P2P Connection is established , Get the other party's audio and video media stream .
pc.ontrack = (e) => {
if (e && e.streams) {
message.log(" Receive each other's audio / Video streaming data ...");
remoteVideo.srcObject = e.streams[0];
}
};
  1. Bidirectional data channel connection

RTCDataChannelton adopt RTCPeerConnection API Point to point can be established P2P interconnection , No mediation server is required , Less delay .

One end establishment datachannel, The other end passes ondatachannel obtain datachannel object

API:pc.createDataChannel
Parameters :label   Channel name
      options?   Channel parameters
return :RTCDataChannel


function receivemsg(e{
  var msg = e.data;
  if (msg) {
    message.log("-> " + msg + "\r\n");
  } else {
    console.error("received msg is null");
  }
}

const dc = pc.createDataChannel("chat");
dc.onmessage = receivemsg;
dc.onopen = function ({
  console.log("datachannel open");
};
dc.onclose = function ({
  console.log("datachannel close");
};


pc.ondatachannel = e => {
  if(!dc){
    dc = e.channel;
    dc.onmessage = receivemsg;
    dc.onopen = dataChannelStateChange;
    dc.opclose = dataChannelStateChange;
  }
}; // This method is called back when the docking creates a data channel .

NAT And ICE frame ( Add )

Mentioned above ICE Integrated multiple NAT Crossing technology , such as STUN、TURN, Can achieve NAT through , Discovery between hosts P2P Transmission path mechanism . Next, I'll give you a brief introduction NAT、STUN、TURN What is it? .

  1. Network address translation ( NAT)

NAT Often deployed at an organization's network exit location . The network is divided into private network and public network ,NAT Gateway is set at the route exit from private network to public network , The two-way data between private network and public network must pass through NAT gateway . A large number of equipment within the organization , adopt NAT You can share a public network IP Address , It's solved IPv4 The problem of insufficient address .

As shown in the figure below , There are two organizations , Every organization's NAT Assign a public network IP, Namely 1.2.3.4 as well as 1.2.3.5. Each organization's private network device passes through NAT Convert intranet address to public address , Then join the Internet

NAT treat UDP The ways to achieve this are 4 Kind of , Respectively : Completely conical 、 Address restricted cone 、 Port limited cone 、 Symmetrical type .

  1. Session Traversal Utilities for NAT (STUN)

STUN Allow in NAT( Or multiple NAT) After the client to find out their own public network address , Find out what kind of NAT After that and NAT The public network port bound to a local port .

STUN yes C/S Protocol of pattern , Sent by client STUN request 、STUN Service response notification by NAT Assigned to the host IP Address and port number , It is also a kind of Request/Response The agreement , The default port number is 3478.

Want the intranet host to know its external network IP, It is necessary to set up one on the public network STUN server, And send it to this server Request, The server will return to its public network IP 了 .

Here's a pair of grabs STUN Binding requests and responses . First, the client sends a message to the address 216.93.246.18 Of STUN Server send Binding Request(STUN Binding request ).

The server returned Binding Response, Return to public network IP:

  1. traversal Using Relay NAT(TURN)

TURN Is a data transmission protocol . Allowed to pass through TCP or UDP Mode penetration NAT Or firewalls .TURN It's a Client/Server agreement .TURN Of NAT Penetration method and STUN similar , It is achieved by obtaining the public network address in the application layer NAT through

  1. ICE collect

ICE Both sides don't know the location of the network and NAT type , adopt ICE Can dynamically find the optimal transmission path .ICE The local address is collected by the client 、 adopt STUN Service collection NAT Internet address 、 adopt TURN Collect relay addresses , So there are three candidate addresses :

host type , That is, the of the local intranet IP And port ;

srflx type , This is the machine NAT Of mapped Internet IP And port ;

relay type , The name of the relay server IP And port .

{
    IP: xxx.xxx.xxx.xxx,
    port: number,
    type: host/srflx/relay,
    priority: number,
    protocol: UDP/TCP,
    usernameFragment: string
    ...
 }

The following figure ,Alice And Bob adopt STUN as well as TURN The server collects three types of data candidate.

ICE collect candidate Connectivity detection after , Determine between hosts P2P Optimal transmission path .

effect

demo project :https://code.byted.org/yuanyuan.wallace/WebRTC/tree/master/demo

Reference resources

https://developer.mozilla.org/zh-CN/docs/Web/API/WebRTC_API

https://github.com/shushushv/webrtc-p2p

https://www.cnblogs.com/pannengzhi/p/5061674.html

[1]

RFC4566: https://tools.ietf.org/html/rfc4566#page-24

[2]

RFC4566: https://tools.ietf.org/html/rfc4566#page-24


版权声明
本文为[Space programming]所创,转载请带上原文链接,感谢
https://cdmana.com/2021/10/20211002145644399Q.html

Scroll to Top