Write this article , Mainly for the convenience of future interview . Because my resume says , The biggest highlight of my last job was that the speed of unlocking the face was changed by 1200ms Optimized to 600ms, So this has been answered countless times . But I always feel that the answer is incomplete every time , Or the total feeling can play better , So here are some simple summary notes .
I 2018 year 4 Entered a mobile phone company in January , I have worked in it for more than two years , During this period, it is mainly responsible for the function of face unlocking . Speed optimization of face unlocking , It's a very important task to start with , It's been going on for a long time . Before optimization , The first is to have a clear goal , When I take over, the speed of face unlocking is 1200ms about , And we are the reference boutique machine Oppo、vivo、 millet 、 The unlocking speed of mainstream models such as Huawei is about 400~700ms Unequal , So this has become my goal of optimization .
The whole face unlocking architecture is shown below ：
The basic method of optimization is to split the whole start-up process into several stages , Then optimize for each stage , Optimize process logic or adopt more efficient technology , To sum up, the main steps are as follows ：
1、 From press power Key to SystemUI Medium KeyguardService
Before optimization , It was broadcast , But we know how to use broadcasting to achieve IPC The performance is very poor , slowly . After optimization , use AIDL Instead of broadcasting , promote 40ms about .
Some interviewers will ask ： Why use AIDL Faster than broadcasting ？
There is no authoritative answer on the Internet , I organized myself , I think there are several reasons for this ：（1） In the process of transmitting and receiving broadcasting , Many times Binder Realized IPC. Broadcasting is also used to realize cross process communication Binder Realized , This and AIDL equally . But the broadcast will register IntentFilter and Receiver Information through Binder Way to register to AMS in , There will be a lot of packaging in it 、 Filtering and other operations , take action,Receiver,Context、IntentFilter Association . When the sender sends a message, it will action And so on Intent in , And then through Binder Ways and AMS signal communication , take Intent Wait for information to come in AMS in .AMS After matching action, Find the corresponding registrant and recipient , And then through Binder The way and Receiver signal communication , This completes the transmission and reception of a broadcast , It happened many times Binder. and AIDL The way is much simpler , Directly between the sender and the receiver Binder that will do . Radio is one of the four components , There will be a lot of intermediate processing in the process of use , For example, yes. Intent In the middle of the process, etc .（2） There are two kinds of Broadcasting: foreground broadcasting and backstage broadcasting , The default is background broadcast . There are countless broadcasts inside the system , Due to the limited system resources , Priority will be given to some very important broadcasts , This allows the broadcast by default to handle these background broadcasts with low priority . Also due to limited system resources , So you can't set the broadcast to the foreground broadcast at will .（3） Depending on the way you register , There is a distinction between static registration and dynamic registration , If it's static registration , It's parallel broadcasting , If more than one place has registered for the broadcast , According to the registration time, broadcast events will be processed in turn . This is different from serial broadcasting , Dynamic registration is serial broadcasting .
2、SystemUI And FaceId Service Stay connected for a long time .
SystemUI And FaceId It's through AIDL To communicate , Before optimization ,SytemUI Every time and FaceId After communication, they will be disconnected , This will result in the next use of face unlock , The connection must be reestablished , There will be a certain degree of delay . The way to optimize is to let SystemUI and FaceId Stay connected for a long time . Of course, this does not shorten much time .
3、 Use Camera2 API Instead of Camera1 API
The system provides Camera API-2, It's official API-1 The optimization of the , More stable performance , It's more convenient to use . There is no official indication that API-2 than API-1 Faster , But in actual development, it was found that , Use API-2 After replacement , The whole process of turning on preview and camera is shortened 150ms about .
4、 Increase the camera frame rate , Try to reduce the waiting time of face unlocking algorithm
The core process of face unlocking is ：SDK The user's face data will be entered in advance , When you need to unlock , The camera obtains image information at a certain frame rate through the camera , adopt API The callback in passes the image information to SDK.SDK A face matching algorithm is encapsulated in , The algorithm will match the image information transmitted by the camera with the pre stored face data , And return the corresponding value according to the matching result , For example, the environment is too dark 、 No face detected 、 It's not the same person as the deposit 、 Matching success and other matching results , There is a number corresponding to it .
SDK In the match , If it works , The time of a match is about 30ms, If it's a failed match , One match is about 50ms~100ms Unequal （ In the process of matching , If there's new image data coming through , Will be filtered out ）. However, it can't match successfully at one time , After this match failed , Quickly take an image from the camera to match , Until at the appointed time （ The settings are 5s） Internal matching succeeded . In order to be able to get the next image quickly after this matching failure , This requires the camera to improve the frame rate of the image . This urges colleagues in the camera team to , Modify camera parameters , Under comprehensive consideration , Take it 15ms Frequency per frame .
5、 Set the initial exposure value of the camera reasonably
The camera will set an exposure value by default , When used in different environments , Then adjust according to the surrounding environment , To adapt to the surrounding light environment . If the default exposure value is not set properly , Will cause the camera to just turn on , The first few images you get are either too dark , Or it's too bright , Self adjustment is needed to achieve a high quality state .
In front of the optimization , Due to the unreasonable setting of the default exposure value , When the face unlock function is turned on , Even under normal light conditions , When facing the right face , Many previous matches are due to poor image quality , Cause matching failure . Every time a match fails, it's wasted 50~100ms Time for , This leads to every successful unlock , It's a waste 300ms Even more time is spent adjusting the exposure value of the camera itself .
In the later optimization , Through a lot of testing and Analysis log, We found the problem that every match can't succeed once , Then the data provided by the camera is converted into pictures , Only then did we find the quality problem of the image . Later, with colleagues from the camera team , Co debugging , Get a suitable initial exposure value , That solved the problem .
6、 Start when you start face unlock CPU Pull frequency , And reasonably deal with the relationship between speed and power saving
Face unlocking algorithm execution is a highly computationally intensive operation , In order to improve the speed of unlocking , Scheduling is used in the optimization process CPU The largest core , And improve CPU Frequency approach , The matching speed is improved .
Dispatch CPU Maximum core and improve CPU frequency , It's a very power consuming process . In order to better balance matching speed and power saving , Here's another design ： Face unlock timeout is set to 5s, But actually , If the surroundings are normal , And it's the right face , Most of the scenes can be in front of 1s Internal unlocking completed , Only when the environment is abnormal or the face is not correct , Will be in 1s After that, we need to match , The probability that this scenario can be unlocked is relatively small . So the approach here is , Before the face is unlocked 1s Dispatch CPU The largest core , If it's not unlocked yet , Will CPU Back to normal , Instead of using it all the time CPU Big core and high frequency .
7、 Reasonable use of parallel instead of serial
After successful face matching , You can immediately go through the unlock process , And call the camera off method , And turning off the camera requires 150ms Time for . Before optimization, the original developer used the serial method , That is, after the face matching is completed , In the same thread, the camera operation is called , After the camera is turned off, the lock screen interface disappears . This should be an obvious optimization point , The operation of closing the camera can be executed in a separate thread , In this way, it can be optimized again 150ms Time for . As for why the original developer used serial , Is not clear .
Of course , There are many other details of the optimization process , For example, time complexity optimization of some processes , Streamlining unnecessary processes , requirement sdk Face matching algorithm optimization , The new mobile phone uses better performance CPU、 Camera hardware, etc .