How accurate is your facial recognition system?
This is a question we’re often asked. And, when we respond, it is obvious that while the customer expects to simply hear a high percentage, such as 99%, they don’t really understand what this would mean in real terms.
The reason for this is that performance statistics quoted for any technology can generally be unclear at best, and at worst can be intentionally misleading. At The Face Recognition Company, we therefore try to answer this question in a more meaningful way by giving potential and current clients some understanding of what ‘accurate’ means in the context of facial recognition, and what they should expect from our solution, or any other.
Fundamentally, a live facial recognition system has several ‘faces’ loaded into a database and its primary job is to raise an alert, or trigger an action, when someone from the database is observed in the connected live camera feeds. Consequently, there are a couple of factors that need to be considered in relation to accuracy.
Can the system identify the person on the database when they appear in the camera feed, and does the system have the sensitivity to still correctly match the person when the image is less than perfect (for instance, if the face is at an awkward angle to the camera, if it is low resolution or an unclear image, or whether it is partially obscured or covered).
Does the system also ever match the people seen in the live feed incorrectly with those on the database?
These two factors tend to work in opposition to each other. The reason for this is that for every face that appears in view of the cameras, the software is calculating a ‘matching score’ for every face stored on the database. Every combination will have a score (we are all somewhat similar, as humans), while clearly, only some combinations are correct matches. Therefore, a threshold is set and all matches scoring above the threshold are then given to the operator as an alert – insofar as they are taken to be ‘correct’ matches. Match scores that fall below the threshold are discarded as the software infers that they are not a legitimate match.
If a system is not inherently very sensitive, with genuinely ‘matched’ persons of interest falling below the threshold for an actual match, a provider (or operator, if given control) may be tempted to lower the threshold, such that alerts are provided. However, the problem with lowering the threshold level of acceptance is that an incorrect match becomes increasingly likely.
This means that a system that correctly matches individuals on watchlists, but also produces incorrect matches, could on one measure be considered ‘accurate’ because it does identify the right person, but on another measure should be considered ‘inaccurate’, because it also makes lots of mistakes.
Why is this a problem? You could argue that it is still useful to get a number of alerts even if some are wrong as you will be ensuring that all the people on your watchlists are identified.
However, to the team at The Face Recognition Company, this simply sounds like an argument to justify poor performance. Beyond this, there are three very real and significant issues.
Every match, whether correct or not takes time for the responsible operator to check, respond and log.
Eventually, operators lose confidence in a system that is more frequently giving incorrect matches than correct ones. It feels a bit too hit-and-miss.
GDPR (General Data Protection Regulation) is very concerned with whether a system is incorrectly matching people, and rightly so. It could be incredibly damaging to an individual to be wrongly accused of being a shoplifter, or some other crime, or barred from a certain pub for violent behaviour or any other misdemeanour. On top of this, it would also be damaging to the brand using the software, as well as to the reputation of the technology in general.
At The Face Recognition Company, we strive to differentiate ourselves from other software providers on the market. We optimise our solution in favour of absolutely minimising the number of false matches. Our technology is good enough that we do not need to compromise sensitivity as a result. In fact, customers who have switched from using other systems to ours have informed us that our software is more sensitive, while "virtually” eliminating any false matches. In one example, a client was regularly receiving several hundred false matches per week with another face recognition solution; a number of weeks after installing our system (at the time of writing this article) they had not received a single false match, and sensitivity had increased too.
In real world terms, genuinely understanding system ‘sensitivity' is far more meaningful than an accuracy percentage, as ‘sensitivity’ cannot be quantified in one single number, so is a harder comparison to make. We state our system to be highly sensitive because it is able to correctly recognise a person even with sub-optimal images, as mentioned above.
Not all systems can do this, and certainly not to our level of sensitivity. The tendency from many providers is to boost their sensitivity by lowering their threshold, but at the expense of generating increased numbers of false matches. They are keen to show sensitivity but at the potential expense of the operator’s reputation and brand.
At The Face Recognition Company we do not settle for this. Our objective is to give all the sensitivity, without also giving false matches, even with multiple thousands of people on the database.
If you want to experience how our software works and how it could help your business, please contact us for an online demo.