FaceEngine¶

FaceEngine is a lightweight python library that provides an easy interface to work with face recognition tasks.

>>> from face_engine import FaceEngine
>>> engine = FaceEngine()
>>> engine.fit(['bubbles1.jpg', 'drive.jpg'], [1, 2])
>>> engine.make_prediction('bubbles2.jpg')
([(270, 75, 406, 211)], [1])

Installation¶

It is distributed on PyPi, and can be installed with pip:

$ pip install face-engine

FaceEngine is supported only on Python 3.6 and above.

Models¶

FaceEngine is built on top of three model interfaces Detector, Embedder and Estimator (see models), and leans on user provided implementations of these models.

Installation provides optional dlib models.

These implementations are using dlib python api and dlib provided pre-trained model files.

Note

FaceEngine installation is not installing dlib by default. To install it, either run pip install dlib (requires cmake) or follow build instructions.

Dlib models implementations are used to show how to work with FaceEngine. Questions and issues according to these models accuracy and performance please address to dlib.

To work with your own custom models you have to implement required models and import it. FaceEngine models are used to register all inheriting imported subclasses (subclass registration PEP 487).

For more information read the full documentation.

Getting started¶

Installation¶

It is distributed on PyPi, and can be installed with pip:

$ pip install face-engine

Note

FaceEngine is supported only on Python 3.6 and above.

Manually¶

Download or pull sources from project repository and run:

$ python setup.py install

Models¶

FaceEngine is built on top of three model interfaces Detector, Embedder and Estimator, and leans on user provided implementations of these models.

Default models¶

Installation provides optional dlib models.

These implementations are using dlib python api and dlib provided pre-trained model files:

Note

FaceEngine installation is not installing dlib by default. To install it, either run pip install dlib (requires cmake) or follow build instructions.

Dlib models implementations are used to show how to work with FaceEngine. Questions and issues according to these models accuracy and performance please address to dlib.

At the moment there is:

Custom models¶

To work with your own custom models you have to implement required models and import it with either directly importing your model or adding it to PYTHONPATH environment variable or using appropriate convenient function from face_engine.tools. This will register your model class object itself in face_engine._models dictionary, from where it becomes visible.

FaceEngine models are used to register all inheriting imported subclasses (subclass registration PEP 487).

For example to initialize FaceEngine with your own custom detector use appropriate keyword argument with model name:

from face_engine import FaceEngine
from my_custom_models import my_custom_detector
engine = FaceEngine(detector='custom_detector')

or use corresponding setter method with model name:

from face_engine import FaceEngine
from my_custom_models import my_custom_detector
engine = FaceEngine()
engine.detector = 'custom_detector'

Usage examples¶

These examples are using default models, therefore before trying it be sure that you have dlib installed. Also in some examples for simplicity purposes, exceptions are not handled.

Getting working instances¶

Working with pre-defined default models.

>>> from face_engine import FaceEngine
>>> engine = FaceEngine()
>>> engine.detector
'hog'
>>> engine.embedder
'resnet'
>>> engine.estimator
'basic'

Notice that if dlib is not installed detector and embedder models will be abstract.

>>> from face_engine import FaceEngine
>>> engine = FaceEngine()
>>> engine.detector
'abstract_detector'
>>> engine.embedder
'abstract_embedder'
>>> engine.estimator
'basic'

To work with your own custom detector use:

>>> from my_custom_models import my_custom_detector
>>> engine = FaceEngine(detector='custom_detector')

or use corresponding setter method:

>>> from face_engine import FaceEngine
>>> from my_custom_models import my_custom_detector
>>> engine = FaceEngine()
>>> engine.detector = 'custom_detector'

Face detection¶

Find all faces in the image with corresponding confidence scores.

from face_engine import FaceEngine
engine = FaceEngine()
bounding_boxes, extra = engine.find_faces('bubbles1.jpg')

Find largest face bounding box in the image with corresponding confidence score.

from face_engine import FaceEngine
engine = FaceEngine()
bounding_box, extra = engine.find_faces('bubbles1.jpg', limit=1)

Face recognition¶

These examples are using imread() function to read image as uint8 array.

Extract facial embedding vectors from the image.

from face_engine import FaceEngine, tools
engine = FaceEngine()
image = tools.imread('bubbles1.jpg)
bbs, extra = engine.find_faces(image)
embeddings = engine.compute_embeddings(image, bbs, **extra)

Predict class name by given face image.

Note

model has to be fitted before making any predictions.

from face_engine import FaceEngine
from face_engine.tools import imread
engine = FaceEngine()
engine.fit(['bubbles1.jpg', 'drive.jpg'], [1, 2])
image = imread('bubbles2.jpg')
bbs, extra = engine.find_faces(image)
embeddings = engine.compute_embeddings(image, bbs, **extra)
scores, class_names = engine.predict(embeddings)

Make (lazy) prediction to find out class names and bounding boxes in one call.

from face_engine import FaceEngine
engine = FaceEngine()
engine.fit(['bubbles1.jpg', 'drive.jpg'], [1, 2])
bounding_boxes, class_names = engine.make_prediction('bubbles2.jpg')

Persistence¶

Save engine state to file:

>>> from face_engine import FaceEngine
>>> engine = FaceEngine()
>>> engine.fit(['bubbles1.jpg', 'drive.jpg'], [1, 2])
>>> engine.save('engine.p')

Load engine state from file:

>>> from face_engine import load_engine
>>> engine = load_engine('engine.p')
>>> engine.make_prediction('bubbles2.jpg')
>>> ([(270, 75, 406, 211)], [1])

Application examples¶

These examples use opencv to read and visualize image data, so you may need to install it before.

Live face detection¶

import cv2
from face_engine import FaceEngine
from face_engine.exceptions import FaceNotFoundError

engine = FaceEngine()
cap = cv2.VideoCapture(0)

while True:
    ret, frame = cap.read()
    if ret:
        rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        try:
            bbs, _ = engine.find_faces(rgb_frame)
            for bb in bbs:
                bb = bb.astype(int)
                cv2.rectangle(frame, bb[:2], bb[2:], (0, 255, 0), 1)
        except FaceNotFoundError:
            pass
        # Display the resulting frame
        cv2.imshow('frame', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

cap.release()
cv2.destroyAllWindows()

Live face recognition¶

This example use make_prediction(), to find out only class names without prediction scores. To get prediction scores use predict() instead.

import cv2
from face_engine import load_engine
from face_engine.exceptions import FaceNotFoundError

# assume that engine is saved before
engine = load_engine('engine.p')
cap = cv2.VideoCapture(0)

while True:
    ret, frame = cap.read()
    if ret:
        rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        try:
            bbs, names = engine.make_prediction(rgb_frame)
        except FaceNotFoundError:
            pass # pass drawing
        else:
            # draw bounding boxes and predicted names
            for bb, name in zip(bbs, names):
                bb = bb.astype(int)
                cv2.rectangle(frame, bb[:2], bb[2:], (0, 255, 0), 1)
                cv2.putText(frame, name, (bb[0], bb[1] - 10),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 1)
        cv2.imshow('frame', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

cap.release()
cv2.destroyAllWindows()

API Documentation¶

Core API¶

FaceEngine core module.

face_engine.core.load_engine(filename)¶

Loads and restores engine object from the file.

This function is convenient wrapper of pickle.load() function, and is used to deserialize and restore the engine object from the persisted state.

Estimator model’s state is loaded separately and is loaded only if there is something saved before by save() method. Estimator model serialization (.save) and deserialization (.load) process steps are the responsibility of it’s inheriting class.

Parameters:	filename (str) – serialized by `save()` method file name
Returns:	restored engine object
Return type:	`FaceEngine`

class face_engine.core.FaceEngine(**kwargs)¶

Bases: object

Face recognition engine base class.

This object provides all steps and tools which are required to work with face recognition task.

Keyword arguments:

detector (str) – face detector model to use
embedder (str) – face embedder model to use
estimator (str) – face estimator model to use

detector¶

Returns:	detector model name
Return type:	str

embedder¶

Returns:	embedder model name
Return type:	str

estimator¶

Returns:	estimator model name
Return type:	str

save(filename)¶

Save engine object state to the file.

Persisting the object state as lightweight engine instance which contains only model name strings instead of model objects itself. Upon loading model objects will have to be re-initialized.

fit(images, class_names, **kwargs)¶

Fit (train) estimator model with given images for given class names.

Estimator’s fit() wrapping method.

Note

the number of images and class_names has to be equal
the image will be skipped if the face is not found inside

Parameters:	images (list[str]) – image file names or uri strings class_names (list) – sequence of class names kwargs – estimator model and data dependent
Returns:	self
Raises:	TrainError

predict(embeddings)¶

Make predictions for given embeddings.

Estimator’s predict() wrapping method.

Parameters:	embeddings (numpy.ndarray) – array of embedding vectors with shape (n_faces, embedding_dim)
Returns:	prediction scores and class names
Return type:	(list, list)
Raises:	TrainError

make_prediction(image, **kwargs)¶

Lazy prediction method to make prediction by given image.

Convenient wrapper method to go over all steps of face recognition problem by one call.

In particular:

find_faces() - detector
compute_embeddings() - embedder
predict() - estimator

Keyword arguments are all parameters of find_faces() method. Returns image all face bounding boxes with predicted class names. May raise same exceptions of all calling methods.

Parameters:	image (Union[str, bytes, file, os.PathLike, numpy.ndarray]) – RGB image content or image file uri.
Returns:	bounding boxes and class_names
Return type:	tuple(list, list)
Raises:	FaceNotFoundError
Raises:	TrainError

find_faces(image, limit=None, normalize=False)¶

Find multiple faces in the image.

Detector’s detect_all() wrapping method.

Parameters:	image (Union[str, bytes, file, os.PathLike, numpy.ndarray]) – RGB image content or image file uri. limit (int) – limit the number of detected faces on the image by bounding box size. normalize (bool) – normalize output bounding boxes
Returns:	face bounding box with shape (n_faces, 4), detector model dependent extra information.
Return type:	(numpy.ndarray, dict)
Raises:	FaceNotFoundError

compute_embeddings(image, bounding_boxes, **kwargs)¶

Compute image embeddings for given bounding boxes.

Embedder’s compute_embeddings() wrapping method.

Parameters:	image (numpy.ndarray) – RGB image with shape (rows, cols, 3) bounding_boxes (numpy.ndarray) – face bounding boxes kwargs – model dependent
Returns:	array of embedding vectors with shape (n_faces, embedding_dim)
Return type:	numpy.ndarray

Exceptions¶

Handled exceptions raised by FaceEngine

exception face_engine.exceptions.FaceNotFoundError¶

Bases: Exception

Raised when the face is not found in the image

exception face_engine.exceptions.TrainError¶

Bases: Exception

Raised when the fit(train) process is failed

Models API¶

class face_engine.models.Model¶

FaceEngine model base class. Used to register all inheriting and imported subclasses (subclass registration PEP 487).

Note

implementing model classes must have name class descriptor

Detector¶

class face_engine.models.Detector¶

Human face detector model base class.

Note

bounding box format is (left, upper, right, lower)

detect(image)¶

Detect all faces in the image.

Parameters:	image (numpy.ndarray) – RGB Image with shape (rows, cols, 3)
Returns:	bounding boxes with shape (n_faces, 4), detector model dependent extra information.
Return type:	(numpy.ndarray, dict)
Raises:	FaceNotFoundError

name = 'abstract_detector'¶

Embedder¶

class face_engine.models.Embedder¶

This object calculates embedding vectors from the face containing image.

Note

implementing model classes should have dim class descriptor

compute_embeddings(image, bounding_boxes, **kwargs)¶

Compute image embeddings for given bounding boxes

Parameters:	image (numpy.ndarray) – RGB image with shape (rows, cols, 3) bounding_boxes (numpy.ndarray) – bounding boxes with shape (n_faces, 4) kwargs – model dependent
Returns:	array of embedding vectors with shape (n_faces, embedding_dim)
Return type:	numpy.ndarray

name = 'abstract_embedder'¶

Estimator¶

class face_engine.models.Estimator¶

Estimator model base class. Used to make predictions for face embedding vectors.

fit(embeddings, class_names, **kwargs)¶

Fit (train) estimator model with given embeddings for given class names.

Note that the passed number of samples for embbedings and class_names has to be equal.

Parameters:	embeddings (numpy.ndarray) – face embedding vectors with shape (n_samples, embedding_dim) class_names (list) – sequence of class names kwargs – model and data dependent
Returns:	self
Raises:	TrainError

predict(embeddings)¶

Make predictions for given embeddings.

Note

Model previously has to be fitted.

Parameters:	embeddings (numpy.array) – array of embedding vectors with shape (n_faces, embedding_dim)
Returns:	prediction scores and class names
Return type:	(list, list)
Raises:	TrainError

name = 'abstract_estimator'¶

save(dirname)¶

Persist estimators’s model state to given directory.

File naming format convention:: name = '%s.estimator.%s' % (self.name, ext)

load(dirname)¶

Restore estimator’s model state from given directory.

File naming format convention:: name = '%s.estimator.%s' % (self.name, ext)

Implementations¶

Dlib models¶

class face_engine.models.dlib_models.HOGDetector¶

Dlib “Histogram Oriented Gradients” model.

Note

bounding box sizes are equal for all detections.
detector does not provide confidence scores for detections.

name = 'hog'¶

class face_engine.models.dlib_models.MMODDetector¶

Dlib pre-trained CNN model with “Max-Margin Object Detection” loss function.

Note

bounding box sizes are equal for all detections.
to run in realtime requires high-end Nvidia GPU with CUDA/cuDNN.

References:

name = 'mmod'¶

class face_engine.models.dlib_models.ResNetEmbedder¶

Dlib pre-trained face recognition ResNet model.

Note

face alignment pre-processing used with 5 point shape_predictor.

References:

name = 'resnet'¶

Basic estimator¶

class face_engine.models.basic_estimator.BasicEstimator¶

Basic estimator model makes predictions by linear comparing each source embedding vector with each fitted embedding vectors.

Model is using python pickle module to persist estimator state. Default file name is 'basic.estimator.p'.

name = 'basic'¶

Tools¶

face_engine.tools.imread(uri, mode=None)¶

Reads an image from the specified file.

References:

Parameters:	uri (Union[str, bytes, file, os.PathLike]) – image file, file path or url mode (str) – format mode to convert the image to. More info on accepted mode types see on [1].
Returns:	image content as unsigned 8-bit numpy array
Return type:	`uint8`

face_engine.tools.import_module(filepath)¶

Convenient function to import module by given filepath.

Note that this function is using file path unlike importlib’s import_module.

Parameters:	filepath (Union[str, bytes, os.PathLike, Path]) – absolute or relative filepath

face_engine.tools.import_package(package)¶

Convenient function to import all modules of given package except __init__.py

Parameters:	package (Union[str, bytes, os.PathLike, Path]) – absolute or relative filepath to the package

Fetching tools¶

Fetching tools. Used to download and unpack project models and testing data.

face_engine.fetching.fetch_file(url, extract_dir=None)¶: Fetch file by URL to extract_dir folder

face_engine.fetching.unpack_archive(filename, extract_dir=None)¶

shutil.unpack_archive wrapper to unpack [‘.dat.bz2’] archive.

Parameters:	filename – name of the archive. extract_dir – name of the target directory, where the archive is unpacked. If not provided, the current working directory is used.

Reference¶

Recognition point¶

Facial recognition like most of current (digital) recognition techniques is based on these steps:

Face detection (detector) - is an essential step as it detects and locates human faces in images and videos. At the moment there are dozens of techniques and solutions starting from Haar-like cascades to deep neural networks. The point is to answer a question: is there a face in this picture? Finally, face detection determines the presence, location of the face as a bounding box and (possibly) the confidence score of how likely is it a human face.
Face digitization (embedder) - so called technique that transforms analogue information (a picture of face) into a set of digital information based on unique person’s facial features, to be more precise - calculates the facial features as n-dimensional embedding vectors to uniquely locate it in n-dimensional space.
Making prediction (estimator) - the last step of facial recognition accumulates previously calculated data and compares it with already existing ones. Comparing as 1to1 (verification task) or 1toN (identification task) does not differ from other machine learning tasks, the key is to make some prediction on given (digital) data.