AR App for Superimposed Video Calling

Having started the AR project, we proposed to analyze the existing prototype first, assuming that there might be issues with the algorithms in its core. The analysis proved this assumption was correct, so we offered to proceed with research and development activities. We moved forward from one working hypothesis to another, implementing, testing, and adjusting the algorithm accordingly to develop a working MVP of a real-time object recognition system.

The MVP we delivered allows for streaming videos, viewing the superimposed video, and configuring rear camera video stream parameters in the real time. Users start a video call and capture themselves using the cameras of their mobile devices. The central server gets two video streams from mobile applications, removes extraneous video elements, overlays video streams, and returns the mixed stream to end users’ mobile devices.

We started the AR project by testing the existing MVP and found several pitfalls in the original algorithm. The algorithm couldn`t remove a shadow from the object and converted an RGB image to grayscale. It detected a hand only on homogeneous and contrast background. Moreover, the recognition area was only 120x120 pixels. Based on static gesture recognition, the algorithm could not track the object motion.

To improve the existing recognition algorithm, we reviewed computer vision technologies available on the market and found that the OpenCV library compensated for the major recognition pitfalls existing in the MVP. With its in-built features, the solution could detect and mask an object properly on a solid color background: it measured object control points, tracked them, and intercepted redundant colors that didn’t conform to the values. Additionally, the problem of grayscale colors and recognition area cuts could be resolved.

Despite the merits, OpenCV library didn’t solve the problem of motion detection and tracking. In case of sudden moves or background changes, the object disappeared. The recognition quality dropped when lighting conditions and shooting angles changed.

To address the issue, we extended the OpenCV code with the following features:

• HSV and CMYK color models to improve recognition quality in poor or dynamic lighting conditions and non-uniform background.
• Gradient-based adaptive thresholding—dynamic approximation of the control points color values to better separate objects of interest from the background.
• Mixing of close color values in unrecognized areas inside and near the object contour to minimize the defects.

One of the major tasks within the AR project was to make the solution available on Nexus 6P and Google Pixel XL smartphones as well as Nexus 9 and Google Pixel C tablets. We conducted tests of the MVP using an exploratory method. Our team connected to the customers’ devices, tracked the event logs and fixed the algorithm in real time.

The list of problems we faced included inaccurate object masking, high transmission latency, and dark video on Google Pixel screens. We managed to fix all camera issues and were ready to deploy the app within the predicted deadline.