Creating an Android Face Filter App Using Banuba Face AR SDK

What are Augmented Reality Face Filters?

Augmented Reality — or AR is an important area in today's software development industry. It is defined by the Oxford dictionary as a technology that combines computer-generated images on a screen with the real object or scene that you are looking at. AR is capable of overlaying virtual objects on top of real-world views, for example, AR filters over people's faces, and is often deployed for various purposes in education, e-commerce, advertising, video conferencing, and other areas.

When it comes to virtual education, face filters can be used to help lecturers increase the attendance of online classes, let students have fun, and help them better visualize and understand the topic they're studying. AR filters used in eCommerce allow customers to try on different products virtually, e.g., cosmetics or accessories, without the need of visiting the brick and mortar stores. In the advertising field, this technology makes it possible for brands to reach their target market audience, without paying for the development of a separate app. Last but not least, Face AR filters can be used in video conferencing apps to make video calls more interesting, personalized, and engaging.

Mobile applications such as beauty and video editors, virtual try-ons and video communication apps often employ face filters for entertainment and product testing purposes.

This article is going to provide a step-by-step guide on how to create an Android face filter app using Banuba Face AR SDK. We will also discuss how face filters work and the advantages of using Banuba Face Filter Catalogue for implementing face filters in your app.

How AR face filters work?

From the user perspective, a face filter app running either on a mobile device or web interface uses the camera to detect the presence of a face in the real-time and overlays digital objects and animations on it. The user can take a photo or record a video augmented with Face AR effects and share it with other people on social networks.

On the technical side, AR face filters are implemented using face tracking algorithms that track the position of face landmarks and a 3D rendering engine that displays the AR filter the way the user sees it. The face tracking technology determines the accuracy and the quality of the face filter output, while the 3D rendering is responsible for the “beauty” and professional look of your effect.

What Is Banuba Face AR SDK?

Banuba Augmented Reality SDK is a set of AR libraries for desktop and mobile applications that help developers implement AR face filters, lenses and other AR effects in their apps. The difference between filters and lenses is that the latter are augmented reality animations, whereas the former is static image overlays.

The Banuba SDK takes advantage of computer vision to help users create different face tracking applications. It allows developers to build various types of AR effects, including face filters with morphing effects, virtual try-ons, face beautification effects, and much more.

The library includes three main components:

• Augmented Reality SDK. The SDK property, which consists of a set of libraries written in Java with the pack of Face AR features such as face filters, beautification, virtual makeovers and 3D backgrounds included in it;
• Filter Editor & Viewer. The designer tool to set up and preview filters in real-time on your desktop computer.

A screenshot illustrating the interface of Banuba Filter Editor. This tool helps users to set effect properties and their behaviour, add audio and beautification components as well as effect triggers. 

A screenshot illustrating Banuba Viewer which allows developers to set up and preview filters in real-time.

Both Filter Editor & Viewer which are separate tools can be used during the project creation stage we're going to discuss in the upcoming section.

• Catalogue. The collection of pre-designed 3D assets that can be purchased/licenced and enabled in any app using Banuba SDK. Developers can request access to it by filling out the contact form on Banuba's official website.

Some of the typical challenges involved in building an Android Face Filter app which can be solved with a help of Banuba Augmented Reality SDK are listed below:

• The time it takes for developers to start coding an Face Filter app. A well-implemented SDK gets developers up and running quickly with the coding process by providing them ready-made APIs, along with examples of coding best practices and documentation.
• The need to develop applications for different mobile and desktop platforms separately, which adds additional time and effort. Banuba's AR SDK employs universal code which can be used to build applications for Android, iOS, Mac OS, Windows, or Unity.
• The need for additional time and resources to implement new features such as 3D masks, AR face filters or AR lenses. Using a third-party SDK simplifies this process, maximizing your in-app engagement and growing your business by simply integrating new functions into an app.

How To Create An Android Face AR App

The first step to build an Android face filter app with the Banuca SDK is to create a Face AR project.

This is a list of required tools for that:

• The latest stable version of Android Studio (version 4.1.2 for Mac and Windows);
• The latest stable version of Gradle Plugin (version 4.1 for Mac and Windows)
• The latest stable version of Android NDK (version r21d for Mac and Windows)

You can create a Face AR project on Android in four steps, which we describe below.

Step 1. Get the client token and configuration file

To use Banuba SDK, you will need a client token along with a JSON configuration file which lists the required SDK resources associated with the token. To request one, please contact Banuba's company's representatives by filling out the form on banuba.com.

Once you have your token, you will need to insert it inside a file named BanubaClientToken.java, as shown below. Here is an example of how this client token can be used, as shown below:

final class BanubaClientToken {
    public static final String KEY = "YOUR_TOKEN_HERE";

    private BanubaClientToken() {
    }
}

Step 2. Download the Banuba SDK archive

Once you receive a client token, you will be granted access to Banuba SDK’s archive for Android, consisting of:

• Banuba Effect Player ( compiled Android library project with .aar extension);
• Banuba SDK (compiled Android library project with .aar extension);
• Effect examples located under the effects folder.

Step 3. Repack your SDK archive (minify SDK size)

The SDK release archive contains all SDK resources you will need to start creating a Face AR project. They may consume additional disk space in your released build, though.

To reduce its size, use the sdk_repacking.py script provided with the SDK archive. Refer to SDK repacking readme file in your SDK archive for more information and usage examples.

Step 4. Start building your project with Banuba AR SDK

(a) Create the libs directory in your project and add both the banuba_effect_player.aar and banuba_sdk.aar files:

(b) Open the plugin named build.grandle (Module: app) and add all dependencies required for your project:

(c) Add the BanubaClientToken.java file to your project:

(d) Check Declared Dependencies in Project Structure:

(e) Also check Android Demo app to use some code snippets.

The example presented here is a simple application with a Face AR mask written in Kotlin. Likewise, Banuba's Augmented Reality SDK contains a fully-functional app developed in Java. Developers can leverage it to create apps written in either Kotlin or Java.

Now you can run your project with Banuba SDK on your Android device.

How To Make An Android Face Filter App

Now we turn to the process of building the actual face filter app on Android.
 

Step 1. Import the Android project under src folder in Android Studio:

Step 2. Import banuba_effect_player-release.aar file by going to "File → New → New Module... → Import .JAR/.AAR Package":

Step 3. Open src/banuba_sdk/build.gradle file. Replace API project banuba_effect_player with the name of the module you've imported in the previous step. By default, it's an API project named banuba_effect_player-release.

Step 4. Resync the Gradle project.

Step 5. Select Build Variant from the left side menu in Android Studio.

(a) Debug build variant allows to properly debug and profile app during its execution;

(b) Release build variant, which enables testing the application's released version to ensure faster performance.

Step 6. Select the build target at the top:

(i) app target, which is used to build the default Banuba SDK Demo app demonstrating major features of Banuba's Augmented Reality SDK;

(ii) beautification target, which is used to build the face beautification example app with the neural networks colouring feature.

How To Combine Several AR Effects Into One

Our Banuba Filter Editor allows users to build simple effects with all scripting and configuration created automatically. However, with more complex effects, you need to manually modify the logic and code implemented into the effect. Before working with effects, please, refer to the Effect Construction section of Banuba's website for a general introduction of Banuba API.

In this section, you will see an example of how to combine multiple Face AR filters into one. In our case, we will merge two effects:

1. Beauty_1
2. Glasses

Merging them will allow your Android app to have two different types of Face AR effects which can be combined into a single one:

1. Beautification and Background separation effects
2. Virtual glasses try-on effects.

The steps to merge these effects are:

Step 1. Choose the ‘base’ effect. The remaining effects will be referred to as ‘additive’.

We choose the Beauty_1 effect as a base as it has more lines in its config.js file. It will be easier to copy a few lines from Glasses then the opposite.

Step 2. Copy all assets (except config.js, config.json, cfg.toml, preview.png - we will deal with them later) from the Glasses effect root folder to Beauty_1. Make sure that your base effect is still working.

NOTE: If some duplicate files are present, choose 'Keep both' in order not to break your base effect and use those duplicated effect files later if needed, or delete them if they aren’t.

Step 3. Open the additive effect (in our case Glasses) config.js. We need to copy the additive effect(s) code over to the base one. The entry point of every effect is the this.init(){...} method defined in the Effect() function.

this.init = function() {

// Spawning models:
        Api.meshfxMsg("spawn", 1, 0, "!glfx_FACE");

        Api.meshfxMsg("spawn", 0, 0, "glasses.bsm2");

// Animation Setup:
        for(var i = 0; i < self.meshes.length; i++) {
            self.meshes[i].animIdx = -1;
            self.meshes[i].endTime = 0;
        }

        self.faceActions = [self.play];

// Show record button
        Api.showRecordButton();
    };

As you can see, in the Glasses effect we have two spawn function calls:

• Api.meshfxMsg("spawn", 1, 0, "!glfx_FACE"): it spawns a recognized face model, which is used by the retouch material.
• Api.meshfxMsg("spawn", 0, 0, "glasses.bsm2"): it spawns a glasses.bsm2 model, which activates its materials ('glasses', 'glass') in the rendering pipeline in cfg.toml.

The call to Api.showRecordButton() will show the camera's default record button. This call is included in Beauty_1 init() method, so you can just skip it.

In other words, we only need to copy over any spawn functions.

Step 4. After copying the code from the “additive” effect(s), we need to modify it, if required, before merging it with the base effect’s code:

a) Make sure all models are spawned with a unique mesh ID Api.meshfxMsg("spawn", mesh id, 0, "modelToSpawn").

b) Check that base function works. If you copied some function declarations (especially trigger function, etc.), check that they are merged appropriately without duplicate and don’t conflict with base effect functions.

Step 5. Merge both effect’s cfg.toml files:

a) Copy all materials and place them into base effect cfg.toml.

In Glasses effect we have 3 materials: retouch, glasses, glass. As we already have retouch in Beauty_1 we don’t need to copy the relevant data (neither will we use its shaders) from the Glasses effect.

# PBR
[materials.glasses]
vs = "glasses.vert"
fs = "glasses.frag"
blend = "off" # "alpha", "premul_alpha", "screen", "add", "multiply", "coverage", "off"
backfaces = false
colorwrite = true
zwrite = true
shadow = false
[materials.glasses.samplers]
tex_diffuse="glasses_BaseColor.png"
tex_normal="glasses_Normal.png"
tex_metallic="glasses_Metallic.png"
tex_roughness="glasses_Roughness.png"
tex_brdf="brdf.ktx"
tex_ibl_diff="ibl_diff.ktx"
tex_ibl_spec="ibl_spec.ktx"

# PBR
[materials.glass]
vs = "glass.vert"
fs = "glass.frag"
blend = "alpha" # "alpha", "premul_alpha", "screen", "add", "multiply", "coverage", "off"
backfaces = false
colorwrite = true
zwrite = true
shadow = false
[materials.glass.samplers]
tex_diffuse="glasses_BaseColor.png"
tex_normal="glasses_Normal.png"
tex_metallic="glasses_Metallic.png"
tex_roughness="glasses_Roughness.png"
tex_brdf="brdf.ktx"

b) Add new material names to draw_order.

By setting draw_order you organize your rendering pipeline. Placing material in the wrong place can make it invisible, e.g., covered by other geometry, etc.

Step 6. Merge config.json files.

In our case, they are the same, but we have many layer settings available (see the docs link above), that we may need for the mixed effect.

Create Face AR Content And Use Banuba Asset Store

By leveraging Banuba's Filter Editor, 3D artists can create their own Face AR effects without programming.

The Filter Editor allows users to import effect assets and configure their properties. Design effects can be created in any 3D graphical tool able to export to the Autodesk .fbx format. The Filter Editor can convert files into supported formats to let developers configure and run visual effects created by 3D artists.

To learn more about the Filter Editor, visit the documentation section of Banuba's website.

On the contrary, to speed up the release of a new Face AR application and compensate for the absence of in-house 3D artists, a company can licence ready-made artwork from Banuba Asset Store. The latter contains more than 600 different masks ready to be used in your next AR app.

A screenshot illustrating different Face AR masks which can be purchased from Banuba Asset Store.

Some of the benefits of Banuba's AR masks are their high quality, the fact that they have been tested on real users, fast selection and integration into your app, regular updates, effort and time savings, reasonable file size, and optimized performance.

Takeaways

1. Banuba Augmented Reality SDK is a tool for creating mobile or desktop applications enhanced with face tracking functionality. It can be used for building video editing and conferencing, social media and beauty editor applications as well as face filters and virtual try-ons.
2. Building a simple Face AR application on Android will require developers to first create a new AR project with Android Studio, Gradle plugin, Android NDK and client's token being the requirements. The latter can be obtained by purchasing the Augmented Reality SDK.
3. Banuba Catalogue is a collection of different 600+ ready-made Face AR masks which can be added to your existing or future face-tracking applications. Purchasing them gives developers a possibility to employ quality AR content for their apps tested on real users, easy selection and integration, regular updates, effort and time-saving, as well as optimized performance and size.

About the author

Ihar Tumašyk is a C++ developer and an SDK team lead working for Banuba. His key responsibilities include the development of the company's main product, Banuba's Face AR SDK and the coordination of his team members with other departments, especially R&D.