Sensifai Mobile Image & Video Enhancement

There are many images in our albums which are far from ideal due to bad lighting conditions, low resolutions of old cameras, or incorrect automatic setting of the camera.

Sensifai Image and Video Enhancement App automatically improve your low-quality images using an advanced artificial intelligence (AI) method based on deep neural networks! This app offers a fast and end2end approach to increase the lighting in dark pictures, improve contrast and brightness, shoot up the resolution, and adjust tones.

The software which is publicly available as an SDK as well as an Android App works in three modes: automatic enhancement, superresolution, and manual improvement. One can use one of these modes to enhance a specific picture according to a required correction and modification.

• In automatic enhancement mode, the software applies a deep-learning model to automatically improve your image quality. For example, it increases the lighting of the images and improves the brightness without any effort from the users. Then it shows both images before and after the enhancement and lets the user compare them and save the enhanced image or remove it.

• In superresolution mode, the program helps to increase the resolution of the images automatically using a deep-neural network system.

• In manual improvement mode, the software provides a set of tools such that the users apply different filters manually to re-color or adjust the brightness of the picture. This mode is also accessible after an image was treated in superresolution or automatic enhancement modes.

There are different image enhancement Apps (Letsenhance.io and Google DeepAngel) that improve the quality of images or edit them automatically through advanced artificial intelligence. These apps require users to send their images to their cloud to process them. This increases the risk of getting hacked, scandalized, or abused and potentially violates the privacy of millions of users. This is all because these apps work based on deep-learning that is computationally heavy and requires strong GPU servers.

SensifAI offers a game-changing technology that solves this problem. We have developed specific deep learning architectures for the smartphone chipsets of most major smartphone manufacturers. With this technology, we can enhance users’ images and videos locally on their smartphones without any connection to the internet. This is an on-device, smart-enhance App that can help millions of people enhance their video/image archives while guaranteeing control over their personal data. Many thanks to the Next Generation Internet (NGI) initiative of the European Commission and NGI Trust consortium partners for their support and help during the development of Sensifai image and video enhancement app.

Enhancement Engine

Earlier methods are mostly based on histogram equalization and gamma correction. The performance of these methods is limited by the fact that individual pixels are enhanced without consideration to contextual information. Many enhancement algorithms based on the Retinex theory [1] have been suggested in the literature [2,3]. More advanced techniques estimate and normalize illumination to get the enhanced image [4,5,6]. Recently, deep convolutional neural networks (DCNNs) have gained great success in image enhancement [7,8]. Ignatov et al. [9] propose DPED to produce DLSR- from mobile quality images by using deep residual networks trained with a composite loss function of content, color, and texture. Also, a Fast and Efficient image Quality Enhancement as FEQE for image super-resolutin and enhancement on mobile devices has been introduced in [10]. In this project, we used residual blocks with some modifications for the model. In our model, we applied Spacial Channel Attention (SCA) Technique to get more discriminative features. Channel attention technique by the implementation of Squeeze and Excitation (SE) Block extracts more discriminative features by weighting each of the channels adaptively. Additionally, we have utilized Spacial Attention as a feature guide block which exploits the nature of low-level features to obtain more discriminative features. Because Low-level features have spatial and local information, therefore , feature guide block can help to better locate the corners and edges of objects. In our model, we have utilized desubpixel and subpixel layers in down-sampling and up-sampling respectively. Fig1. illustrates the Res Block with some modifications.

a new penalty term to multiply in MSE loss function. Since the luminance and contrast are not considered directly in mean square error (MSE) loss function and due to the importance of them, we used the luminance and contrast as a penalty term. This penalty term consists of difference between mean and variance of the input image and it’s ground truth.

Loss function = Penalty_Term * MSE

MSE=

Penalty_Term= [1+(-log(1-(difference/2)))], 0 < difference/2 < +1

difference=

Finally, SENSIFAI_512 dataset have been created by SENSIFAI company. Since the size of downloaded images was very larg, we decided to resize images to 512x512. The total number of images in this dataset is 15000 which we have selected 20% as a validation part. Input images in this dataset have been created by Rawtherapee app.

Android SDK

This repository contains an Android SDK for Sensifai Image & Video Enhancement. This SDK brings the ability to any Android mobile application which needs to enhance the quality of images or videos. The AI engine of this SDK enhances the color range and resolution of input image or video in real-time.

Build

Clone this repository and import into Android Studio From the gradle menu choose assembleRelease/assembleDebug Option to build the AAR file (in /enhancement/build/outputs/aar)

Usage

Create a new Android Application Project and import the AAR file as a dependency add the following dependencies to build.gradle file:
TFLite

    implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly'
    implementation 'org.tensorflow:tensorflow-lite-gpu:0.0.0-nightly'
    implementation 'org.tensorflow:tensorflow-lite-support:0.0.0-nightly'
    implementation 'org.tensorflow:tensorflow-lite-hexagon:0.0.0-nightly'

SNPE (import snpe-release.aar file as module dependency into the project)

    implementation project(':snpe-release')

Code Implementation

Init
Before You can use the model it has to be initialized first. The initialization process needs these parameters:

• Context: Activity or Application context
  
• ModelName: Name of the model that you want to use
  
• Device: A specific device (The alternatives for device are CPU, GPU, DSP and NNAPI).
  
• NumOfThreads: The Number of threads to use for process
  

TFLite

    import com.sensifai.enhancement.TFLite.Enhancement;
    ...
    public class MainActivity extends AppCompatActivity{
        Enhancement enhancement;
        ...
        @Override public void onCreate(Bundle savedInstanceState){
            enhancement = new Enhancement({isDynamicInput});//Whether the model size can change or not
            enhancement.init(this, "{Enhancement_TFLite_Model_Name}", Device.GPU, 1);
        }
    }

SNPE

    import com.sensifai.ehancement.SNPE.Enhancement;
    ...
    public class MainActivity extends AppCompatActivity{
        Enhancement enhancement;
        ...
        @Override public void onCreate(Bundle savedInstanceState){
            enhancement = new Enhancement();
            boolean enhancementInited = enhancement.init(this, "{Enhancement_Model_Name}", Device.GPU, 1);
        }
    }

Model Run
After the model is initialized you can pass a list of bitmaps to be processed. If your model does not support dynamic input you have to resize these bitmaps to the supported dimensions before passing them.

TFLite

    ProcessResult<EnhancementResult> result = enhancement.process(new Bitmap[]{YOUR_BITMAP_HERE});

SNPE
SNPE model can work with specific image sizes therefore we added sizeOperation parameter to the Process method, you have to pass one of the below items:

• 0: Without changes
  
• 1: if (IMAGE_WIDTH = INPUT_HEIGHT and IMAGE_HEIGHT = INPUT_WIDTH)
  
• 2: if ((IMAGE_WIDTH / INPUT_WIDTH) = (IMAGE_HEIGHT / INPUT_HEIGHT))
  
• 3: If need to both rotate and grow (based on the above descriptions)
  

    int sizeOperation = 0;  
    ProcessResult<EnhancementResult> result = enhancement.process(new Bitmap[]{YOUR_BITMAPS_HERE},sizeOperation);

Resluts
The procoess results is returned in a ProcessResult object of generic type EnhancementResult. The EnhancementResult contains an array of bitmaps each representing an Enhancement process (in our models it has only one bitmap). The ProcessResult consists of a list of generic type E (EnhancementResult in this case) which has an entry for each input bitmap of the process (1 here). Alongside this list there are also the inferenceTime and totalTime of the process for performance information.

    if (result == null) {
                Log.e(TAG, "Processing returned with error!");
                return;
            }
            if (result.getResults() == null || result.getResults().size() == 0) {
                Log.e(TAG, "Image Size Not Supported!");
                return;
            }

            if (saveBitmap(getApplicationContext(), result.getResults().get(0).getEnhancedImages()[0], processedTempFile)) {
                    Log.e(TAG, "Processed Image Saved Successfully");
            }
            Log.e(TAG, String.format(Locale.US, "Total time:  %d ms\nInference time:  %d ms",
                    result.getTotalTime(), result.getInferenceTime()));

Release
Since the model initiation process happens on the native side it will not be garbage collected, therefore, you need to release the model after you are done with it.

TFLite

    enhancement.release();

SNPE

    enhancement.release();

Using Your Own Models

You can extend the processor class of either TFLite or SNPE packages to use your own models. To create such custom class you will need to pass a list of ModelInfo supported by that custom class.

TFLite's Model info need PreProcess and PostProcess to be defined according to the corresponding Interfaces already defined. There are already some predefined PreProcess and PostProcess classes compatible with some of popular models.

SNPE's Model info get the preProcess info of the specified model.

The defined Enhancement class (extended from processor) in each package (TFLite and SNPE) can be used for most cases of Automatic Image Processing. This class uses the predefined result classes as generic types to format the processed outcome. For each bitmap given to the model as input an EnhancementResult is added to the results list in the output ProcessResult object.

To further customize the usage you can use your own format for the result retrieval.

Coding Style

This Section is Useful for Contributors and Developers Who Need to Customize Some Functionality

References

1. Land, E.H.: The retinex theory of color vision. Scientific american (1977)
2. Bertalmı́o, M., Caselles, V., Provenzi, E., Rizzi, A.: Perceptual color correction through variational techniques. TIP (2007)
3. Palma-Amestoy, R., Provenzi, E., Bertalmı́o, M., Caselles, V.: A perceptually inspired variational framework for color enhancement. TPAMI (2009)
4. Zhang, S., Tang, G.j., Liu, X.h., Luo, S.h., Wang, D.d.: Retinex based low-light image enhancement using guided filtering and variational framework. Optoelectronics Letters 14(2) (2018) 156–160
5. Fu, X., Sun, Y., LiWang, M., Huang, Y., Zhang, X.P., Ding, X.: A novel retinex based approach for image enhancement with illumination adjustment. In: Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, IEEE (2014) 1190–1194
6. Li, D., Zhang, Y., Wen, P., Bai, L.: A retinex algorithm for image enhancement based on recursive bilateral filtering. In: Computational Intelligence and Security (CIS), 2015 11th International Conference on, IEEE (2015) 154–157
7. Shen, L., Yue, Z., Feng, F., Chen, Q., Liu, S., Ma, J.: Msr-net: Low-light image enhancement using deep convolutional network. arXiv preprint arXiv:1711.02488 (2017)
8. Tao, F., Yang, X., Wu, W., Liu, K., Zhou, Z., Liu, Y.: Retinex-based image enhancement framework by using region covariance filter. Soft Computing 22(5) (2018) 1399–1420
9. Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Van Gool, L.: Dslr-quality photos on mobile devices with deep convolutional networks. In: ICCV. (2017)
10. Vu, Thang, Cao Van Nguyen, Trung X. Pham, Tung M. Luu, and Chang D. Yoo. "Fast and efficient image quality enhancement via desubpixel convolutional neural networks." In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0. 2018.