编程知识 cdmana.com

Application of kotlin, rxjava and traditional machine learning in mobile phone quality inspection

One . Business background

Privacy clearance is an important part of mobile phone quality inspection , After the automatic quality inspection of our recycled mobile phones is completed , Will clean up the privacy of the mobile phone .

Before privacy clearance , You need to make sure that the mobile phone exits the account of the cloud service . for example iPhone The phone needs to exit iCloud , Huawei 、 Xiaomi and other mobile phones have to quit the corresponding cloud services . Otherwise, it will cause the risk of privacy data leakage , It will also make the subsequent users who purchase this mobile phone unable to enjoy the function of cloud service .

therefore , Account detection is a very important function . This section is to Android For example, check whether the mobile phone account exits , Mainly for Huawei 、 Xiaomi and other mobile phones with obvious characteristics , Through image preprocessing 、OCR For identification .

Our privacy clearance tool is a desktop app , Running on the Ubuntu On the system .

 Privacy clearance 1.jpg

about Android mobile phone , Desktop tools through adb Order privacy to be cleared App Install on your phone , And then the two go through WebSocket communicate , Privacy clearance for mobile phones .

 Privacy clearance 2.jpg

Two . Design thinking

Before doing the account checking function , I've tried a lot of ways to determine whether the account exits , For example, find related adb command , Or the corresponding manufacturer's API, It didn't work very well . After constant exploration , In the following way :

  • Use adb Command to modify the sleep time of the phone , Make sure the phone doesn't go off for a while .
  • Use adb The command jumps to the system settings page ( Different phones use slightly different commands )
  • Use adb Command to take a screenshot of the current page
  • Use adb Command to transfer images to the desktop machine
  • Cut the original image through the program , Keep the original 40%
  • Binarization of the cut image ( Different mobile phones use different binarization algorithms )
  • call OCR Recognition of character string .
  • Compare string similarity , Finally determine whether the account exits .

 Account detection .png

This way in Huawei 、 Xiaomi's mobile phone has achieved good results .

3、 ... and . Code implementation and the holes that have been stepped on

Core code

Core code usage RxJava Connect all the above processes in series , Every process is a map operation , The following shows how to check whether the account number of Huawei mobile phone exits :

object HuaweiDetect : IDetect {

    val logger: Logger = LoggerFactory.getLogger(this.javaClass)

    val list by lazy {
        arrayOf(" Huawei account number 、 Cloud space 、 Application market, etc "
            ," Huawei account number 、 Payments and bills 、 Cloud space, etc "
            ," Huawei account number 、 Cloud space "
            ," Huawei account number 、 Payments and bills ")
    }

    override fun detectUserAccount(serialNumber:String,detail:String): Observable<Boolean> {

        val file = File(detectAccountPath)
        if (!file.exists()) {
            file.mkdir()
        }

        val timeoutCmd = CommandBuilder.buildSudoCommand("aihuishou","$adbLocation -s $serialNumber shell settings put system screen_off_timeout 600000")
        CommandExecutor.executeSync(timeoutCmd,appender = object : Appender {
            override fun appendErrText(text: String) {
                println(text)
            }

            override fun appendStdText(text: String) {
                println(text)
            }

        }).getExecutionResult()

        val cmd = CommandBuilder.buildSudoCommand("aihuishou","$adbLocation -s $serialNumber shell am start -S com.android.settings/.HWSettings")

        val fileName = "${serialNumber}-${detail}.png"

        return CommandExecutor.execute(cmd)
                .asObservable()
                .delay(2, TimeUnit.SECONDS)
                .map {
                    val screencapCmd = CommandBuilder.buildSudoCommand("aihuishou","$adbLocation -s $serialNumber shell screencap -p /sdcard/$fileName")
                    CommandExecutor.executeSync(screencapCmd, appender = object : Appender {
                        override fun appendErrText(text: String) {
                            println(text)
                        }

                        override fun appendStdText(text: String) {
                            println(text)
                        }

                    }).getExecutionResult()
                }
                .map {
                    val pullCmd = CommandBuilder.buildSudoCommand("aihuishou","$adbLocation -s $serialNumber pull /sdcard/$fileName ${detectAccountPath}/${fileName}")
                    CommandExecutor.executeSync(pullCmd, appender = object : Appender {
                        override fun appendErrText(text: String) {
                            println(text)
                        }

                        override fun appendStdText(text: String) {
                            println(text)
                        }

                    }).getExecutionResult()
                    fileName
                }
                .map {
                    val input = File("$detectAccountPath/$it")
                    val image = ImageIO.read(input)
                    val width = image.width
                    val height = image.height

                    return@map imageCutByRectangle(image, 0, 0, width, (height * 0.4).toInt())
                }
                .map { //  Two valued 
                    binaryForHuawei(it)
                }
                .map{
                    val ocrValue = newTesseract().doOCR(it)
                    logger.info("ocrValue = $ocrValue")
                    ocrValue
                }
                .map { ocrValue->
                    if (ocrValue.contains(" Huawei account number 、 Cloud space 、 Application market, etc ")
                            || ocrValue.contains(" Huawei account number 、 Payments and bills 、 Cloud space, etc ")
                            || ocrValue.contains(" Huawei account number 、 Cloud space ")
                            || ocrValue.contains(" Huawei account number 、 Payments and bills ")) {
                        return@map true
                    } else {
                        val array = ocrValue.split("\n".toRegex())
                        array.map {
                            it.replace("\\s+".toRegex(),"")
                        }.toList().forEach{ s->
                            for (item in list) {
                                val d = levenshtein(s,item) //  String similarity comparison 
                                if (d>=0.7) {
                                    return@map true
                                }
                            }
                        }
                        return@map false
                    }
                }
    }

}

among ,imageCutByRectangle() Used to cut pictures

/**
 *  Rectangular cut , Set the starting position , Cut width , Cut length 
 *  The clipping range should be less than or equal to the image range 
 * @param image
 * @param xCoordinate
 * @param yCoordinate
 * @param xLength
 * @param yLength
 * @return
 */
fun imageCutByRectangle(
    image: BufferedImage,
    xCoordinate: Int,
    yCoordinate: Int,
    xLength: Int,
    yLength: Int
): BufferedImage {
    // Judge x、y Does the direction exceed the maximum range of the image 
    var xLength = xLength
    var yLength = yLength
    if (xCoordinate + xLength >= image.width) {
        xLength = image.width - xCoordinate
    }
    if (yCoordinate + yLength >= image.height) {
        yLength = image.height - yCoordinate
    }
    val resultImage = BufferedImage(xLength, yLength, image.type)
    for (x in 0 until xLength) {
        for (y in 0 until yLength) {
            val rgb = image.getRGB(x + xCoordinate, y + yCoordinate)
            resultImage.setRGB(x, y, rgb)
        }
    }
    return resultImage
}

binaryForHuawei() For image binarization .

Image binarization ( Image Binarization) Is to set the gray value of pixels on the image to 0 or 255, That is to say, the process of presenting the whole image with obvious black-and-white effect .
In digital image processing , Binary image plays a very important role , The binarization of the image greatly reduces the amount of data in the image , So as to highlight the outline of the target .

fun binaryForHuawei(bi: BufferedImage):BufferedImage = binary(bi)

/**
 *  Image binarization operation 
 * @param bi
 * @param thresh  Threshold of binarization 
 * @return
 */
fun binary(bi: BufferedImage,thresh:Int = 225):BufferedImage {

    //  Gets the height of the current picture , wide ,ARGB
    val h = bi.height
    val w = bi.width
    val rgb = bi.getRGB(0, 0)
    val arr = Array(w) { IntArray(h) }

    //  Get the gray value of each pixel in the picture 
    for (i in 0 until w) {
        for (j in 0 until h) {
            // getRGB() Return to the default RGB Color model ( Decimal system )
            arr[i][j] = getImageRgb(bi.getRGB(i, j)) // The gray value of the point 
        }
    }

    val bufferedImage = BufferedImage(w, h, BufferedImage.TYPE_BYTE_BINARY) //   Construct a  BufferedImage,TYPE_BYTE_BINARY( Represents an opaque byte packed  1、2  or  4  Bit image .)

    for (i in 0 until w) {
        for (j in 0 until h) {
            if (getGray(arr, i, j, w, h) > thresh) {
                val white = Color(255, 255, 255).rgb
                bufferedImage.setRGB(i, j, white)
            } else {
                val black = Color(0, 0, 0).rgb
                bufferedImage.setRGB(i, j, black)
            }
        }
    }

    return bufferedImage
}

private fun getImageRgb(i: Int): Int {
    val argb = Integer.toHexString(i) //  Convert decimal color values to hexadecimal 
    // argb Each represents transparency , red , green , blue   Accounting for 16 Base number 2 position 
    val r = argb.substring(2, 4).toInt(16) // The following parameter is the use of base 
    val g = argb.substring(4, 6).toInt(16)
    val b = argb.substring(6, 8).toInt(16)
    return ((r + g + b) / 3)
}

// Add yourself around 8 And then divide it by 9, Calculate the relative gray value 
private fun getGray(gray: Array<IntArray>, x: Int, y: Int, w: Int, h: Int): Int {
    val rs = (gray[x][y]
            + (if (x == 0) 255 else gray[x - 1][y])
            + (if (x == 0 || y == 0) 255 else gray[x - 1][y - 1])
            + (if (x == 0 || y == h - 1) 255 else gray[x - 1][y + 1])
            + (if (y == 0) 255 else gray[x][y - 1])
            + (if (y == h - 1) 255 else gray[x][y + 1])
            + (if (x == w - 1) 255 else gray[x + 1][y])
            + (if (x == w - 1 || y == 0) 255 else gray[x + 1][y - 1])
            + if (x == w - 1 || y == h - 1) 255 else gray[x + 1][y + 1])
    return rs / 9
}

For different phones , When dealing with binarization, you need to use different thresholds , What's more, different binarization algorithms are used .

The following figures show the use of adb The command cuts through the system settings page , And the cut and binarized image .

HUAWEI-ELE-AL00.png

HUAWEI-ELE-AL00-debug.png

newTesseract().doOCR(it) It's using Tesseract To call the binary image OCR Algorithm for text content recognition .

fun newTesseract():Tesseract = Tesseract().apply {

    val path = SystemConfig.TESS_DATA
    this.setDatapath(path)
    this.setLanguage("eng+chi_sim")
    this.setOcrEngineMode(0)
}

Our model here is in English , At present, only Chinese and English contents can be recognized .

The identified content may be different from what we expected , Final adoption Levenshtein As a comparison of string similarity . To a certain value , We'll think it's in line with expectations .

Levenshtein distance , Also called editing distance , Between two strings , The minimum number of editing operations required to convert from one to another . Permitted editing operations include replacing one character with another , Insert a character , Delete a character .

The pit of tread

  • Tesseract Cannot be used in multithreading . Later, we used object pools , But it still doesn't work . Only one new instance at a time Tesseract object , So I have to deal with JVM tuning .
  • For different brands of mobile phones , Binarization of images needs to be processed separately .
  • Mobile phones of the same brand , Different models may require different strategies .

Four . Follow up planning

Although the above implementation has met most of the requirements , But only in Chinese and English , And the algorithm model needs to be deployed on the desktop . We've started to implement deep learning algorithms OCR The function of .

In the next stage of work , Deploy algorithms and models in the cloud . On the one hand, it reduces the pressure on the desktop , On the other hand, it can support multiple languages and improve character recognition rate .

Click to see more

版权声明
本文为[osc_ 9xpkfxc9]所创,转载请带上原文链接,感谢
https://cdmana.com/2020/12/20201225112343219k.html

Scroll to Top