fix: Fix discrepancy between barcode bounding box and barcode capture size #1169

navaronbracke · 2024-09-04T11:14:40Z

This PR fixes a discrepancy between what is meant with the bounding box of a scanned barcode and the size of the barcode capture output image.

It also adds the following changes:

Added support for returnImage on MacOS
Fixes a bug where returnImage set to false would not do anything on iOS
Cleans up some unused code and refactors some control flow to use early returns

Fixes #1162

cc @fulstadev

…g box for Android

…rcode in each result; update MacOS to use the new format for image data

navaronbracke · 2024-09-04T11:15:17Z

CHANGELOG.md

+## 5.2.2
+
+Improvements:
+* [MacOS] Adds Swift Package Manager support.


This was added by a previous PR, but wasn't yet in the changelog.

navaronbracke · 2024-09-04T11:16:10Z

android/src/main/kotlin/dev/steenbakker/mobile_scanner/MobileScannerHandler.kt

                "name" to "barcode",
                "data" to barcodes,
-                "image" to image,
-                "width" to width!!.toDouble(),


Since the width/height is only for the output image data, I put them in a Map for the image data.

I also added safe access to the width / height. The method channel implementation handles null width/height by falling back to Size.zero

navaronbracke · 2024-09-04T11:17:21Z

android/src/main/kotlin/dev/steenbakker/mobile_scanner/MobileScannerUtilities.kt

-        "driverLicense" to driverLicense?.data, "email" to email?.data,
-        "geoPoint" to geoPoint?.data, "phone" to phone?.data, "sms" to sms?.data,
-        "url" to url?.data, "wifi" to wifi?.data, "displayValue" to displayValue
+        "calendarEvent" to calendarEvent?.data,


I sorted these properties alphabetically, to make it easier to follow. The line "size" to boundingBox?.size, is new.

navaronbracke · 2024-09-04T11:18:17Z

android/src/main/kotlin/dev/steenbakker/mobile_scanner/MobileScannerUtilities.kt

+
+private val Rect.size: Map<String, Any?>
+    get() {
+        // Rect.isValid can't be accessed for some reason, so just do the check manually.


When inspecting the source code of android.graphics.Rect, there was an isValid() method, but I could not access it. So I just took the check from there.

navaronbracke · 2024-09-04T11:18:44Z

ios/Classes/MobileScanner.swift

    var scanner = BarcodeScanner.barcodeScanner()

-    /// Return image buffer with the Barcode event
-    var returnImage: Bool = false


This was moved

navaronbracke · 2024-09-04T11:18:56Z

ios/Classes/MobileScanner.swift

    /// Gets called when a new image is added to the buffer
    public func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
-            print("Failed to get image buffer from sample buffer.")


This print doesn't add any value

navaronbracke · 2024-09-04T11:19:11Z

ios/Classes/MobileScanner.swift

                    if (error == nil && barcodesString != nil && newScannedBarcodes != nil && barcodesString!.elementsEqual(newScannedBarcodes!)) {
                        return
-                    } else if (newScannedBarcodes?.isEmpty == false) {
+                    }


Early return instead of else

navaronbracke · 2024-09-04T11:19:42Z

ios/Classes/MobileScanner.swift


    var barcodesString: Array<String?>?

-    //    /// Convert image buffer to jpeg


navaronbracke · 2024-09-04T11:20:00Z

ios/Classes/MobileScanner.swift

    }

    /// Start scanning for barcodes
-    func start(barcodeScannerOptions: BarcodeScannerOptions?, returnImage: Bool, cameraPosition: AVCaptureDevice.Position, torch: Bool, detectionSpeed: DetectionSpeed, completion: @escaping (MobileScannerStartParameters) -> ()) throws {


This parameter is no longer needed

navaronbracke · 2024-09-04T11:21:48Z

ios/Classes/MobileScannerPlugin.swift

-                    } else {
-                        return barcode.data
-                    }
+            if error != nil {


Rewriting to use early returns

navaronbracke · 2024-09-04T11:22:31Z

ios/Classes/MobileScannerPlugin.swift

+                return
+            }
+
+            if (!MobileScannerPlugin.returnImage) {


This is new. Previously we would always send the image data, which is incorrect.

navaronbracke · 2024-09-04T11:23:00Z

ios/Classes/MobileScannerPlugin.swift

-                DispatchQueue.main.async {
-                    result(["name": "barcode", "data": barcodesMap])
-                }
+                return


Early return

navaronbracke · 2024-09-04T11:23:44Z

ios/Classes/MobileScannerUtilities.swift

-        let corners = cornerPoints?.map({$0.cgPointValue.data})
-        return ["corners": corners, "format": format.rawValue, "rawBytes": rawData, "rawValue": rawValue, "type": valueType.rawValue, "calendarEvent": calendarEvent?.data, "contactInfo": contactInfo?.data, "driverLicense": driverLicense?.data, "email": email?.data, "geoPoint": geoPoint?.data, "phone": phone?.data, "sms": sms?.data, "url": url?.data, "wifi": wifi?.data, "displayValue": displayValue]
+        return [
+            "calendarEvent": calendarEvent?.data,


Like on Android, I sorted these on alphabetical keys, and added the size.

navaronbracke · 2024-09-04T11:25:02Z

lib/src/method_channel/mobile_scanner_method_channel.dart

    final List<Map<Object?, Object?>> barcodes =
        data.cast<Map<Object?, Object?>>();

-    if (defaultTargetPlatform == TargetPlatform.macOS) {


I updated the MacOS implementation to use the same keys / datatypes as iOS/Android. So this can be removed.

navaronbracke · 2024-09-04T11:25:30Z

lib/src/method_channel/mobile_scanner_method_channel.dart

-      final double? width = event['width'] as double?;
-      final double? height = event['height'] as double?;
+        defaultTargetPlatform == TargetPlatform.iOS ||
+        defaultTargetPlatform == TargetPlatform.macOS) {


MacOS is added here, as the format has been updated to match the other platforms.

navaronbracke · 2024-09-04T11:26:41Z

lib/src/method_channel/mobile_scanner_method_channel.dart


      return BarcodeCapture(
-        raw: data,
+        raw: event,


MacOS used the event Map as raw data. This wasn't true for iOS/Android, so I updated it to match MacOS.

Unlikely that a lot of people need the raw data anyway.

navaronbracke · 2024-09-04T11:27:26Z

lib/src/objects/barcode.dart

  factory Barcode.fromNative(Map<Object?, Object?> data) {
    final Map<Object?, Object?>? calendarEvent =
        data['calendarEvent'] as Map<Object?, Object?>?;
-    final List<Object?>? corners = data['corners'] as List<Object?>?;


Moved down, to be sorted on alphabetical names

navaronbracke · 2024-09-04T11:28:03Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift


    var symbologies:[VNBarcodeSymbology] = []

-    //    var analyzeMode: Int = 0


This was unused

navaronbracke · 2024-09-04T11:28:15Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

            setScale(call, result)
        case "resetScale":
            resetScale(call, result)
-            //        case "analyze":


navaronbracke · 2024-09-04T11:28:48Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

            return
        }
        guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
-            print("Failed to get image buffer from sample buffer.")


Again, a print that didn't add any value.

navaronbracke · 2024-09-04T11:29:07Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

-                            }
-                        } else {
+
+                        if error != nil {


Refactored to use early returns.

navaronbracke · 2024-09-04T11:30:19Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

+                            return
+                        }
+
+                        let barcodes: [VNBarcodeObservation] = results.compactMap({ barcode in


There was actually a bug here, too. MacOS would call DispatchQueue.main.async {} for every single barcode, instead of once, for all of them together. This has been corrected, by collecting the barcodes - that match the scan window - and passing them as a list.

navaronbracke · 2024-09-04T11:31:24Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

+                        })
+
+                        DispatchQueue.main.async {
+                            if (!MobileScannerPlugin.returnImage) {


This is new, and similarly to iOS, we only pass the image data if requested.

navaronbracke · 2024-09-04T11:31:43Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

        result(nil)
    }

-    //    func switchAnalyzeMode(_ call: FlutterMethodCall, _ result: @escaping FlutterResult) {


navaronbracke · 2024-09-04T11:32:08Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

+}
+
+extension VNBarcodeObservation {
+    public func toMap() -> [String: Any?] {


This is similar to the extension in the iOS implementation

navaronbracke · 2024-09-04T11:32:36Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

 }

+extension CGImage {
+    public func jpegData(compressionQuality: CGFloat) -> Data? {


I needed this extension to get the (compressed) JPEG data from the input image.

The signature matches what the iOS implementation uses.

navaronbracke · 2024-09-04T11:34:52Z

lib/src/objects/barcode_capture.dart

  /// The list of scanned barcodes.
  final List<Barcode> barcodes;

-  /// The bytes of the image that is embedded in the barcode.


Clarified a bunch of docs, that were incorrect.

navaronbracke · 2024-09-04T11:35:23Z

lib/src/objects/barcode_capture.dart

@@ -1,3 +1,6 @@
+/// @docImport 'package:mobile_scanner/src/mobile_scanner_controller.dart';


New doc import annotation, for line 27, which has a doc reference to a different class.

navaronbracke · 2024-09-05T11:57:17Z

macos/mobile_scanner/Sources/mobile_scanner/MobileScannerPlugin.swift

+                                "data": barcodes.map({ $0.toMap() }),
+                                "image": cgImage == nil ? nil : [
+                                    "bytes": FlutterStandardTypedData(bytes: cgImage!.jpegData(compressionQuality: 0.8)!),
+                                    "width": Double(cgImage!.width),


The image on iOS uses doubles for width/height. On Android it uses integers, but we did convert them into doubles.

On MacOS, CGImage also uses integers, so I added a double conversion here.

navaronbracke · 2024-09-05T11:58:39Z

example/ios/Runner/AppDelegate.swift

 import UIKit
 import Flutter

-@UIApplicationMain


UIApplicationMain is deprecated. This was updated by the Flutter tool project migrators.

navaronbracke added 15 commits September 4, 2024 13:09

sort Barcode.data keys alphabetically on Android; add missing boundin…

c179310

…g box for Android

fix discrepancy for barcode capture image size on Android

c83dd61

fix docs for barcode capture

28be45d

add size to barcode

0856a59

update the image parsing from the method channel

fa763a2

adjust raw value on Android & iOS

48d83fe

clean up some redundant statements in ios/MobileScanner.swift

fa23d7d

fix early returns in iOS implementation of mobile scanner

f772f29

fix return image being ignored on iOS

2e9e246

sort barcode map keys for iOS; add barcode size on iOS

e503f0a

remove unused code in MacOS impl

2f7de30

add jpeg data extension & toMap extension

02e0102

set return image flag on macos

cc23a80

use early returns on MacOS; fix MacOS from only returning a single ba…

cfc3643

…rcode in each result; update MacOS to use the new format for image data

let MacOS also use the same barcode format parsing as iOS/Android

9209f59

navaronbracke requested a review from juliansteenbakker as a code owner September 4, 2024 11:14

github-actions bot assigned navaronbracke Sep 4, 2024

navaronbracke commented Sep 4, 2024

View reviewed changes

ios/Classes/MobileScanner.swift

var barcodesString: Array<String?>?

// /// Convert image buffer to jpeg

Copy link

Collaborator Author

navaronbracke Sep 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused

navaronbracke commented Sep 4, 2024

View reviewed changes

update changelog & version

3bb18bd

navaronbracke force-pushed the size_fix_android branch from 16bac88 to 3bb18bd Compare September 4, 2024 11:21

navaronbracke commented Sep 4, 2024

View reviewed changes

navaronbracke added 4 commits September 5, 2024 12:48

update MacOS example project

1d1b0f9

fix type error

3450814

fix double values for MacOS

6dea0d3

update ios example app

75418f2

navaronbracke commented Sep 5, 2024

View reviewed changes

navaronbracke merged commit c4bf426 into juliansteenbakker:master Sep 5, 2024
5 checks passed

navaronbracke deleted the size_fix_android branch September 5, 2024 11:59


		var barcodesString: Array<String?>?

		// /// Convert image buffer to jpeg


		var symbologies:[VNBarcodeSymbology] = []

		// var analyzeMode: Int = 0

		@@ -1,3 +1,6 @@
		/// @docImport 'package:mobile_scanner/src/mobile_scanner_controller.dart';

Uh oh!

fix: Fix discrepancy between barcode bounding box and barcode capture size #1169

fix: Fix discrepancy between barcode bounding box and barcode capture size #1169

Uh oh!

Conversation

navaronbracke commented Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navaronbracke Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navaronbracke Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navaronbracke Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

navaronbracke commented Sep 4, 2024 •

edited

Loading

navaronbracke Sep 4, 2024 •

edited

Loading

navaronbracke Sep 4, 2024 •

edited

Loading

navaronbracke Sep 4, 2024 •

edited

Loading