|
| 1 | +# Real-time detection |
| 2 | + |
| 3 | +In this example, I show how to use the Fast OpenCV library together with the Vision Camera library's frame processor. Our goal will be to detect a coloured object and mark it on the screen. |
| 4 | + |
| 5 | + |
| 6 | + |
| 7 | +### Requirements |
| 8 | + |
| 9 | +- We must have the react-native-fast-opencv library installed. |
| 10 | +- VisionCamera together with WorkletsCore to handle the frame processors must be installed. Detailed instructions are available [here](https://react-native-vision-camera.com/docs/guides/frame-processors). |
| 11 | +- Installed [vision-camera-resize-plugin](https://github.com/mrousavy/vision-camera-resize-plugin) library to perform efficient frame scaling. |
| 12 | +- Installed [react-native-skia](https://github.com/Shopify/react-native-skia) for drawing elements on frames. |
| 13 | + |
| 14 | +### Code |
| 15 | + |
| 16 | +We will start by constructing the base. Our component will have a VisionCamera component used and a frame processor constructed using Skia. |
| 17 | + |
| 18 | +```js |
| 19 | +import { PaintStyle, Skia } from '@shopify/react-native-skia'; |
| 20 | +import { useEffect } from 'react'; |
| 21 | +import { StyleSheet, Text } from 'react-native'; |
| 22 | +import { |
| 23 | + Camera, |
| 24 | + useCameraDevice, |
| 25 | + useCameraPermission, |
| 26 | + useSkiaFrameProcessor, |
| 27 | +} from 'react-native-vision-camera'; |
| 28 | +import { useResizePlugin } from 'vision-camera-resize-plugin'; |
| 29 | + |
| 30 | +const paint = Skia.Paint(); |
| 31 | +paint.setStyle(PaintStyle.Fill); |
| 32 | +paint.setColor(Skia.Color('lime')); |
| 33 | + |
| 34 | +export function VisionCameraExample() { |
| 35 | + const device = useCameraDevice('back'); |
| 36 | + const { hasPermission, requestPermission } = useCameraPermission(); |
| 37 | + |
| 38 | + const { resize } = useResizePlugin(); |
| 39 | + |
| 40 | + useEffect(() => { |
| 41 | + requestPermission(); |
| 42 | + }, [requestPermission]); |
| 43 | + |
| 44 | + const frameProcessor = useSkiaFrameProcessor((frame) => { |
| 45 | + 'worklet'; |
| 46 | + |
| 47 | + const height = frame.height / 4; |
| 48 | + const width = frame.width / 4; |
| 49 | + |
| 50 | + const resized = resize(frame, { |
| 51 | + scale: { |
| 52 | + width: width, |
| 53 | + height: height, |
| 54 | + }, |
| 55 | + pixelFormat: 'bgr', |
| 56 | + dataType: 'uint8', |
| 57 | + }); |
| 58 | + |
| 59 | + }, []); |
| 60 | + |
| 61 | + if (!hasPermission) { |
| 62 | + return <Text>No permission</Text>; |
| 63 | + } |
| 64 | + |
| 65 | + if (device == null) { |
| 66 | + return <Text>No device</Text>; |
| 67 | + } |
| 68 | + |
| 69 | + return ( |
| 70 | + <Camera |
| 71 | + style={StyleSheet.absoluteFill} |
| 72 | + device={device} |
| 73 | + isActive={true} |
| 74 | + frameProcessor={frameProcessor} |
| 75 | + /> |
| 76 | + ); |
| 77 | +} |
| 78 | +``` |
| 79 | + |
| 80 | +We scale the frame to reduce its size and enable faster processing. In addition, we handle permissions for the Camera. |
| 81 | + |
| 82 | +Now let's focus on the function of our frameProcessor. Our aim will be to detect a bright green object (card) and mark it on the screen in real time. |
| 83 | + |
| 84 | +To do this, let us add a new Mat object. |
| 85 | + |
| 86 | +```js |
| 87 | +const src = OpenCV.frameBufferToMat(height, width, resized); |
| 88 | +``` |
| 89 | + |
| 90 | +And another object that will contain our processed image. |
| 91 | + |
| 92 | +```js |
| 93 | +const dst = OpenCV.createObject(ObjectType.Mat, 0, 0, DataTypes.CV_8U); |
| 94 | +``` |
| 95 | + |
| 96 | +In order to find the object easily, it will be necessary to change the colour to HSV. We can also create objects (Scalar) that will be the beginning of our detected range and its end. The `cvtColor` function changes the colour format, while the `inRange` function leaves only those pixels whose colour fits within the specified range. |
| 97 | + |
| 98 | +```js |
| 99 | +const lowerBound = OpenCV.createObject(ObjectType.Scalar, 30, 60, 60); |
| 100 | +const upperBound = OpenCV.createObject(ObjectType.Scalar, 50, 255, 255); |
| 101 | +OpenCV.invoke('cvtColor', src, dst, ColorConversionCodes.COLOR_BGR2HSV); |
| 102 | +OpenCV.invoke('inRange', dst, lowerBound, upperBound, dst); |
| 103 | +``` |
| 104 | + |
| 105 | +We further split the image into channels and extract the first channel. |
| 106 | + |
| 107 | +```js |
| 108 | +const channels = OpenCV.createObject(ObjectType.MatVector); |
| 109 | +OpenCV.invoke('split', dst, channels); |
| 110 | +const grayChannel = OpenCV.copyObjectFromVector(channels, 0); |
| 111 | +``` |
| 112 | + |
| 113 | +Now we will deal with finding the contours in order to do this we will use the `findContours` function. |
| 114 | + |
| 115 | +```js |
| 116 | +const contours = OpenCV.createObject(ObjectType.MatVector); |
| 117 | +OpenCV.invoke( |
| 118 | + 'findContours', |
| 119 | + grayChannel, |
| 120 | + contours, |
| 121 | + RetrievalModes.RETR_TREE, |
| 122 | + ContourApproximationModes.CHAIN_APPROX_SIMPLE |
| 123 | +); |
| 124 | +``` |
| 125 | + |
| 126 | +Our detected card must be quite large to be detected. We therefore filter out those objects that are too small. To do this, we use the `contourArea` function to take the size of the contour and then, if it is larger than a fixed value, find a rectangle that will be able to cover it (`boundingRect` function). |
| 127 | + |
| 128 | +```js |
| 129 | +for (let i = 0; i < contoursMats.array.length; i++) { |
| 130 | + const contour = OpenCV.copyObjectFromVector(contours, i); |
| 131 | + const { value: area } = OpenCV.invoke('contourArea', contour, false); |
| 132 | + |
| 133 | + if (area > 3000) { |
| 134 | + const rect = OpenCV.invoke('boundingRect', contour); |
| 135 | + rectangles.push(rect); |
| 136 | + } |
| 137 | +} |
| 138 | +``` |
| 139 | + |
| 140 | +We can mark the elements detected in this way using Skia. |
| 141 | + |
| 142 | +```js |
| 143 | +frame.render(); |
| 144 | + |
| 145 | +for (const rect of rectangles) { |
| 146 | + const rectangle = OpenCV.toJSValue(rect); |
| 147 | + |
| 148 | + frame.drawRect( |
| 149 | + { |
| 150 | + height: rectangle.height * 4, |
| 151 | + width: rectangle.width * 4, |
| 152 | + x: rectangle.x * 4, |
| 153 | + y: rectangle.y * 4, |
| 154 | + }, |
| 155 | + paint |
| 156 | + ); |
| 157 | +} |
| 158 | +``` |
| 159 | + |
| 160 | +**IMPORTANT.** Remember to remove objects from the memory buffer at the end. Lack of this step, will result in continuous holding of values in memory - and consequently, in the case of frame processors, very fast filling of memory. |
| 161 | + |
| 162 | +```js |
| 163 | +OpenCV.clearBuffers(); // REMEMBER TO CLEAN |
| 164 | +``` |
| 165 | + |
| 166 | +Our finished frame processor looks as follows: |
| 167 | + |
| 168 | +```js |
| 169 | +const frameProcessor = useSkiaFrameProcessor((frame) => { |
| 170 | + 'worklet'; |
| 171 | + |
| 172 | + const height = frame.height / 4; |
| 173 | + const width = frame.width / 4; |
| 174 | + |
| 175 | + const resized = resize(frame, { |
| 176 | + scale: { |
| 177 | + width: width, |
| 178 | + height: height, |
| 179 | + }, |
| 180 | + pixelFormat: 'bgr', |
| 181 | + dataType: 'uint8', |
| 182 | + }); |
| 183 | + |
| 184 | + const src = OpenCV.frameBufferToMat(height, width, resized); |
| 185 | + const dst = OpenCV.createObject(ObjectType.Mat, 0, 0, DataTypes.CV_8U); |
| 186 | + |
| 187 | + const lowerBound = OpenCV.createObject(ObjectType.Scalar, 30, 60, 60); |
| 188 | + const upperBound = OpenCV.createObject(ObjectType.Scalar, 50, 255, 255); |
| 189 | + OpenCV.invoke('cvtColor', src, dst, ColorConversionCodes.COLOR_BGR2HSV); |
| 190 | + OpenCV.invoke('inRange', dst, lowerBound, upperBound, dst); |
| 191 | + |
| 192 | + const channels = OpenCV.createObject(ObjectType.MatVector); |
| 193 | + OpenCV.invoke('split', dst, channels); |
| 194 | + const grayChannel = OpenCV.copyObjectFromVector(channels, 0); |
| 195 | + |
| 196 | + const contours = OpenCV.createObject(ObjectType.MatVector); |
| 197 | + OpenCV.invoke( |
| 198 | + 'findContours', |
| 199 | + grayChannel, |
| 200 | + contours, |
| 201 | + RetrievalModes.RETR_TREE, |
| 202 | + ContourApproximationModes.CHAIN_APPROX_SIMPLE |
| 203 | + ); |
| 204 | + |
| 205 | + const contoursMats = OpenCV.toJSValue(contours); |
| 206 | + const rectangles: Rect[] = []; |
| 207 | + |
| 208 | + for (let i = 0; i < contoursMats.array.length; i++) { |
| 209 | + const contour = OpenCV.copyObjectFromVector(contours, i); |
| 210 | + const { value: area } = OpenCV.invoke('contourArea', contour, false); |
| 211 | + |
| 212 | + if (area > 3000) { |
| 213 | + const rect = OpenCV.invoke('boundingRect', contour); |
| 214 | + rectangles.push(rect); |
| 215 | + } |
| 216 | + } |
| 217 | + |
| 218 | + frame.render(); |
| 219 | + |
| 220 | + for (const rect of rectangles) { |
| 221 | + const rectangle = OpenCV.toJSValue(rect); |
| 222 | + |
| 223 | + frame.drawRect( |
| 224 | + { |
| 225 | + height: rectangle.height * 4, |
| 226 | + width: rectangle.width * 4, |
| 227 | + x: rectangle.x * 4, |
| 228 | + y: rectangle.y * 4, |
| 229 | + }, |
| 230 | + paint |
| 231 | + ); |
| 232 | + } |
| 233 | + |
| 234 | + OpenCV.clearBuffers(); // REMEMBER TO CLEAN |
| 235 | +}, []); |
| 236 | +``` |
| 237 | + |
| 238 | +### Result |
| 239 | + |
| 240 | +Cards are detected in real time when they are close enough to the lens. |
| 241 | + |
| 242 | + |
0 commit comments