Do you need a hand?

[gunar]

Hi @cxcorp,

In a half-dream last night I've envisioned an "AR" code breaker for Cyberpunk. I was looking into tesseract.js when I've stumbled upon your work!

I was wondering whether we should work together on this (and incur the coordination costs) or if you'd rather work at your own pace (so maybe I'll start from scratch or fork).

Let me know what you think.

Thanks!

[cxcorp]

Hey. Thanks for the interest!

I was planning on integrating this to https://github.com/cxcorp/cyberpunk2077-hacking-solver at some point but right now am working on an improved version that does all of the matrix detection and CV calls directly in C++ that's compiled to wasm. I'll put that on GitHub soon. I also got better results when detecting the matrix with DBSCAN than my previous ad-hoc solution.

Some of the things I haven't worked on yet are for example the UI shown to the user, but I had some ideas like showing a confirmation prompt where the user can manually correct the digits tesseract missed.

What did you have in mind? If you meant that you'd show the solution tracked on top of the matrix, a major challenge there is how slow and ultimately not completely trustworthy tesseract is, requiring manual corrections from the user.

[gunar]

I envisioned using AR to show an arrow overlaying the original matrix. You could make the arrow move as codes get selected. But that's a dumb idea since you have to use the controller with both hands. Overall, I was thinking that perhaps you could create some issues that I could tackle so as not to do work in duplicity. Maybe let me know here when you open source the new repo?

…

On Sat, 21 Aug 2021, 09:28 Joona Heikkilä, ***@***.***> wrote: Hey. Thanks for the interest! I was planning on integrating this to https://github.com/cxcorp/cyberpunk2077-hacking-solver at some point but right now am working on an improved version that does all of the matrix detection and CV calls directly in C++ that's compiled to wasm. I'll put that on GitHub soon. I also got better results when detecting the matrix with DBSCAN than my previous ad-hoc solution. Some of the things I haven't worked on yet are for example the UI shown to the user, but I had some ideas like showing a confirmation prompt where the user can manually correct the digits tesseract missed. What did you have in mind? If you meant that you'd show the solution tracked on top of the matrix, a major challenge there is how slow and ultimately not completely trustworthy tesseract is, requiring manual corrections from the user. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2UDXRUC5XCW7KHQUNWVK3T55PUDANCNFSM5CRTSQIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[cxcorp]

Yeah, that would definitely be cool and is in the realm of possibility! I also think that it might get a bit old after the first few times since you'd have to work on the controller with just one hand, and keep your phone pointed at the screen all the time 😄

My initial vision was to render the green rectangle on the code matrix and run Tesseract on it, then overlay the guess on top of the actual code matrix. However, Tesseract's performance isn't great, and OpenCV is also quite heavy on older phones (I get like 5 FPS on an older Nokia smartphone (5.1?), whereas I get 20-50 FPS on an OnePlus 8 Pro. So performance does limit creativity here a little bit.

Right now the prototype shows my current vision a little bit, which is to have the (possibly low FPS) camera feed visible with the green rectangle showing the detected code matrix' edges, then wait for user to press a button, after which Tesseract is ran.

I won't be able to work on this this week, but I'll let you know (probably the week after the next week).

[gunar]

Sounds good. I think we might want to move towards pictures as opposed to live camera feeds.

Also I've been thinking that maybe we don't need OCR but just feature detection. There are only 8 hex possible values so maybe that makes processing cheaper. Have you considered something like that?

[cxcorp]

I did initially did train a Tensorflow model with a few hundred images, but Tesseract just worked better (after training Tesseract's model with some samples of the correct font). I can't remember how the performance was. The Tesseract model could be improved by feeding it more images of hex bytes of varying image quality and distortion. I did not try OpenCV's feature matching, so that's definitely something worth checking out! Maybe could run it on the individual cells (of the code matrix) of the perspective corrected & thresholded image? In the worst case that'd be something like 8*7*7=392 tests if the feature matcher can't match for all 8 simultaneously. Would also need to check if feature matching is compiled into the dist OpenCV.js, or if can even be compiled in.

…

On Sun, Aug 22, 2021, 23:44 Gunar Gessner ***@***.***> wrote: Sounds good. I think we might want to move towards pictures as opposed to live camera feeds. Also I've been thinking that maybe we don't need OCR but just feature detection. There are only 8 hex possible values so maybe that makes processing cheaper. Have you considered something like that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBCGFAJC4DTJ62NDF7ZAGTT6FORVANCNFSM5CRTSQIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .