M. Selenium, Chromium and Web Scraping

It is quite impossible to find a decent driver that can help you headless scrape annoying javascript web pages, I spent two days trying to install gekodriver and i always got stuck with the same OSError: [Errno 8] Exec format error so something was not right, but what?!

After endless google searches I found that you can use Chromium as your go to browser but again i found myself in the same error rabbit hole🐇 and what do you know? the same error OSError: [Errno 8] Exec format error, so i was going mad, but it turns out i was doing it all wrong so without furhter adue, here are the steps:

A. Install Selenium

$ sudo pip3 install selenium

B. Install Chromium web driver

You don't need the whole browser (I think) I have installed because it's faster but because it's not that secure I only use it to browse StackOverflow, you only need the web driver.

I've went through hundreds of web pages, all saying you should install venv, gekodriver, and a bunch of other stuff but the solutios was right there, in my face.

$ sudo apt-get install chromium-chromedriver

$ whereis chromedriver

chromedriver: /usr/bin/chromedriver

and now we run the script : selenium_headless_test.py

Results: