ModuleNotFoundError -- 'elftools.common.py3compat' -- after installed `compiledcode`

[chinyeungli]

Description

I installed compiledcode in the SCTK's venv aiming to use the --dwarf plugin

(venv) thomas@DESKTOP-6IMEJ77:/mnt/c/Users/thoma/vscode/scancode-toolkit$ pip install compiledcode

When I run the --dwarf option, I get "ModuleNotFoundError"

(venv) thomas@DESKTOP-6IMEJ77:/mnt/c/Users/thoma/vscode/scancode-toolkit$ scancode --dwarf /mnt/c/Users/thoma/Downloads/perl-Event-Lib-1.03-1.el6.rf.i686.rpm-extract/usr/lib/perl5/vendor_perl/auto/Event/Lib/Lib..s --json-pp -
========================================================================
ERROR: UNABLE TO IMPORT SCANCODE PLUGINS.
Check your installation configuration (setup.py) or re-install/re-configure ScanCode.
The following plugin(s) are referenced and cannot be loaded/imported:
No module named 'elftools.common.py3compat'
========================================================================
Traceback (most recent call last):
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/venv/bin/scancode", line 5, in <module>
    from scancode.cli import scancode
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/src/scancode/cli.py", line 132, in <module>
    raise e
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/src/scancode/cli.py", line 119, in <module>
    plugin_classes, plugin_options = PluginManager.load_plugins()
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/venv/lib/python3.12/site-packages/plugincode/__init__.py", line 199, in load_plugins
    mgr_setup = manager.setup()
                ^^^^^^^^^^^^^^^
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/venv/lib/python3.12/site-packages/plugincode/__init__.py", line 225, in setup
    raise e
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/venv/lib/python3.12/site-packages/plugincode/__init__.py", line 223, in setup
    self.manager.load_setuptools_entrypoints(entrypoint)
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/venv/lib/python3.12/site-packages/pluggy/_manager.py", line 416, in load_setuptools_entrypoints
    plugin = ep.load()
             ^^^^^^^^^
  File "/usr/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/venv/lib/python3.12/site-packages/compiledcode/dwarf/__init__.py", line 27, in <module>
    from compiledcode.dwarf import dwarfng
  File "/mnt/c/Users/thoma/vscode/scancode-toolkit/venv/lib/python3.12/site-packages/compiledcode/dwarf/dwarfng.py", line 19, in <module>
    from elftools.common.py3compat import bytes2str
ModuleNotFoundError: No module named 'elftools.common.py3compat'
(venv) thomas@DESKTOP-6IMEJ77:/mnt/c/Users/thoma/vscode/scancode-toolkit$

Note: After installing compiledcode, I'm unable to use ScanCode-Toolkit.
Running any scancode command, even scancode -h, results in a ModuleNotFoundError.

System configuration

• What OS are you running on? (Windows/MacOS/Linux) Windows
• What version of scancode-toolkit was used to generate the scan file? latest from the develop branch
• What installation method was used to install/run scancode? (pip/source download/other) git clone

[AyanSinhaMahapatra]

https://github.com/aboutcode-org/scancode-plugins/tree/main/binary-analysis/scancode-compiledcode is quite old, and it seems pyelftools has since dropped the elftools.common.py3compat module since this was maintaining some sorts of backwards compatibility and is no longer needed anymore? Can https://docs.python.org/3/library/stdtypes.html#bytes.decode be used instead?

If this is not tested in the CI soemwhere, there is no proof that we actively support these at all. Since modules can be removed deprecated and dependencies can be updated.

We need to:

1. Update https://github.com/aboutcode-org/scancode-plugins with latest skeleton
2. Make sure that there are tests running in the CI for the python versions we choose to support, that installs these plugins with scancode and runs some basic tests across python versions in ubuntu atleast. This can be at https://github.com/aboutcode-org/scancode-plugins or even on the scancode side for important plugins

[chinyeungli]

@AyanSinhaMahapatra Can you instruct me on how can I use the --dwarf plugin.

I was told that I need to do pip install compliedcode to use --dwarf

[chinyeungli]

I followed the instruction at https://scancode-toolkit.readthedocs.io/en/latest/tutorials/how_to_add_a_post_scan_plugin.html#how-to-add-post-scan-plugin , but encountered the same error related to compiledcode. I cloned the scancode-plugins repository (https://github.com/aboutcode-org/scancode-plugins), navigated to binary-analysis/scancode-compiledcode, installed it using pip install ., and ran a scancode scan — the same error persisted.