Scan of perl5-5.31.6.tar.gz completes with errors

[DennisClark]

A recent scan of perl5-5.31.6.tar.gz from https://github.com/Perl/perl5/archive/v5.31.6.tar.gz reported some errors that may want attention. The scan was initiated using scancode.io and was mostly successful.

Extracting archives...
ERROR extracting: perl5-5.31.6.tar.gz-extract/perl5-5.31.6/cpan/IO-Compress/t/files/encrypt-aes.zip: Password protected archive, unable to extract
ERROR extracting: perl5-5.31.6.tar.gz-extract/perl5-5.31.6/cpan/IO-Compress/t/files/encrypt-standard.zip: Password protected archive, unable to extract
Extracting done.
Setup plugins...
Collect file inventory...
ERROR: failed to run pre-scan plugin: classify:
Traceback (most recent call last):
File "/opt/scancode-toolkit/src/scancode/cli.py", line 1044, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "/opt/scancode-toolkit/src/summarycode/classify.py", line 153, in process_codebase
set_classification_flags(resource)
File "/opt/scancode-toolkit/src/summarycode/classify.py", line 345, in set_classification_flags
resource.is_legal = is_legal = name.startswith(_LEGAL) or name.endswith(_LEGAL)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)

Scan files for: info, licenses, copyrights, packages, emails, urls with 3 process(es)...
Scanning done.
Some files failed to scan properly:
ERROR: failed to run pre-scan plugin: classify:
Traceback (most recent call last):
File "/opt/scancode-toolkit/src/scancode/cli.py", line 1044, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "/opt/scancode-toolkit/src/summarycode/classify.py", line 153, in process_codebase
set_classification_flags(resource)
File "/opt/scancode-toolkit/src/summarycode/classify.py", line 345, in set_classification_flags
resource.is_legal = is_legal = name.startswith(_LEGAL) or name.endswith(_LEGAL)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)
Summary: info, licenses, copyrights, packages, emails, urls with 3 process(es)
Errors count: 1
Scan Speed: 3.96 files/sec. 50.54 KB/sec.
Initial counts: 7258 resource(s): 6289 file(s) and 969 directorie(s)
Final counts: 7258 resource(s): 6289 file(s) and 969 directorie(s) for 78.29 MB
Timings:
scan_start: 2019-12-19T005312.260356
scan_end: 2019-12-19T012055.047237
setup_scan:licenses: 3.71s
setup: 3.71s
inventory: 1.90s
pre-scan:classify: 1.63s
pre-scan: 1.63s
scan:packages: 2.17s
scan: 1586.20s
post-scan:classify-package: 2.16s
post-scan:consolidate: 7.82s
post-scan:summary: 27.35s
post-scan:is-license-text: 2.16s
post-scan:license-clarity-score: 17.86s
post-scan:summary-key-files: 2.24s
post-scan: 59.60s
output:json-pp: 6.82s
output: 6.82s
total: 1671.77s
Removing temporary files...done.
perl5-5.31.6.tar.gz_scan.json.zip

[DennisClark]

It appears that the problems originally mentioned in this issue have been addressed, but a scan of the latest perl5 release by SCTK v32.3.3 presents different errors, including the mysterious REPLACEMENT CHARACTER :

scancode-toolkit-v32.3.3 % ./scancode -clipeu /Downloads/perl5-5.41.10/ --json-pp /Downloads/perl5-5.41.10-scan-results.json
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights, emails, urls with 1 process(es)...
[----------------#---] 294 Scanned: README.irix            Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
[--------#-----------] 296 Scanned: README.jp              Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
[-----------#--------] 1726 Scanned: jisx0208.enc                          Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
[-------------------#] 1728 Scanned: jisx0208.utf                          Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
[-----------#--------] 12478 Scanned: dbic_c3.t                            Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
[-----#--------------] 12498 Scanned: isa_dfs.t                            Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
[------------------#-] 13388 Scanned: bless.t                              Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
[####################] 13630                                               
Scanning done.
Summary:        info, packages, licenses, copyrights, emails, urls with 1 process(es)
Errors count:   0
Scan Speed:     1.12 files/sec. 15.22 KB/sec.
Initial counts: 7815 resource(s): 6815 file(s) and 1000 directorie(s) 
Final counts:   7815 resource(s): 6815 file(s) and 1000 directorie(s) for 90.09 MB
Timings:
  scan_start: 2025-04-03T204436.718842
  scan_end:   2025-04-03T222543.790600
  setup_scan:licenses: 2.23s
  setup: 2.23s
  inventory: 1.05s
  scan:packages: 0.53s
  scan:licenses: 0.44s
  scan: 6063.07s
  output:json-pp: 1.65s
  output: 1.65s
  total: 6068.87s
Removing temporary files...done.