Folder lock (check new mails) seems to abort import process

[kenjiuno]

Hi, I love this import-export-tools-ng and it is great!
I'm just here to report a issue I have encountered.

I'm trying to import about 20,000 EML files from Windows Live Mail into Inbox, Send Items and other folders respectively.
The import process breaks sometimes. It is after adding 5,844 mails or 699 mails or such. It is unstable count.

Keep open the Ctrl + Shift + I window, and I have observed random errors while importing process.

The following comment from my googling guesses its failure reason.

wangvisual/autoarchive#14 (comment)

While I'm importing EMLs into Inbox, check new mail seems to terminate import process.

I'll test if disable check new mail works or not, soon!

[kenjiuno]

Ok I have successfully imported 15,561 EML files from WLM Inbox into TB Inbox by turning off periodical mail checking feature.

I'm happy if it can retry and continue import process on such a error...

[cleidigh]

@kenjiuno
Very glad you like the extension!
I have been working on several issues with large imports. terror several problems some of which I have fixes some still outstanding.
I also have a new prototype for EML import that is completely new.
I will be rewriting a lot of the coat over the next couple months as it needs updating/better error handling. One known issue occurs with over ~400 folders. The test extension below as number of options to do testing. It solves some of these issues and should be somewhat faster.
10k messages/780MB took 86 seconds on my machine. If you are interested, you can give it a try for comparison:

• Use the folder menu option "ImportEMLStructuredExt"
• choose your folder and should import everything
• extension goes off-line automatically
• It will also tell you the total time at completion

I am really trying to get this better, any help appreciated.
Did you see the new help ? I was able to fully localize it!
Thanks!
@cleidigh
https://github.com/cleidigh/iet-tests/blob/master/xpi/iet-ng-tests-0.1.8-tb.xpi

[kenjiuno]

@cleidigh Thanks for reply!
I have tested the new iet-ng-tests-0.1.8-tb.xpi and importing feature works very well!

(as a side note, while import process with ImportExportTools NG, I have added thunderbird.exe to Windows Defender exclusion process so that it can import 1,000 mails in one minute!)

If it is possible, I'm glad if import recursive on/off can be selected as ImportExportTools NG does:

At Windows Live Mail, the actual folder structure doesn't match the folder view I see in Windows Live Mail.
Windows Live Mail doesn't move or rename the actual folder whenever I rename or move the folder seen in WLM.
It is not mandatory because I can arrange the folder structure by drag and drop operation in TB after import!

Did you see the new help ? I was able to fully localize it!

I'm so surprised that all (add-on menu, popup, and help too) are translated!
I have glanced at texts, it will be OK!
I'm very thank you for great works!

[kenjiuno]

The new version iet-ng-tests-0.1.8-tb.xpi has new feature Use off-line mode for import.
Thus I believe that the import process won't disturbed by periodical tasks any more...

[cleidigh]

@kenjiuno
iet-test is supposed to be just a test setup, however, I would be happy to make it possible to do either single, folder or for structured imports.
I am very curious if you did any speed tests compare to prior versions.
I am going to continue to add to this and improve both the speed and quality.

[kenjiuno]

Ok here is a speed compare of 45,539 mails import:

	Windows Defender on	Windows Defender off
ImportExportTools NG	*	695 sec
iet-ng-tests-0.1.8-tb	1233 sec	810 sec

• 4,619 in 900 secs

[cleidigh]

@kenjiuno
Excellent, what was the amount of data?
v0.1.10 now has the ability to import either folder/structure
The new experimental trial ( option three)
is rough , however it takes advantage of web workers and should be ~30% ,
then v.0.1.8
curious what you see

[kenjiuno]

Ok thanks!

I have tested iet-ng-tests-0.1.14-tb.xpi today.
45,539 mails (total 4.24 GB) in 1257.471 secs.

Windows defender on:

Windows defender off:

45,539 mails (total 4.24 GB)	Windows Defender on	Windows Defender off
ImportExportTools NG	*	695 sec
iet-ng-tests-0.1.8-tb	1233 sec	810 sec
iet-ng-tests-0.1.14-tb.xpi	1257 sec	601 sec

The converter performance is incredible!

[cleidigh]

@kenjiuno
Thanks, that's excellent! 15+ I have even improved things further. The main thing I wanted to do with this is to correct any import errors/edge conditions. I want to roll these into the main extension but I have to figure out license issues.
Just for your understanding:

• the algorithm is completely new, for utilizing web workers to improve performance
• I am using the OS.File API for the best web worker performance
• The following long existing problems have been fixed:
  • detect and handle paths that are too long (250 characters on Windows)
  • handle subfolder creation where there are >500 Windows file handles open - cause errors
  • handle similar problem with adding messages to many different folders
    I will tweak a few more things and make a new Release.

What did you think of the new status dialogue it is not completely pretty yet, but I think useful?