Feature Request Tracker
|Submitted by:||Morten Høgholm|
|Summary:||Load databases faster|
I think it is possible to speed up the loading of databases from external files significantly.
Currently, each new row is appended to the toks register piece-wise: First the row ID is added to the toks register and then each subsequent element is added within this where it splits the contents of the toks register in two before inserting. This is fine if you're adding single entries one at a time.
However, when loading a file with several hundred records, you could make some different arrangements to avoid the constant splitting. I was thinking a good method would be to store all row details in a separate toks register while parsing the current line. Once parsing of the entire line is complete, the resulting row entries could be appended to the database as a fast put-right operation rather than a split-up-and-insert near the end. I haven't written the code to do it but I would expect the performance to be significantly better.
Note that for my purposes it is unfortunately not an option to use the datatooltk tools.
CommentsComment from Nicola Talbot
Date: 2016-02-01 12:23:10 CST
your suggestions are much appreciated :-) Your original redesign in v2.0 was a big improvement to datatool. I think that's a good idea. I'll try implementing it when I have the time.
Watch This Report
If you supply your name, it will be used in the email greeting, which provides a more personal message, otherwise you'll just get a generic greeting. If you have previously supplied your name when signing up for notifications, you don't need to resupply it unless you want to change it.
If you have previously subscribed to notifications for this report, you can unsubscribe by clicking on the "Stop Notification" button.
The "Confirm Topic ID" field helps to protect against spambots. Please enter the feature request ID (which you can find at the top of this page).
To unsubscribe from all notifications use the notifications page.
You can append a comment to this report using the form below. Comments are checked first before being added. Any spam or offensive content will be removed first according to this site's Terms of Website Use. Please bear in mind that I develop and maintain free software in my spare time. If you want commerical level support then you can hire a TeX consultant.
The "Confirm Feature Request ID" field helps to protect against spambots. Please enter the feature request ID (which you can find at the top of this page).