Feature Request Tracker

ID:37 🔗
Submitted by:Morten Høgholm
Date:2016-02-01 10:42:42
Summary:Load databases faster


Hi Nicola,

I think it is possible to speed up the loading of databases from external files significantly.

Currently, each new row is appended to the toks register piece-wise: First the row ID is added to the toks register and then each subsequent element is added within this where it splits the contents of the toks register in two before inserting. This is fine if you're adding single entries one at a time.

However, when loading a file with several hundred records, you could make some different arrangements to avoid the constant splitting. I was thinking a good method would be to store all row details in a separate toks register while parsing the current line. Once parsing of the entire line is complete, the resulting row entries could be appended to the database as a fast put-right operation rather than a split-up-and-insert near the end. I haven't written the code to do it but I would expect the performance to be significantly better.

Note that for my purposes it is unfortunately not an option to use the datatooltk tools.


Comment from Nicola Talbot
Date: 2016-02-01 12:23:10 CST

Hi Morten,
your suggestions are much appreciated :-) Your original redesign in v2.0 was a big improvement to datatool. I think that's a good idea. I'll try implementing it when I have the time.

Add comment or return to search results.

Watch This Report

If you would like to be notified whenever updates are made to this report, please fill in your email address in the box below and click on "Notify Me of Changes" button. (Please ensure the address is valid.) Your details won't be passed on to third parties in line with this site's Privacy Policy.

If you supply your name, it will be used in the email greeting, which provides a more personal message, otherwise you'll just get a generic greeting. If you have previously supplied your name when signing up for notifications, you don't need to resupply it unless you want to change it.

If you have previously subscribed to notifications for this report, you can unsubscribe by clicking on the "Stop Notification" button.

The "Confirm Topic ID" field helps to protect against spambots. Please enter the feature request ID (which you can find at the top of this page).

E mail:
Confirm E mail:
Confirm Topic ID:

To unsubscribe from all notifications use the notifications page.


You can append a comment to this report using the form below. Comments are checked first before being added. Any spam or offensive content will be removed first according to this site's Terms of Website Use. Please bear in mind that I develop and maintain free software in my spare time. If you want commerical level support then you can hire a TeX consultant.

The "Confirm Feature Request ID" field helps to protect against spambots. Please enter the feature request ID (which you can find at the top of this page).

(Optional. If provided, it will be shown with the comment.)
Confirm Feature Request ID:

You can use the following markup:

[pre]Displayed verbatim[/pre]
[tt]monospace text[/tt]
[em]emphasized text[/em]
[b]bold text[/b]
[url]web address[/url]

Ordered list:
[li]first item[/li]
[li]second item[/li]

Unordered list:
[li]first item[/li]
[li]second item[/li]

Click on the Preview button to preview the message.

Return to search results.