Feature Tracker RSS feed

I’m currently working on a major new version of the datatool package. This may take a while. Please be patient. (Experimental version available for testing.)

ID 37🔗
Date: 2016-02-01 10:42:42
Status Open
Category datatool
Summary Load databases faster

Sign in to subscribe to notifications about this report.

Description

Hi Nicola,

I think it is possible to speed up the loading of databases from external files significantly.

Currently, each new row is appended to the toks register piece-wise: First the row ID is added to the toks register and then each subsequent element is added within this where it splits the contents of the toks register in two before inserting. This is fine if you're adding single entries one at a time.

However, when loading a file with several hundred records, you could make some different arrangements to avoid the constant splitting. I was thinking a good method would be to store all row details in a separate toks register while parsing the current line. Once parsing of the entire line is complete, the resulting row entries could be appended to the database as a fast put-right operation rather than a split-up-and-insert near the end. I haven't written the code to do it but I would expect the performance to be significantly better.

Note that for my purposes it is unfortunately not an option to use the datatooltk tools.

MWE

No mwe.tex

Evaluation

Comments

1 comment.

🔗Comment from Nicola Talbot 🦜
Date: 2016-02-01 18:23:10

Hi Morten,
your suggestions are much appreciated :-) Your original redesign in v2.0 was a big improvement to datatool. I think that's a good idea. I'll try implementing it when I have the time.

Replying to Comment #20:

Name (optional):

Are you human? Please confirm the feature request ID (which can be found at the top of this page) or login if you have an account.

Comment:


You can use the following markup:

Block:

[pre]Displayed verbatim[/pre]
[quote]block quote[/quote]

In line:

[tt]code[/tt]
[file]file/package/class name[/file]
[em]emphasized text[/em]
[b]bold text[/b]
[url]web address[/url] [sup]superscript[/sup]
[sub]subscript[/sub]

Ordered list:
[ol]
[li]first item[/li]
[li]second item[/li]
[/ol]

Unordered list:
[ul]
[li]first item[/li]
[li]second item[/li]
[/ul]

You can use the Preview button to review your message formatting before submitting.


Add Comment

Name (optional):

Are you human? Please confirm the feature request ID (which can be found at the top of this page) or login if you have an account.

Comment:


You can use the following markup:

Block:

[pre]Displayed verbatim[/pre]
[quote]block quote[/quote]

In line:

[tt]code[/tt]
[file]file/package/class name[/file]
[em]emphasized text[/em]
[b]bold text[/b]
[url]web address[/url] [sup]superscript[/sup]
[sub]subscript[/sub]

Ordered list:
[ol]
[li]first item[/li]
[li]second item[/li]
[/ol]

Unordered list:
[ul]
[li]first item[/li]
[li]second item[/li]
[/ul]

You can use the Preview button to review your message formatting before submitting.

Page permalink: https://www.dickimaw-books.com/featuretracker.php?key=37