I’m currently working on a major new version of the datatool package. This may take a while. Please be patient. (Experimental version available for testing.)
Sign in if you want to like this report.
|Load databases faster
Sign in to subscribe to notifications about this report.
I think it is possible to speed up the loading of databases from external files significantly.
Currently, each new row is appended to the toks register piece-wise: First the row ID is added to the toks register and then each subsequent element is added within this where it splits the contents of the toks register in two before inserting. This is fine if you're adding single entries one at a time.
However, when loading a file with several hundred records, you could make some different arrangements to avoid the constant splitting. I was thinking a good method would be to store all row details in a separate toks register while parsing the current line. Once parsing of the entire line is complete, the resulting row entries could be appended to the database as a fast put-right operation rather than a split-up-and-insert near the end. I haven't written the code to do it but I would expect the performance to be significantly better.
Note that for my purposes it is unfortunately not an option to use the datatooltk tools.
Page permalink: https://www.dickimaw-books.com/featuretracker.php?key=37