many standard library realloc () are incorrect, I have therefore never used, I often have the fragmentation in my projects prevented either by unique allocation of a sufficiently large buffers, or by malloc and free (and retrieve it if it is necessary a larger buffer to allocate for example (10kb). So at start of scans alloc a probably large enough block, and if a new one is needed first check whether you need a bigger, if so then free () and then renewed malloc () when the required buffer smaller is then allocated to previously used again and again.
Excuse me, it's just a tip, I would not make me important.

that's one of the things i'm intending to try out within the confines of how things currently work i.e. attempt to allocate things in one go, though i think trying to use something like a memory mapped file might instead be the better way to go (it's used for a few other things, but not the main bulk of the db handling and that's something the OS can help out with if i remember how all of that works - has been a decade since i last looked into such things). is just time that's the issue to try things out...

caution the mmap (CreateFileMapping) win32 implementation is very very bad the fragmented os the total memory until pagefile is full. It is ok if you always need a couple of a few files.