You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I try to fetch multiple single files from a large osf repo with many files (https://osf.io/9fw7).
Due to the know constraints (e.g. #155, #148, #149 ) osfclient recursively goes through all files and checks each file against the one we want to fetch.
This is of course highly inefficient, because not every file from every subfolder needs to be tested, because we can stop if a subfolder already doesnt match the path of the to-be-fetched-file.
E.g. if the file is in /A/B/C/D/E.txt, we don't have to go through /A/A/... or /A/B/A etc.
But that is what is currently happening and, therefore, it takes ages.
I didn't have a good idea how we could fix it though, it relies on the recursive call to "children", which doesnt have a method to end early / select children in a smarter way.
Best, Bene
The text was updated successfully, but these errors were encountered:
I try to fetch multiple single files from a large osf repo with many files (https://osf.io/9fw7).
Due to the know constraints (e.g. #155, #148, #149 ) osfclient recursively goes through all files and checks each file against the one we want to fetch.
This is of course highly inefficient, because not every file from every subfolder needs to be tested, because we can stop if a subfolder already doesnt match the path of the to-be-fetched-file.
E.g. if the file is in /A/B/C/D/E.txt, we don't have to go through /A/A/... or /A/B/A etc.
But that is what is currently happening and, therefore, it takes ages.
I didn't have a good idea how we could fix it though, it relies on the recursive call to "children", which doesnt have a method to end early / select children in a smarter way.
Best, Bene
The text was updated successfully, but these errors were encountered: