Memoize permission Error
I've been working on a project in which I've decided to analyze both pupils. For this I'm using the eyelink parser, then load in the data twice (once for the right eye, once for the left eye). All this works perfectly, untill I get to 17 files. Suddenly i get the error message:
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '.memoize\\.2839819153104-7068.memmap'
When I remove any file (tried multiple different ones) the error disappears and it works fine again. I accepted this as the limit, which is still found a little strange since 16 files is not that much, so I made an new folder, added the next 16 files in there, made the loading a function that uses the right paths and the eyelink parser. But then suddenly it can only take 10 files and not even 16 anymore. Made a new folder, that one could store 10 again, but the next one only 8. So i'm a little lost. It also does not seem to be caused by the size of the folder (they range from 800mb (do not work) to 1070 mb (still work)).
@fnc.memoize(persistent=True)
def get_data(folder_path):
# The heavy lifting is done by eyelinkparser.parse()
dm = parse(
folder=folder_path, # Folder path to .asc files
traceprocessor=defaulttraceprocessor(
blinkreconstruct=True, # Interpolate pupil size during blinks
downsample=10, # Reduce sampling rate to 100 Hz,
mode='advanced' # Use the new 'advanced' algorithm
)
)
dm = dm[dm.trialid, dm.blockcount, dm.invalid, dm.ptrace_bl, dm.ptrace_stim, dm.participant,
dm.stim_con, dm.xtrace_stim, dm.ytrace_stim, dm.xtrace_bl, dm.ytrace_bl, dm.condition, dm.tact1]
dm = dm.invalid == 0 # to filter all the invalid trials
dm = dm.trialid != 0 # to filter the duplicate first trials
return dm
get_data.clear() # clear data, otherwise is stored
left_folder = 'C:\\N&C Internship\\DATA_ANALYSIS\\Left_20_25' # path for the left eye placebo
right_folder = 'C:\\N&C Internship\\DATA_ANALYSIS\\Right_20_25'
this is the general code i use in which i can the folder paths to 1_8, 9_13, 14_19
Long story short, my question is: what are the limitations of the memoize function and how can i prevent this from happening? Is there a better way to load in all the data (I have 4 files per participant, 2 sessions x 2 eyes, 32 participants).
Any help is welcome, thank you in advance!
Comments
Hi @Bernd_Douze ,
In general terms, what happens is that the size of the data exceeds the available memory. DataMatrix then starts to offload the data to disk, which results in
.memmapfiles. Apparently, multiple processes are trying to access these files simultaneously, which Windows does not allow. This appears to be the situation, but I'm not sure why this situation arises. Can you post the full traceback?For now, reducing memory consumption may resolve the issue as well (because it will avoid memory offloading). To do so, you can use the
phasefilterargument to retain only specific phases, themaxtracelenargument to make sure that traces aren't unnecessarily long, and thepupil_size,gaze_pos, andtime_tracearguments to retain only the data that you need.— Sebastiaan
Check out SigmundAI.eu for our OpenSesame AI assistant!
Thank you for taking the time to reply! I'll look into reducing the data size to see if that works. The full traceback is: