Memoize permission Error

Bernd_Douze

I've been working on a project in which I've decided to analyze both pupils. For this I'm using the eyelink parser, then load in the data twice (once for the right eye, once for the left eye). All this works perfectly, untill I get to 17 files. Suddenly i get the error message:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '.memoize\\.2839819153104-7068.memmap'

When I remove any file (tried multiple different ones) the error disappears and it works fine again. I accepted this as the limit, which is still found a little strange since 16 files is not that much, so I made an new folder, added the next 16 files in there, made the loading a function that uses the right paths and the eyelink parser. But then suddenly it can only take 10 files and not even 16 anymore. Made a new folder, that one could store 10 again, but the next one only 8. So i'm a little lost. It also does not seem to be caused by the size of the folder (they range from 800mb (do not work) to 1070 mb (still work)).

@fnc.memoize(persistent=True)
def get_data(folder_path):

    # The heavy lifting is done by eyelinkparser.parse()
    dm = parse(
        folder=folder_path,       # Folder path to .asc files
        traceprocessor=defaulttraceprocessor(
          blinkreconstruct=True,  # Interpolate pupil size during blinks
          downsample=10,          # Reduce sampling rate to 100 Hz,
          mode='advanced'         # Use the new 'advanced' algorithm
        )
    )
    dm = dm[dm.trialid, dm.blockcount, dm.invalid, dm.ptrace_bl, dm.ptrace_stim, dm.participant,
            dm.stim_con, dm.xtrace_stim, dm.ytrace_stim, dm.xtrace_bl, dm.ytrace_bl, dm.condition, dm.tact1]
    dm = dm.invalid == 0  # to filter all the invalid trials
    dm = dm.trialid != 0  # to filter the duplicate first trials

    return dm


get_data.clear()                                                           # clear data, otherwise is stored
left_folder = 'C:\\N&C Internship\\DATA_ANALYSIS\\Left_20_25'      # path for the left eye placebo
right_folder = 'C:\\N&C Internship\\DATA_ANALYSIS\\Right_20_25' 

this is the general code i use in which i can the folder paths to 1_8, 9_13, 14_19

Long story short, my question is: what are the limitations of the memoize function and how can i prevent this from happening? Is there a better way to load in all the data (I have 4 files per participant, 2 sessions x 2 eyes, 32 participants).

Any help is welcome, thank you in advance!

sebastiaan

Hi @Bernd_Douze ,

In general terms, what happens is that the size of the data exceeds the available memory. DataMatrix then starts to offload the data to disk, which results in .memmap files. Apparently, multiple processes are trying to access these files simultaneously, which Windows does not allow. This appears to be the situation, but I'm not sure why this situation arises. Can you post the full traceback?

For now, reducing memory consumption may resolve the issue as well (because it will avoid memory offloading). To do so, you can use the phasefilter argument to retain only specific phases, the maxtracelen argument to make sure that traces aren't unnecessarily long, and the pupil_size , gaze_pos , and time_trace arguments to retain only the data that you need.

• https://github.com/open-cogsci/eyelinkparser

— Sebastiaan

Bernd_Douze

Thank you for taking the time to reply! I'll look into reducing the data size to see if that works. The full traceback is:

  File "C:\Users\bernd\PycharmProjects\N&C_Internship_Pupil&Touch\Data_analysis_pupil.py", line 60, in <module>
    dm_comb = get_data(left_folder)                    # get data from left_eye folder, 'comb' bc will add right to this
              ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bernd\PycharmProjects\N&C_Internship_Pupil&Touch\.venv\Lib\site-packages\datamatrix\_functional\_memoize.py", line 234, in _call_without_arguments
    return self._write_cache(
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\bernd\PycharmProjects\N&C_Internship_Pupil&Touch\.venv\Lib\site-packages\datamatrix\_functional\_memoize.py", line 324, in _write_cache
    io.writebin(retval, cache_path)
  File "C:\Users\bernd\PycharmProjects\N&C_Internship_Pupil&Touch\.venv\Lib\site-packages\datamatrix\io\_bin.py", line 143, in writebin
    aux_path.unlink()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\pathlib.py", line 1147, in unlink
    os.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '.memoize\\.2216457578384-18380.memmap'


Process finished with exit code 1