Datamerger

Comments

• sebastiaan

  April 2017

  Hi Nikkie,

  The DataMerger is quite outdated by now. Below is a simple script that allows you to do more or less the same thing. The way it works:

  • Add all .csv files that you want to merge to the OpenSesame file pool
  • Add this script as an inline_script item
  • In the inline_script item, select all text, and click on the 'Run selected text' button. (So don't run the experiment itself!)
  • You will then find the merged files in the file pool

  I'll see if I can turn this into a simple extension to make it easier.

  Cheers!
  Sebastiaan

  import os
  from datamatrix import io, DataMatrix
  
  # This points toward the file pool if you run the script from within OpenSesame
  # by selecting the script and clicking on the 'Run selected text' button.
  # This will *not* work when running as an experiment, or as a standalone script.
  DIRNAME = opensesame.experiment.pool_folder
  
  dm = DataMatrix()
  # Loop through all files in the source directory
  for basename in os.listdir(DIRNAME):
      # Skip files that aren't csv or start with merged (because that's the
      # output file)
      if not basename.endswith('.csv') or basename.startswith('merged'): 
          continue
      try:
          filedm = io.readtxt(os.path.join(DIRNAME, basename))
      except Exception as e:
          print('Failed to open %s (Exception: %s)' % (basename, e))
      else:
          # Add a source column, and then concatenate the input file and the full
          # DataMatrix
          print('Read %s' % basename)
          filedm.source = basename
          dm <<= filedm
  # Save to the source directory as both csv and xlsx
  io.writetxt(dm, os.path.join(DIRNAME, 'merged.csv'))
  io.writexlsx(dm, os.path.join(DIRNAME, 'merged.xlsx'))
  

  
  
  Check out SigmundAI.eu for our OpenSesame AI assistant!
  
• NikkieK

  April 2017 edited April 2017

  Hello again...
  So this happened when I ran the script...what went wrong?

   /usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building  the font cache using fc-list. This may take a moment.
     warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
   /usr/lib/python2.7/dist-packages/datamatrix/_datamatrix/_basecolumn.py:33: UserWarning: Install  fastnumbers for better performance
     warnings.warn('Install fastnumbers for better performance')
   Read subject-8.csv
   Read subject-9.csv
   Read subject-13.csv
   Read subject-6.csv
   Read subject-5.csv
   Read subject-14.csv
   Read subject-11.csv
   Read subject-7.csv
   Read subject-10.csv
   Read subject-15.csv
   Read subject-1.csv
   Read subject-3.csv
   Read subject-12.csv
   Read subject-2.csv
   Read subject-4.csv
   /usr/lib/python2.7/dist-packages/datamatrix/_datamatrix/_numericcolumn.py:35: UserWarning: Install  fastnumbers for better performance
     warnings.warn('Install fastnumbers for better performance')
  ---------------------------------------------------------------------------
  CellCoordinatesException                  Traceback (most recent call last)
  <ipython-input-1-d06b02b6e2e2> in <module>()
       26 # Save to the source directory as both csv and xlsx
       27 io.writetxt(dm, os.path.join(DIRNAME, 'merged.csv'))
  ---> 28 io.writexlsx(dm, os.path.join(DIRNAME, 'merged.xlsx'))
  
  /usr/lib/python2.7/dist-packages/datamatrix/io/_excel.pyc in writexlsx(dm, path)
       90         ws = wb.active
       91         for colnr, colname in enumerate(dm.column_names):
  ---> 92                 ws[chr(ord('A')+colnr)+'1'] = colname
       93         for rownr, row in enumerate(dm):
       94                 for colnr, (colname, value) in enumerate(row):
  
  /usr/lib/python2.7/dist-packages/openpyxl/worksheet/worksheet.pyc in __setitem__(self, key, value)
      337 
      338     def __setitem__(self, key, value):
  --> 339         self[key].value = value
      340 
      341 
  
  /usr/lib/python2.7/dist-packages/openpyxl/worksheet/worksheet.pyc in __getitem__(self, key)
      333         if ":" in key:
      334             return self.iter_rows(key)
  --> 335         row, column = coordinate_to_tuple(key)
      336         return self._get_cell(row, column)
      337 
  
  /usr/lib/python2.7/dist-packages/openpyxl/utils/__init__.pyc in coordinate_to_tuple(coordinate)
      160     Convert an Excel style coordinate to (row, colum) tuple
      161     """
  --> 162     col, row = coordinate_from_string(coordinate)
      163     return row, _COL_STRING_CACHE[col]
      164 
  
  /usr/lib/python2.7/dist-packages/openpyxl/utils/__init__.pyc in coordinate_from_string(coord_string)
       40     if not match:
       41         msg = 'Invalid cell coordinates (%s)' % coord_string
  ---> 42         raise CellCoordinatesException(msg)
       43     column, row = match.groups()
       44     row = int(row)
  
  CellCoordinatesException: Invalid cell coordinates ([1)`
  
• sebastiaan

  April 2017

  Ow right. There used to be a bug when writing .xlsx files with a large number of columns. You can work around this by either:

  • commenting out the line where the .xlsx file is written. You'll still have the .csv, so that's probably fine; or:
  • update python-datamatrix. You can do this with pip install python-datamatrix --upgrade.

  
  
  Check out SigmundAI.eu for our OpenSesame AI assistant!