Datamerger

NikkieK

Hello Daniel,
I am unable to install Datamerger on my system. I am using Ubuntu 16.04 and i followed the steps mentioned. It says "unable to locate package datamerger". How do i install it?

Thank you!

sebastiaan

Hi Nikkie,

The DataMerger is quite outdated by now. Below is a simple script that allows you to do more or less the same thing. The way it works:

• Add all .csv files that you want to merge to the OpenSesame file pool
• Add this script as an inline_script item
• In the inline_script item, select all text, and click on the 'Run selected text' button. (So don't run the experiment itself!)
• You will then find the merged files in the file pool

I'll see if I can turn this into a simple extension to make it easier.

Cheers!
Sebastiaan

import os
from datamatrix import io, DataMatrix

# This points toward the file pool if you run the script from within OpenSesame
# by selecting the script and clicking on the 'Run selected text' button.
# This will *not* work when running as an experiment, or as a standalone script.
DIRNAME = opensesame.experiment.pool_folder

dm = DataMatrix()
# Loop through all files in the source directory
for basename in os.listdir(DIRNAME):
    # Skip files that aren't csv or start with merged (because that's the
    # output file)
    if not basename.endswith('.csv') or basename.startswith('merged'): 
        continue
    try:
        filedm = io.readtxt(os.path.join(DIRNAME, basename))
    except Exception as e:
        print('Failed to open %s (Exception: %s)' % (basename, e))
    else:
        # Add a source column, and then concatenate the input file and the full
        # DataMatrix
        print('Read %s' % basename)
        filedm.source = basename
        dm <<= filedm
# Save to the source directory as both csv and xlsx
io.writetxt(dm, os.path.join(DIRNAME, 'merged.csv'))
io.writexlsx(dm, os.path.join(DIRNAME, 'merged.xlsx'))

NikkieK

Hello again...
So this happened when I ran the script...what went wrong?

 /usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building  the font cache using fc-list. This may take a moment.
   warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
 /usr/lib/python2.7/dist-packages/datamatrix/_datamatrix/_basecolumn.py:33: UserWarning: Install  fastnumbers for better performance
   warnings.warn('Install fastnumbers for better performance')
 Read subject-8.csv
 Read subject-9.csv
 Read subject-13.csv
 Read subject-6.csv
 Read subject-5.csv
 Read subject-14.csv
 Read subject-11.csv
 Read subject-7.csv
 Read subject-10.csv
 Read subject-15.csv
 Read subject-1.csv
 Read subject-3.csv
 Read subject-12.csv
 Read subject-2.csv
 Read subject-4.csv
 /usr/lib/python2.7/dist-packages/datamatrix/_datamatrix/_numericcolumn.py:35: UserWarning: Install  fastnumbers for better performance
   warnings.warn('Install fastnumbers for better performance')
---------------------------------------------------------------------------
CellCoordinatesException                  Traceback (most recent call last)
<ipython-input-1-d06b02b6e2e2> in <module>()
     26 # Save to the source directory as both csv and xlsx
     27 io.writetxt(dm, os.path.join(DIRNAME, 'merged.csv'))
---> 28 io.writexlsx(dm, os.path.join(DIRNAME, 'merged.xlsx'))

/usr/lib/python2.7/dist-packages/datamatrix/io/_excel.pyc in writexlsx(dm, path)
     90         ws = wb.active
     91         for colnr, colname in enumerate(dm.column_names):
---> 92                 ws[chr(ord('A')+colnr)+'1'] = colname
     93         for rownr, row in enumerate(dm):
     94                 for colnr, (colname, value) in enumerate(row):

/usr/lib/python2.7/dist-packages/openpyxl/worksheet/worksheet.pyc in __setitem__(self, key, value)
    337 
    338     def __setitem__(self, key, value):
--> 339         self[key].value = value
    340 
    341 

/usr/lib/python2.7/dist-packages/openpyxl/worksheet/worksheet.pyc in __getitem__(self, key)
    333         if ":" in key:
    334             return self.iter_rows(key)
--> 335         row, column = coordinate_to_tuple(key)
    336         return self._get_cell(row, column)
    337 

/usr/lib/python2.7/dist-packages/openpyxl/utils/__init__.pyc in coordinate_to_tuple(coordinate)
    160     Convert an Excel style coordinate to (row, colum) tuple
    161     """
--> 162     col, row = coordinate_from_string(coordinate)
    163     return row, _COL_STRING_CACHE[col]
    164 

/usr/lib/python2.7/dist-packages/openpyxl/utils/__init__.pyc in coordinate_from_string(coord_string)
     40     if not match:
     41         msg = 'Invalid cell coordinates (%s)' % coord_string
---> 42         raise CellCoordinatesException(msg)
     43     column, row = match.groups()
     44     row = int(row)

CellCoordinatesException: Invalid cell coordinates ([1)`

sebastiaan

Ow right. There used to be a bug when writing .xlsx files with a large number of columns. You can work around this by either:

• commenting out the line where the .xlsx file is written. You'll still have the .csv, so that's probably fine; or:
• update python-datamatrix. You can do this with pip install python-datamatrix --upgrade.