reading .xpd file

Sarah

In the block design code, I saved "reaction_time" or rt and reponse_key (kr).

                response_keys, rt = exp.keyboard.wait(keys=rk ,duration=rt)
          ...
            exp.data.add([kr, rt])

The output .xpd file looks look as follow:

    #Expyriment 0.9.0 (Python 2.7.13), .xpd-file, coding: UTF-8
    #date: Tue Jun 13 2017 19:19:35
    #--EXPERIMENT INFO
    #e mainfile: Semantic_word_selection.py
    #e sha1: None
    #e modules: 
    #e Experiment: semantic verbal fluency
    #e no between subject factors
    #e Block 0: unnamed
    #e     block factors: 
    #e     n trials: 7
    #e     trial factors: 
    #e Block 1: unnamed
    #e     block factors: 
    #e     n trials: 7
           ......
            #--SUBJECT INFO
    #s id: 2
    subject_id,
    2,276,281
    2,None,None
    2,276,377
    2,None,None
    2,276,387
    2,None,None
    2,275,359
    2,275,852
    2,275,317
    2,275,79
    2,None,None
    2,276,251

The first question is why the name of the variables (rt, kr) did not save after subject Id in .xpd file?
nevertheless when I open the file using agg = data_preprocessing.Aggregator(data_folder, file_name, suffix='.xpd'), I could see the name of the variables are empty and I get the error below: do you have any idea what this issue happened?

    ** RuntimeError: Different variables in 
    [u'subject_id', u'']
    instead of
    [u'subject_id']
    (Pdb) 

How I could be able to use the command of agg.set_subject_variables(...) and
agg.set_computed_variables() to analayse this data ?

Thanks

fladd

Hi Sarah,

it seems that you forgot to add the data variable names to the data file.
You have to define them once at the beginning of your experiment (see here for an example).

In your case that would be:

exp.data_variable_names = ["Response", "RT"]

Best,
Florian

Sarah

I had exp.data_variable_names in my code but still I cant see the name of the variables
here is the code:

    exp.add_data_variable_names = ["button", "rt"]
    #### start experiment############
    expyriment.control.start()


    for rule, block in zip(rule_list, exp.blocks)


        for trial in block.trials:


            st=trial.stimuli[0].present()

            button, rt = exp.keyboard.wait(keys=response_keys,duration=reaction_time-st)
            if rt is not None:
               exp.clock.wait(reaction_time-rt) 
            exp.data.add([button,rt])
            exp.data.save()  

        bt=blank.present()
        exp.clock.wait(blank_page_time-bt)

any comment?

fladd

Hi Sarah,

you have add_data_variable_names in your code, which is a method, not an attribute, like data_variable_names which I suggested to use.

Hence, either of the following two will work:
1. exp.add_data_variable_names(["response_keys", "RT"])
2. exp.data_variable_names = ["response_keys", "RT"]

Best,
Florian

Sarah

Here is the result of the task in brief:

    subject_id,block.name,button,rt
    3,A4,275,2
    3,A4,275,121
    3,A4,None,None
    3,A4,None,None

Is there any way of programming that I could open .xpd file in a way that I could above result in an array? I think working with an array is much easier for further processing. Unfortunately, The short code that I paste below did not do the job

    data_folder='./data'
    file_name='./name'
    agg = data_preprocessing.Aggregator(data_folder, file_name, suffix='.xpd')
    #
    agg.set_subject_variables(["subject_id","b.name","button", "rt"])
    agg.set_computed_variables(["button", "rt")]

BTW: In my code, I only saved 4 variables but in the .xpd file I could see a long list of block name is saved:
Is there any way to remove these on .xpd file?

        #e     block factors: 
    #e     n trials: 7
    #e     trial factors: 
    #e Block 1: AS6
    #e     block factors: 
    #e     n trials: 7
    #e     trial factors: 
    #e Block 2: T6
    #e     block factors: 
    #e     n trials: 7
    #e     trial factors: 
    #e Block 3: T3
    #e     block factors: 
    #e     n trials: 7
    #e     trial factors: 
    #e Block 4: A8

Thanks

fladd

The Expyriment log files are all saved in a standard comma separated value (csv) format, and it should hence be relatively straight forward to convert them into any other data structure or parse them with external tools if necessary.
The block names (the whole design actually) are entered in the log files as a csv comment, so it will not interfere with your data, when processing it further outside of Expyriment.

Concerning the code snippet of the Expyriment preprocessing, it seems that filename is not the correct value. It should be set to a string that represents how the data files begin. For instance, if my experiment is called "MyExp", then I would also set filename in the Aggregator to "MyExp", since all the data files will start with this. For more information on the data preprocessing Aggregator, please see: http://docs.expyriment.org/expyriment.misc.data_preprocessing.Aggregator.html

Best,
Florian

Sarah

Dear Flad, Thanks for your reply. I am sorry but I don't why in the output csv file I only could see the *first rows od the data * ( instead of 20 rows). It's very starnge... Here is the code:

    data_folder='./data'
    file_name='nback'
    agg = data_preprocessing.Aggregator(data_folder, file_name, suffix='.xpd')

    agg.set_subject_variables(["b.name","button", "rt"])
            agg.aggregate(output_file="out.csv")



    import csv
    import codecs

    filename= "out.csv"

    data = []
    with codecs.open(filename) as f:
        for line in f:
            data.append(line.strip().split(","))    

Do you have any idea why this happens?
Best

fladd

Hi Sarah,

so you are saying you only get the aggregated data for the first subject? Are the data files of the remaining 19 subjects named in the same way (i.e. starting with nback)?

I also don't understand why you define "button" and "rt" as between-subject factors. From your last examples it seemed that they are within-subject factors. Maybe you could explain in more detail what kind of experiment you were running and what exactly you would like to achieve?

Best,
Florian

Sarah

Hi Florian,

Thank you for your email. In only run one experiment. Therefore, in the folder, there was only one file associated with one participant. In this single .xpd file, I had 20 trials. When I open the .xpd file manually, I could see the results of those 20 trials. In the next step, I would like to have all those 20 trials ( for one participant) in a .csv file and I used the code below. When I open the created .csv file I noticed I only could see the information for first trial ( only first row) so the other 19 trials are missing. Please let me know what is your thoughts: here is the code:

    from __future__ import print_function
    from expyriment.misc import data_preprocessing, constants


    data_folder='./data'
    file_name='nback'
    agg = data_preprocessing.Aggregator(data_folder, file_name, suffix='.xpd')
        agg.aggregate(output_file="out.csv")



    #import csv
    #import codecs

    filename= "out.csv"

    data = []
    with codecs.open(filename) as f:
        for line in f:
            data.append(line.strip().split(","))

Thanks

fladd

Dear Sarah,

I think there is some confusion about what the Aggregator in the data_preprocessing package does. Its purpose is to aggregate data from several participants into a single file by creating per-subject averages for all factor-level combinations, which you can then use to run a repeated-measures ANOVA on (for instance in SPSS). If there is only data from a single subject, this data will be averaged into a single line.

You further state that your goal is to create a csv file with all 20 trials for your participant. You already have this: This is exactly what the .xpd file should be, if you add the data to it after every trial (which I understood you do, given your earlier posts). So there is no need to transform the .xpd file into csv format - it already is in csv format. If you are using Windows and some other software there expects your csv files to end with .csv for some reason, you can just rename them to end with .csv, instead of .xpd, but in either case, the content of the file fulfills the csv specification: https://en.wikipedia.org/wiki/Comma-separated_values.

I hope this helps. Let me know if you have any further questions.