Problem with JATOS result conversion

michedini

Hello,

I am trying to use OpenSesame to convert the results exported from JATOS, but the converted file turns out to be empty (both in .csv and .xlsx format).

For completeness, I am trying to convert results from online experiments run on JATOS, but which I did not create using OpenSesame (I used lab.js in this particular case). Could that be the issue?

Any help would be greatly appreciated.

Thanks

elisa

Hi, yes. The OS converter probably expects results in a very specific format.

What do your exported results look like? Depending on the software you use to analyze data, maybe the best is to use an R package (like jsonlite) to read your data in

michedini

Thank you for your help, Elisa.

So far, I have run just a couple of trials to test data export and conversion.

If I export the single trial from JATOS, I am able to access it without issues using python's pandas.read_json function. However, when I looked the file which I get when I click "export all", it seems to me that each trial is reported sequentially, with a newline as separator. When I tried opening it in python as described above, I got "ValueError: Trailing data".

I am unfamiliar with R, so I was trying to look for alternatives. Since I plan to collect a lot of trials per participant, I was hoping to analyse the data using python, and processing each raw output individually would be unfeasible.

michedini

jatos_results_20221111092210.txt

I have attached the output file I am currently having trouble with, maybe it can help you find the issue more easily.

It contains the results of 3 trials.

elisa

Hi, I have to run and can look at this later but yes, the problem is most likely that you have the following:

]

[{"url":{},"meta":{"labjs_version":"20.2.4","labjs_build":

etc. That's not a valid json.

In the evening I can send you a solution, now I have to run

elisa

Hi again,

Sorry for the chaotic reply sooner :)

I don't use python, I use R. However, I was wrong: the format where each line is a valid JSON is apparently called jsonlines, and there's a library for that in Python. Have you tried this already?

https://jsonlines.readthedocs.io/en/latest/

If that doesn't work, what I've done in the past in R is to turn the separate lines into a single json string, like this

data <- readtext(here("jatos_results_20221111092210.txt"))
#This is wrong and reproduces your error
fromJSON(data$text)

data_better <- str_replace(data$text, fixed("}]\n[{"), "},{")   

In your case that doesn't do it, not sure why. I'm sure I/you can figure it out, let me know if that python library doesn't help

Best

michedini

Thank you, I was able to load the JSON file as pandas dataframes using the jsonlines package as you suggested. Here is the code I used:

import jsonlines
import pandas as pd

json_filename = #name of json file (as a string)

with jsonlines.open(json_filename) as reader:
    for line in reader:
        display(pd.DataFrame(line))

This allows you to display all individual JSON outputs as tables. I then came up with a quick solution to concatenate all files in a single dataframe using the following code.

import jsonlines
import pandas as pd

i = 0

with jsonlines.open(json_filename) as reader:
    for line in reader:
        if i == 0:
            df = pd.DataFrame(line)
            i += 1
        else:
            df = pd.concat([df, pd.DataFrame(line)])
            i += 1

display(df)

I am sure someone will come up with a better solution for this, but for now this works well enough for me. Thanks again for your help!

Fab

Hi there,

I'm following up on this to report a similar problem. We recently updated out server to Jatos 3.9.3 to run an experiment programmed with OSWeb. The data are registered by Jatos but when I export them and try to convert it with Open Sesame, the Excel file is empty.

I tried the same task on the Mindprobe server and that worked out.

In both cases the json file doesn't seem well structured (it ends with a coma and the "]" is missing). Yet for some reason Open Sesame can convert it when is it exported from the Mindprobe server but not from ours (our the settings are the default ones).

Does anyone have any idea about what may be happening?

Thanks in advance!

Best,

Fabrice.

kri

Hi Fabrice!

I guess you export the result data as an text file and your data are in JSON format. JATOS just stores the result data in its database as text without caring for the format (e.g. JSON). It's weird that it works on Mindprobe but not on your server although both have the same version. But anyway JATOS can't really help here since the JSON format is determined somewhere in your study or maybe even in OpenSesame.

Best,

Kristian

Fab

Hi Kristian, @kri (message updated, see below)

Open Sesame doesn't let me chose any particular data format. I just use the default logger, and the data stored in JATOS do look like JSON. In the past, back when I was using Jatos 3.6.1, I'd export the data (as a text file), use Open Sesame's converter and I'd get my data in Excel. Now it's no longer the case on our server but it works just fine through Mindprobe. So, it can't be the task or Open Sesame. I'm attaching the output from the same task run on our own server and on Mindprobe.

Jatos output.zip

I'm really puzzled. I haven't been able to determined yet whether there is something in these files that is structured differently. I'll probably program a basic and simpler task just for the sake of testing it.

After Open Sesame failed to process the file outputted by Jatos on own server (but did just fine with that from Mindprobe), I asked ChatGPT to convert it. It complained that the file was not properly structured (basically it ends with a coma and no bracket-), but that is also true of the file generated by Jatos on Mindprobe, so that can't be the issue. ChatGPT produced an empty Excel too, I pointed that out, it tried again and parsed manually using some Python code and then eventually outputted the data in Excel format. I checked the data in that Excel sheet and all the data was there (no missing variable or trial).

I also tried exporting data I collected two years ago, using an earlier version of Open Sesame and Jatos 3.6.1, and this too fails.

Very strange... I don't think the issue comes from Open Sesame (I uploaded the same task to our own Jatos server and to Mindprobe and used the same Open Sesame program to read try converting the output from both). I'm left wondering whether there is some configuration element somewhere that's not right on our Jatos server, but we did the default installation, didn't modify the configuration.

WIll keep investigating...

Best,

Fabrice.

Update:

I've programmed a very basic task in Open Sesame, exported it as a JATOS experiment, ran it on our server and on MindProbe, downloaded the results file (as text), converted it into an Excel file through Open Sesame and... both worked! 🤔 So there must be something about the interaction of my initial task and how Jatos runs on our server (because my initial task produced a results file that I could convert without any issue). Yet I really can't think of what that might be and it doesn't seem to very very specific to that task since exporting the data from a task I set up two years ago and that was a lot simpler fails too.