Participant data visible in individual result but missing from bulk CSV export
Hi,
I am running a behavioural experiment on a JATOS server, with the experiment implemented in PsychoPy/PsychoJS.
For each participant, I generated a unique JATOS link. When generating each link, I entered the participant's participant code and name in the Comment field so that I could identify which result belonged to which participant.
For most participants, the data appears normally in the bulk results export. However, for some participants who have definitely completed the task, their results are not appearing in the combined/bulk exported data.
I have checked the individual results on the JATOS server, and the situation is interesting:
- The individual result is present on the JATOS server.
- When I select the individual result and download/extract the result as a TXT file, the task data is clearly present in the TXT file.
- However, when I convert this TXT data to CSV, the resulting CSV does not contain the expected data.
- I have also noticed that some fields/variables may be missing or formatted differently in these problematic files, although the actual trial data appears to be present.
- Additionally, the Participant Code that was entered in the JATOS link/comment is not appearing in the final exported raw dataset, making it difficult to associate these results with participants.
I would therefore like to understand:
- Is the Comment field associated with each generated JATOS link stored in the result metadata, and is there a way to include/export this Comment field together with the corresponding result data?
- Why might a result that is visible and contains data when downloaded individually as TXT not appear correctly in the bulk CSV export?
- Is there a recommended way to recover/export these individual results while preserving all the trial-level data and metadata?
- Could differences in the structure or data types of individual
data.txtfiles cause the CSV export/conversion to omit the data? - Is there a recommended JATOS export format or procedure for recovering these results without losing any trial data?
I am particularly concerned because the individual result files appear to contain the data, so I would like to make sure that the data are not being lost simply during the export/conversion process.
I can provide a sample of the problematic .txt result file and/or the JATOS results export if that would help diagnose the issue.
Thank you!
Comments
Hi,
Yes, it would be helpful if you could include the JATOS results export. But it sounds like this is the expected behaviour, and that there is an easy solution
The participant code that you entered in JATOS (when creating the link) is something that happens in the backend. It will not be automatically included in the data that the experiment creates and submits to the server (this is the result data, what you see on the .csv). (This is important: JATOS will not modify your data unless you tell it to do so in the front end, using functions from the jatos.js library).
So you are trying to merge metadata into your result data. What I tend to do is write a simple script (e.g. in R or python) that reads in both the metadata and the result data and merges them together.
Hope this helps
Elisa
If I understand your issue correctly, some completed study results are missing or empty in the bulk CSV export, even though the corresponding individual results exist in JATOS and their downloaded
data.txtfiles contain the trial data. Can you give us an example of the data that are missing in CSV but exists in the data.txt? How do you convert from TXT -> CSV?Best,
K.
@kri I have been converting from txt to csv in opensesame. in the results column, these participants' data show data_reterived but not finished. whereas some data_reterived data files do show up on the final csv and txt.
@elisa if i may ask, which file do you run the code on, and what do you specify exactly in the code? if you could share your extraction procedure, that would be great. Thank you so much.
I'd like to see an example of the data that are visible in JATOS but not in OpenSesame after converting them to CSV. Since the status is "data_retrieved" it just might that the study never fully finished and somehow the CSV converter can't deal with unfinished data entries.
But there are other data entries corresponding to links that are showing data retrieved as well. About showing you the data, how do i do that? should i simple attach it in reply here? sorry I am new to the forum so...
You can message me directly (click on my name).
Hi,
Thanks for the description and sample files. A few details would help clarify where the problem occurs:
opensesame_codename: "osweb"andopensesame_version: "2.1.0".The supplied TXT is valid JSON and contains 43 logged records, all with
Participant_Code: "C025". However, the attached XLSX belongs to a different study run: the TXT identifies JATOS result 1968, while the XLSX identifies result 2001. Could you provide the converted CSV for the same result as the TXT?One plausible cause is the JSON-to-CSV conversion. The records contain different sets of variables, with additional fields appearing later in the experiment. A converter that determines its columns only from the first record could omit those later fields. Also, a bulk TXT export can contain multiple JSON documents separated by line breaks, which requires a converter that understands that format.
JATOS exports the submitted result data as text; it does not perform this trial-data JSON-to-CSV conversion. If the affected participant’s data is present in the bulk TXT but absent from the converted CSV, that would point to the conversion step.
The link Comment is a separate matter: for Personal study links, it is included in JATOS result metadata, rather than automatically added to the experiment’s trial data. I recommend exporting a JATOS Results Archive, which preserves the raw data and metadata together. The comment can then be associated with the trial data using the JATOS result ID.
Best,
K