Results for one component not stored - and dataHash question

[Deleted User]

Hello!

After several runs of my study, I found that one participant did not have data for a single component within their study. From the application logs, I am certain they accessed that component and spent time on it. After further investigation into the study logs, I found the following for this participant:

The component that did not have data should have appeared on line 4 in the image above. Rather, the UUID on that line corresponds to the component that follows the missing component (for which we do have full data). Moreover, line 5 also corresponds to this following component (so it appears twice here despite only being completed once), and the 'dataHash' is now present. I am not sure how the dataHash should typically appear, but here is what a participant with complete data looks like in my study logs:

My questions:

• From this, is there any indication as to why we may have not collected data for that component?
• Why does 'dataHash' typically show 'no data', and in what context is the hash present?

Thanks!

kri

Hi,

The 'no data' is written in the Study Log (it's not the application log - the application.log logs admin related stuff while the Study Log logs study run related stuff) when one of the result sending functions is used (e.g. jatos.submitResultData or jatos.appendResultData) but there are no actual data sent. Why there is no data being sent I can't say without seeing the code. But it is strange that in your Study Log is mostly 'no data' - you should have mostly empty result data.

The 'dataHash' field is a Sha-256 hash of the result data. It can be use to prove that the data were not tempered with afterwards.

Best,

Kristian

[Deleted User]

Hi Kristian,

Looking at all my old studies, the dataHash is always 'no data'- but we consistently use 'jatos.submitResultData' and have data fully collected. As an FYI, we use MySQL rather than H2- not sure if that makes a difference.

Do you know why this might be the case? And why in this case, with this glitch, there would be a dataHash? This is the only case of a dataHash in all of our studies going back to April, for all of which we have complete data and use ''jatos.submitResultData'.

kri

MySQL or H2 - there should be no difference in the 'dataHash'. Strange. Which version of JATOS are you using?

[Deleted User]

We're using 3.7.3.

kri

Is it possible that you call the 'jatos.submitResultData' more than once in each component (the second time with empty data)? Can you please post the code here where you use the function?

[Deleted User]

I don't believe so- we use jsPsych, and the way we submit data is the same for every single component- one line within the jsPsych 'on_finish' function, which only executes at the very end of the component. Here is the code:

kri

Agreed, 'jatos.submitResultData' only once. Strange. I checked JATOS source code and this should not happen. Maybe jsPsych does something unexpected. Would it be possible to send me your study? Per email (lange.kristian@gmail.com) would be fine. Or in this forum per direct message.

Best,

Kristian

[Deleted User]

Sent to your email earlier today- many thanks, Kristian.

kri

Hi,

I looked again at the study logger source code and found a bug there. This bug seems to have been introduced in version 3.7.2. This bug causes the 'no data' entry in the dataHash field although result data were present. Actually it is a little bit more complicated: The dataHash field is written, but only with the hash of the previously submitted result data. E.g. for the first calling of jatos.submitResultData (or jatos.appendResultData) with data A 'no data' is written in the study log, then on the second calling with data B the hash of A is written, then on the third calling with data C the hash of B is written ... and so on.

That means for your study you always get 'no data'. Except in this one case were there seem to have been two calls of jatos.submitResultData or jatos.appendResultData roughly 11 seconds apart. From your code I can't see why this have happened. Is it possible that your participant just reloaded the component?

But more importantly (at least for JATOS) I'll fix the bug in the study logger in the next release. Thank you for pointing this out!

Best,

Kristian

[Deleted User]

Hi Kristian,

Many thanks for looking into this a few months ago. I wanted to message that I've noticed the initial issue I pointed out above several more times, in different studies and different components.

Essentially, despite evidence that someone accessed a component and spent 10+ minutes on it, no data is written to JATOS. Moreover, what I've noticed, as was the case in my initial post above, is that this data writing error always seems to happen after several calls of 'Stored component result data' for a previous component (although not always the immediately preceding component); which, based on your previous comment, is possibly due to a reload of that component. What's also strange, however, is that we don't allow reloading of components, we use General Single Worker, and we should see a reload listed in the JATOS GUI for that subject, which we don't.

As an added note, I found a few other instances of multiple unexpected calls to store data for a component, without any data writing issues for that subject. However, in the converse direction, every time we have these data writing issues we see these unexpected calls.

I'm not sure if these two things are connected: the multiple calls to store data for a previous component, and the lack of data being written for a current component. Perhaps once you push the bug fix you mentioned, this may address this problem?

For additional context, we use MySQL to store data due to issues with large H2 backup logs (see this previous post). We actually weren't having these data writing issues with H2, but our switchover also coincided with a JATOS update on our end, so I'm not sure the ultimate source of the issues.

Thanks!

kri

Hi,

After a couple of weeks gone I need to refresh my memory ;)

• Your participants run your study but sometimes a component is missing result data (as can be seen in JATOS GUI / result page).
• Now you found the same issue in other studies.
• We found a bug in JATOS' study log that caused 'no data' entries and other entries with wrong hashes. This bug was introduced in 3.7.2.
• I assumed that participants might have reloaded the component.

I hope I've got this right. Now to what you write this time (by the way, sorry for not answering earlier - I was busy with birthday preparations for my son over the weekend).

What's also strange, however, is that we don't allow reloading of components, we use General Single Worker, and we should see a reload listed in the JATOS GUI for that subject, which we don't.

Are you sure you don't allow reloading components? A General Single worker does allow reloads of components - it does not allow running the study more than once. If you want to prevent component reloading you have to uncheck the 'Allow reload' box in the component properties. And you can identify if a component was reloaded in the result page: a reloaded component has the state 'RELOADED' (see screenshot, first one 'RELOADED', second 'FINISHED').

As an added note, I found a few other instances of multiple unexpected calls to store data for a component, without any data writing issues for that subject. However, in the converse direction, every time we have these data writing issues we see these unexpected calls.

When you say 'unexpected calls', do you see them in the study log or application.log?

I'm not sure if these two things are connected: the multiple calls to store data for a previous component, and the lack of data being written for a current component. Perhaps once you push the bug fix you mentioned, this may address this problem?

Unfortunately not, the bugfix will only fix the study log issue. I haven't found any bug in result data writing yet and still hope that this is not a JATOS bug (fingers crossed).

For additional context, we use MySQL to store data due to issues with large H2 backup logs (see this previous post). We actually weren't having these data writing issues with H2, but our switchover also coincided with a JATOS update on our end, so I'm not sure the ultimate source of the issues.

H2 and MySQL should behave equally. I'll assume for now that this is not the cause.

Best,

Kristian

[Deleted User]

Hi Kristian,

Thanks for the extensive reply! No immediate rush in replying.

After a couple of weeks gone I need to refresh my memory ;)

Your participants run your study but sometimes a component is missing result data (as can be seen in JATOS GUI / result page).

Now you found the same issue in other studies.

We found a bug in JATOS' study log that caused 'no data' entries and other entries with wrong hashes. This bug was introduced in 3.7.2.

I assumed that participants might have reloaded the component.

All of this is correct!

When you say 'unexpected calls', do you see them in the study log or application.log?

I am referring to the evidence in the study log that data was written multiple times for the same component (as seen in that first screenshot in my first post). I call it an 'unexpected' call because we see no evidence of it in our GUI (no "RELOADED" state).

Are you sure you don't allow reloading components?

We actually do allow reloading on some components, but not others. But I can confirm that the component that apparently reloaded in my first post does NOT allow reload.

That said, in the two most recent cases of this error, the reloaded component does happen to be reloadable. But, we nevertheless see no evidence of this in the GUI. See below:

The repeated component in the study logs ("2c038d91-41c1-4df9-be37-899cafc108e6") is Component 50 ("Consent"). This actually happens to be a reloadable component, but there is no evidence for reloading in the GUI.

The component that has no data ("Phase 2") is not present in the study logs. For reference, here is what a complete subject data looks like in the study logs (this is reposted from my original post above):

Also, if it helps, here is the application log for the problematic study run:

From this, it does appear that Component 50 has two submitOrAppendResultData calls. Component 6 is the component with no data ("Phase 2"), and it has no submitOrAppendResultData calls. So this aligns with the study log. Nevertheless, you verified in our previous email exchange that my study code should be correctly submitting data.

All of this said, it's not clear to me that the unexpected data submission calls and the missing component data are connected. Could these be independent issues?