Newbie problems with JASP
I'd like to report the difficulties I had (and am having) getting JASP up and running, in a collaboration with two other distant persons. I'm an old, old-school, British statistician, and a European mathematical statistician. (Those are two different persona.) I'm working with a French pure mathematician from algebraic geometry, and a young Chinese bachelor student of mathematics. We are doing Covid-19 analyses; in fact, we are looking at the very controversial recent results from Marseilles on Hydroxycholroquin and Azitromycin. We are talking with the French researchers themselves. See https://doi.org/10.1016/j.ijantimicag.2020.105949 for the starting point.
Warning - this posting will degenerate into various rants. But there is also some real content. JASP looks great and I want to use it and I want to get my friends to use it. But I ran into a lot of hurdles. And moments of despair.
To begin with I had repeated problems with bugs in the Mac version of JASP. Moreover, it took aeons to download. The Windows version also took an aeon to download. Haven't tried burning my finger with Linux yet (I have a pretty good MacBook Pro with Parallels Desktop virtual Windows and Linux machines).
Then I wanted to get four numbers into JASP in order to compare the parameters "p" of one independent observation each of two binomial distributions each with known "n". I would imagine that this is one of the most basic and common things someone might do. A true disciple of Fisher would use Fisher's exact test, though a more liberal minded frequentist might well prefer Barnard's. Apparently, one has to go to the "frequencies" menu to do this, and apparently it is nowadays called an "A/B test" by woke young millenials who think that statistics is dead and the in thing is now machine learning. I don't know if the JASP interface would be easier to young psychology students who have been brainwashed and made brain dead by using SPSS (click and drool), and I have often seen the disastrous result of that.
I couldn't find any decent written text explaining step by step what to do. I don't want to have a live Zoom demonstration nor do I want to watch a YouTube video.
Oh well. Finally I was up and running though I fear the upcoming ordeal when I try to explain what I have learnt (anb explain what is actually going on behind the scenes) to my French and Chinese friends who are very interested in the controversial Marseilles treatment but not exactly statistical wizards... Yet doctors, and algebraic geometers, and young Bachelor students, are precisely the sort of people who in the past used SPSS and in the future will use JASP. Please don't repeat the mistakes which are pretty much hard-wired into SPSS.
Finally some complaints. It was explained to me that the philosophy of JAGS was first test, and then estimate conditional on the results of tests. I think that that often leads to bad science though it can be useful in exploratory work. That is what exploratory work is, all the time! So, OK. Still: I wanted to start with a spike and slab prior, and then still see the remnants of the spike in the posterior; when I ask for the posterior probability that the log odds ratio is between 1 and 4, I don't want to condition on "no effect at all". I think that at present the annotation of the output is misleading if not actually wrong. I'd like,for forensic statistical applications,to use log base 10 (forensic evidence is measured in units of "bans" - Turing, Banbury, ...) and for information theoretic applications log base 2. I think that so-called "natural logarithms" are very unnatural. Why they are foisted on psychologists, I have no idea. I suppose to make them think (or rather, make their clients think) that they are doing real science? But real scientists in many real scientific fields don't use "natural" logarithms!
The fancy heatmap is fancy but I did not see a good reference to a good description of what it actually is, what it does, what it means.
There are some inconsistencies in notation. Treatment A/B, group 1 vs group 2, BF01 or BF10??? These tiny little things can really confuse a beginner. OK, so at the end you need and want all those notations and terminologies because they are all common and all useful and all meaningful in their own contexts.
Comments
Hi Gill1109,
Thanks for giving JASP a go, and thanks for reporting some issues. I'll deal with these one by one below:
I can ask the programmers what's up with that. The install file is about 350 MB but maybe there was a glitch on the site (I does download very quickly for me).
2. "Then I wanted to get four numbers into JASP in order to compare the parameters "p" of one independent observation each of two binomial distributions each with known "n". I would imagine that this is one of the most basic and common things someone might do. A true disciple of Fisher would use Fisher's exact test, though a more liberal minded frequentist might well prefer Barnard's. Apparently, one has to go to the "frequencies" menu to do this, and apparently it is nowadays called an "A/B test" by woke young millenials who think that statistics is dead and the in thing is now machine learning. I don't know if the JASP interface would be easier to young psychology students who have been brainwashed and made brain dead by using SPSS (click and drool), and I have often seen the disastrous result of that."
Ha! Well, if you want to do the frequentist analysis you'd go to "Frequencies" and then select "Contingency tables". A special case of the contingency table is the 2x2 one where two proportions are compared, and we recently implemented a separate Bayesian analysis for that one (the "A/B test"). We could have called the A/B test "comparison of two proportions" I guess. The reason why it gets a separate menu is because (a) it occurs so often (b) our priors are different than the one under the contingency table menu -- we assume dependent proportions, and we do not assign each p an independent beta prior. Now I do agree this organisation is not optimal. We'll have to do something about that in the future.
3. "I couldn't find any decent written text explaining step by step what to do. I don't want to have a live Zoom demonstration nor do I want to watch a YouTube video."
OK. Well, a list of resources is here: https://jasp-stats.org/jasp-materials/ where you can also find this old-fashioned manual (currently, frequentist only: http://static.jasp-stats.org/Manuals/Statistical_Analysis_in_JASP_-_A_Students_Guide_v0.12.pdf). For details on the Bayesian A/B test see https://arxiv.org/abs/1905.02068.
4. "Oh well. Finally I was up and running though I fear the upcoming ordeal when I try to explain what I have learnt (anb explain what is actually going on behind the scenes) to my French and Chinese friends who are very interested in the controversial Marseilles treatment but not exactly statistical wizards... Yet doctors, and algebraic geometers, and young Bachelor students, are precisely the sort of people who in the past used SPSS and in the future will use JASP. Please don't repeat the mistakes which are pretty much hard-wired into SPSS."
There's always a balancing act, and we have to catch up on the documentation/books. As far as examples of Bayesian A/B tests are concerned, see the following papers:
and, of particular relevance:
5. "Finally some complaints."
:-) What, we are done with the compliments already? ;-)
"It was explained to me that the philosophy of JAGS was first test, and then estimate conditional on the results of tests. I think that that often leads to though it can be useful in exploratory work. That is what exploratory work is, all the time! So, OK. Still: I wanted to start with a spike and slab prior, and then still see the remnants of the spike in the posterior; when I ask for the posterior probability that the log odds ratio is between 1 and 4, I don't want to condition on "no effect at all". I think that at present the annotation of the output is if not actually ."
We intend to revamp the Bayesian analyses along these lines. What you suggest is a useful addition.
"I'd like,for forensic statistical applications,to use log base 10 (forensic evidence is measured in units of "bans" - Turing, Banbury, ...) and for information theoretic applications log base 2. I think that so-called "natural logarithms" are very unnatural. Why they are foisted on psychologists, I have no idea. I suppose to make them think (or rather, make their clients think) that they are doing real science? But real scientists in many real scientific fields don't use "natural" logarithms!"
The reason we included the log at all is because it makes the scale symmetric: a log BF of -x for H0 means a log BF of x for H1. We did not think of the base. And I am not sure very many people use the log scale at all, to be honest. We'll see what we can do (maybe a setting in preferences to set the base).
6. "The fancy heatmap is fancy but I did not see a good reference to a good description of what it actually is, what it does, what it means."
Good point. If you take a look at the two one-page papers above you see a rough description. We'll make sure that the revision of our paper on ArXiv will include a description. Basically, it is a sensitivity analysis where the BF results of a N(mu,sigma) prior on the log odds ratio is given for a user-specified range of mu and sigma.
7. "There are some inconsistencies in notation. Treatment A/B, group 1 vs group 2, BF01 or BF10??? These tiny little things can really confuse a beginner. OK, so at the end you need and want all those notations and terminologies because they are all common and all useful and all meaningful in their own contexts."
Yeah, so we have BF01 and BF10 throughout JASP. We could indeed use "group A" and "group B", that would be more clear. I'll take this up with the team.
Thanks again for your feedback!
Cheers,
E.J.
Actually, I just archived two analyses that show the robustness heatmap -- I realized that the ones I gave you may not show the heatmap, as this is a feature we added recently (which is also why it is not yet discussed in the paper on ArXiv). Anyway, they are available at
and
Cheers,
E.J.