Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

File size in JASP – What determines the file size of analyses?

Dear JASP community,

I kindly ask, what determines the file size of JASP files when having run and saved analyses? I have managed to reach a file size of 1.15 GB with a relatively small dataset (10 rows, 22 variables).

The analyses contain:

2 Bayesian linear regressions (sampling method BAS default), 2 frequentist linear regressions, 2 PCAs, 2 EFAs, 2 Bayesian correlations, 1 Bayesian paired-samples t-test (student), 2 descriptive statistics, and 1 Bayesian single-test reliability (MCMC sampling, 500 burn-in, 50,000 iterations, thinning 1, no. chains 3, rhat, and traceplots). The method listed at the end, MCMC sampling makes always GBs of file size under these settings, especially the traceplot seems to be take much space when trying to depicting it in another software.

Does anyone know why MCMC samples of 500 burn-in and 500,000 create at least 1 GB or more GBs of data file in JASP whereas they are that easy to handle and safe in R where they are just of a few KBs? It is just an assumption that it is this procedure since it is congruent with other GB-heavy data files that contain the same procedure. Loading and opening the file in JASP also requires significantly more time than all other files that are in the area of a few MB or even KB range. I would be that happy about an answer!

Best regards,

GoldenRetriever

Comments

  • Hello GoldenRetriever,

    This is more of a question for the JASP GitHub page, I think. However, I will forward it to the team. My guess is that JASP saves all of the MCMC samples -- these are needed to plot the figures. It does seem wasteful to retain this information, so perhaps we could ask the user whether he/she wants to save this information.

    Cheers,

    E.J.

  • Hey GoldenRetriever,


    This sounds like something we should look at!

    It would be very helpful if you could open an issue here: https://github.com/jasp-stats/jasp-issues/issues

    If you could upload the jasp-file somewhere and link it to me (either via a PM here or better yet: through an issue you make at the above link) then I can check out what went wrong (because it should really not be that big!)

  • Ah it has been brought to my attention that in fact this issue has been known to us for a while at: https://github.com/jasp-stats/jasp-issues/issues/999


    And this will be fixed in 0.15, I've made sure a message is shown in the 0.14 versions of JASP to notify peopl where it matters (in the analysis itself).

  • Ah well, we are definitely aware of the problem and are working on it. Apparently 0.15 could be optimistic, but if it is we solve it in 0.15.1


    What would help us out quite a lot is if you could send us this file.

  • edited March 27

    I am glad that the issue is under work and hopefully solved soon. Those are research data and are not shared due to other circumstances. The issue is well-described and should be easily re-producible with any other datasets, even with randomly generated data. I do not see a need for requesting the specific dataset because it is a general issue and not specific to a dataset.

    I agree with EJ comment in subsequent. When exporting/copying the plot file most text editors struggle with displaying and loading the plot. It seems each part of the plot is generated which is why also the image file has a file size that is almost not possible to work with yet. I disagree that it is the same issue as cited. I have had a dataset of only 10 rows and even with 10 rows JASP struggles in MCMC modeling as described. The issue 999 differs in description from this here. It is clearly an issue that involves MCMC samples of 500 burn-in and 500,000 iterations independently from rows of the file. Maybe even other MCMC procedures in JASP suffer from that.

  • Well this more detailed description of what you did should also suffice, thank you for getting back to me.

    I'll add this info to our issue so that this can be tested and if we have taken the unnecessary data out or not.

  • Dear @GoldenRetriever,

    I tried to recreate the issue you re describing, and you are right, the jasp files does get way too big. We have managed to alter the code so that the filesize would be reduced quite a bit, however, with 50,000 posterior samples the file is still around 200Mb (for the case you described). We are also working on adding a button that would allow a user to choose not to store some of the posterior samples in the file. This will, however, lead to a reduced computation speed. These changes will then be available in 0.15.

    Regarding the issue you were describing about the plots. I have tried to recreate what you describe, but have not managed to find the problem. Can you go into a bit more detail? Are you referring to the traceplots? Do you obtain that plot (outside of jasp) by exporting the plot directly into a file or do you obtain the plot by unzipping the .jasp file and accessing the plot? What kind of file is that plot then? For example, is it a .png file for the omega traceplot?

    Thanks for your help,

    Julius

  • edited March 31

    Quick update, I have found the issue with the size of the plots. No need for you to explain further. However, it seems as though the size of the plots is something we cannot further reduce. Having implemented the button to avoid saving any samples I managed to reduce the size of the .jasp file from 1.5gb to 40mb. This might be the best we can do at the moment.

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games