Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links

Categories

Supported by

Machine learning

MBDMBD
in JASP & BayesFactor

Hello there,

I ran a machine learning analysis using the k nearest neighbor classification algorithm to differentiate between two groups (patients and controls) based on six variables. The problem (but maybe it's not) is that different runs yielded different results. It's not clear to me, therefore, how to report these results. In classical statistics, I run a log regression, and the results (based on the same dataset) are consistence, but this is not the case in machine learning.

It would be great if anyone could help me with this question regarding reporting the different range of machine learning results. I need to report these results for a scientific paper, so any guidelines/or reporting rules in this regard will be helpful.

Many thanks!

Comments

  • EJEJ

    How different are the results? Most of ML is based on resampling algorithms so results are expected to differ -- but if they differ by a lot than this would be surprising to me.

    E.J.

  • MBDMBD

    The range is between 76% to 90% of correct prediction (patient vs control). I feel like it's a wide range of results. Isn't it?

  • koenderkskoenderks
    edited June 2022

    Each time you run the analysis it randomly selects a training(, validation) and test set to use, so it is expected that the results will be different across runs. You can disable this behavior of the analysis by fixing the seed in the training parameters section. This wil enable you to compare results for the same data set each time the analysis runs.

  • MBDMBD

    Thank you for your response.

    I've seen that fixed seed isn't recommended in ML. But let say that I run ML with fixed seed. Do I supposed to report the entire range of different results?

    Since different runs yield different results, it seems like I'll have endless results and my point is that I'm not sure when it's recommended to stop running the algorithm again and again & which runs' results to report? Is that okay to choose reporting the run with the best results?

    Thanks!

  • koenderkskoenderks

    I’m not sure that I follow. If you fix the seed then the results should be the same every time you run the analysis :)

  • MBDMBD

    Okay. Got it. Thank you.

  • EJEJ

    Yes, but Koen, isn't it worrying that the results would differ a lot depending on the seed?

  • koenderkskoenderks

    I would say this partially depends on whether parameters in the algorithm are optimized under "Training parameters". In the k-nearest neighbors algorithm, the optimal number of neighbors is trained on the training set and after that optimized on the validation set. Hence, this optimization is dependent on the specification of both the training set and the validation set. Because of this, the results on the test set may differ between runs of the analysis depending on the representativity of the training and validation set. The results will probably differ much less when you compare results across different training and tests sets but keep the number of neighbors fixed under "Training parameters". Let me know if this helps.

Sign In or Register to comment.
Sebastiaan Mathôt » v2.0.18 » Copyright 2011 — 2019 » Vanilla flavored » License

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games