tag:blogger.com,1999:blog-202791059041586496.post1213422709374643191..comments2023-06-21T02:48:18.822-07:00Comments on nevatiaritika's Blog: Why Apache Spark and what is parallel processing?nevatiaritikahttp://www.blogger.com/profile/16655501873541408940noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-202791059041586496.post-46557606681238281992018-01-17T01:29:53.129-08:002018-01-17T01:29:53.129-08:00Thanks for the article. Data Scientists have the p...Thanks for the article. Data Scientists have the potential to earn about 19 lakhs a year in India and are in demand throughout the world.Leveluphttps://www.level-up.onenoreply@blogger.comtag:blogger.com,1999:blog-202791059041586496.post-77989802530233384942017-10-31T03:01:28.258-07:002017-10-31T03:01:28.258-07:00Thank you! While we did not observe a slowdown, th...Thank you! While we did not observe a slowdown, the performance did become consistent after a certain threshold. Unfortunately I didn't observe a formulaic trend as such - this was more dependant on the kind of operation that was being performed in the code!nevatiaritikahttps://www.blogger.com/profile/16655501873541408940noreply@blogger.comtag:blogger.com,1999:blog-202791059041586496.post-39629164872668466732017-09-27T09:07:32.710-07:002017-09-27T09:07:32.710-07:00Good post. Yes parallel processing does speed thin...Good post. Yes parallel processing does speed things up. But after a certain number of threads, we observed that it actually slows things down as the threads keep waiting on each others to acquire locks. In general i have been able to have Threadcount = 1.5 * Number of cores work as optimal. Would like to know how many parallel threads/processes/jobs work for you. Sankethttps://www.blogger.com/profile/02052624698604965676noreply@blogger.com