Scala concurrency performace issues -


i have data mining app.

there 1 mining actor receives , processes json containing 1000 objects. put list , foreach, log data sending 1 logger actor logs data many files.

processing list sequentially, app uses 700mb , takes ~15 seconds of 20% cpu power process (4 core cpu). when parallelize list, app uses 2gb , ~ same amount of time , cpu process.

my questions are:

  1. since parallelized list , computation, shouldn't compute-time decrease? think having 1 logger actor bottleneck in case. computation may faster bottleneck hides speed increase. if add more loggers pool, app time should decrease?

  2. why memory usage jump 2gb? jvm have store entire collection in memory parallelize it? , after computation done, jvm garbage collector should deal it?

without more details, answer guess. however, guess might point right direction.

  1. parallelized execution should decrease running time problem might lie elsewhere. reason, cpu idling lot in single-threaded mode. not specify whether read input disk or network or write output to. explicitly write logs lot of files. disk , network reading/writing might in case take longer data processing. process idle due i/o waiting. should not expect speedups parallelizing job spends 80% of time waiting on i/o. therefore suspect loggers not bottleneck here.
  2. the memory usage might jump if threads allocate lot of memory each. in case, more threads have more memory required. don't know kind of collection parallelizing on, stored in memory, completely. yes, garbage collector free resources not require explicitly free them, such files.

Comments

Post a Comment

Popular posts from this blog

javascript - jQuery: Add class depending on URL in the best way -

i have html progress meter: <div class="col-md-3" style="margin-left: -20px;"> <div class="progress-pos active" id="progess-1"> <div class="progress-pos-inner"> login </div> </div> </div> <div class="col-md-3 progress-pos-container"> <div class="progress-pos" id="progess-2"> <div class="progress-pos-inner"> scan items </div> </div> </div> <div class="col-md-3 progress-pos-container"> <div class="progress-pos" id="progess-3"> <div class="progress-pos-inner"> confirm address </div> </div> </div> <div class="col-md-3 progress-pos-container"> <div class="progress-pos" id="progess-4"> <div class="progress-pos-in
Read more

caching - How to check if a url path exists in the service worker cache -

i need check if particular url path exists in service worker cache. example, suppose url is: /myserviceworker/service?a=110&b=70 this url exists in cache, there many of them different values of , b. now, suppose want refresh of these urls, how can that? i want know how access key values service worker cache. if know key, plan follows: var url = new url(key); if(url.pathname === "\/myserviceworker/service") refetch key but not sure how cache key , in format is. mean, string or url? cache api has match() method returns promise resolving in response object if match or undefined if no match exists. second parameter object can specify ignoresearch not take account url parameters. the ignoresearch option supported firefox ( chrome status here ). in other hand, retrieve cache entries, can use keys() method.
Read more

Redirect to a HTTPS version using .htaccess -

i need modify .htaccess file redirect urls https version without "www". http://example.com --> https://example.com http://www.example.com --> https://example.com https://www.example.com --> https://example.com this how .htaccess file looks like: rewriteengine on rewritecond %{request_filename} !-d rewritecond %{request_filename} !-f rewriterule ^(.*)$ redirect.php?id=$1&page=$2 [qsa,l] what modifications need make .htaccess in order forward https without "www"? it depends on doing current code. looks file called redirect trying display php page seo friendly url. think might confusing people. anyway, need , force https without www, need 1 more rule above. also second rule needs two capture groups in rewriterule test string because wanting 2 different values using $1 , $2 references. rewriteengine on #rediect www non www and/or http https --- combinations. rewritecond %{http_host} !^example\.com [nc,or] rewritecond %{https} !^
Read more