c - MPI parallelization help for reading a large amount of data multiple times -


i have been asked parallelize existing c program in order decrease runtime. have (very limited) experience using basic mpi (and programming knowledge self taught, spotty). trying figure out best parallelization approach.

currently, during every iteration of main loop (m = number of iterations), program sequentially accesses set of input files (n= number of files)-each of varying length. after input files read program sorts data , updates set of output files. both n , m known @ start, , n larger m. in fact, n large read input data memory, each time files read, information pertinent main loop iteration kept.

i confident can make each main loop iteration independent, every iteration still need access n files. best way use openmpi (technically openrte 1.6.2 running on rocks- i.e. redhat linux) parallelize program?

my first idea split read-in of input files across multiple threads- each thread handling subset of files , ordering inputs @ end.

my second idea instead split main m loop across threads, better utilization of mpi. method require copies input files in every thread (to avoid reading conflicts)? if so, worried copying files may offset time gained parallelizing main loop. also, besides building test program each approach, there easier way determine method faster?

edit: file system nfs.

after reading comments went , ran few tests on code. program spends 93% of runtime reading in data. has been said seems parallelization alone may not best solution. @ point seems necessary program’s calculations , try minimize read-in requirements.

thank replies.

based on comment responses, file system being nfs mean reading files across network? can problematic if parallelize n number of files. if n large risk exceeding maximum number of open file pointers @ time, typically defined in /etc/security/limits.conf. know if shell type either csh or tcsh, if type limit @ prompt display values. sorry forget command display in bash shell. risk overloading nfs, , problems lan or wan bandwidth. if network @ 100 mbps 12 megabytes of data per second @ best. , if don't check how know not value in kilobytes per second?

if biggest cause of run time of program reading in data, there may little can it. besides nfs issue, suggest thinking in terms of how hard drive (wherever may located) commanded read each chunk/file of data. think best have 1 file pointer reading sequentially possible data disk drive, , leave how buffer data used in program. need math , figure if have enough ram. if not need increase otherwise forced rely on disk i/o killer.


Comments

Post a Comment

Popular posts from this blog

java - pagination of xlsx file to XSSFworkbook using apache POI -

right in code, reading xlsx file, xssfworkbook, , writing database. but, when size of xlsx file increases, causes outofmemory error. can not increase server size, or divide xlsx file pieces. tried loading workbook using file (instead of inputstream), didn't either. i looking way read 10k rows @ time (instead of entire file @ once) , iteratively write workbook , database. is there way apache poi? poi contains called "eventmodel" designed purpose. it's mentioned in faq : the ss eventmodel package api reading excel files without loading whole spreadsheet memory. require more knowledge on part of user, reduces memory consumption more tenfold. based on awt event model in combination sax. if need read-only access, best way it. however, may want double check first if issue somewhere else. check out this item : i think poi using memory! can do? 1 comes quite lot, reason isn't might think. so, first thing check - what's source of prob...
Read more

Unlimited choices in BASH case statement -

consider simple bash snippet: case $option in 1) image=${options[0]%.tar} ;; 2) image=${options[1]%.tar} ;; 3) image=${options[2]%.tar} ;; 4) image=${options[3]%.tar} ;; *) echo "invalid option" exit 1 esac in real script numbers goes 30. makes pretty long. can somehow specify cases variable? something this: case $option in $i) image=${options[$(($i-1))]%.tar} any pointer appreciated. you can match against multiple patterns in single clause: case $option in 1|2|3: echo "$option one, 2 or three" ;; esac if still typing you, can use simple pattern matching: case $option in # following matches # - single-digit numbers 0-9 # - two-digit numbers starting either 1 or 2 # - number 30 [0-9]|[12][0-9]|30) image=${options[$(($option-1))]%.tar} ;; *) echo "invalid option" 1>&2 exit 1 esac
Read more

apache - How do I stop my index.php being run twice for every user -

we've got webapp initial html served using apache , php. i noticed index.php being run twice every request. seems caused / effected rewrite rules in our .htaccess - rewritecond %{http_host} ^(www\.|hotels\.)ourdomain\.com$ [nc,or] rewritecond %{server_name} ^(www\.|hotels\.)ourdomain\.com$ [nc] rewriterule (.*) http://ourdomain.com/$1 [r=301] rewriterule ^hotels/([^/]+)/?\??(.*)$ ?d=$1&$2 [qsa] the final rule moving parameter url path query string. i.e. http://ourdomain.com/hotels/vegas?someparam=1 becomes http://ourdomain.com?d=vegas&someparam=1 if go directly query string version index.php run once. if use url redirected, index.php run twice (i'm checking adding error_log('end of index.php') file). so example going http://ourdomain.com/hotels/paris hits file twice http://ourdomain.com?d=paris hits once. i've seen this question , had @ blog mentioned , , can't find empty string url's (i've tried using yslow process). ...
Read more