python - Loading numpy arrays stored in npz archive in PySpark -


i have large number of numpy arrays in s3 stored in npz archive. best way load them pyspark rdd/dataframe of numpy arrays? have tried load file using sc.wholetextfiles api. 

rdd=sc.wholetextfiles("s3://[bucket]/[folder_containing_npz_files]")

however numpy.load requires file handle. , loading file contents in memory string takes lot of memory.

you cannot memory requirements otherwise bytesio should work fine:

from io import bytesio  def extract(kv):     k, v = kv     bytesio(v) r:         f, x in np.load(r).items():             yield "{0}\t{1}".format(k, f), x  sc.binaryfiles(inputpath).flatmap(extract)

Comments

Post a Comment

Popular posts from this blog

android - net_scheduler holding wakelock -

i have observed on occasions, net_scheduler keeps acquiring wakelock repeatedly on short durations , prevents android device sleep. from logcat, not able obtain conclusive information why might happening. of gms services active. can suggest when there higher chance occur perspective of gms services. this seems coming gcmnetworkmanager , schedules network-oriented tasks, google play services able batch network operations across system. also if use more advanced scheduling firebase jobdispatcher , internally gcmnetworkmanager used.
Read more

sql - MySQL : Getting Entries from a many-to-many table -

i have following table, select records should have mentioned values. example select companyid should have 3 providerid 200,300,400. running query should retrun companyid 2. opposite of in . in advance ! companyid | providerid -------------------------------------- 1 100 2 200 3 500 4 600 2 300 2 400 7 100 do group by , select rows of 3 wanted employeeid values. use having make sure 3 different employeeid values. select companyid tablename employeeid in (200, 300, 400) group companyid having count(distinct employeeid) = 3 another approach double self join, 1 each employeeid. however, less flexible: select distinct t1.companyid (select companyid tablename employeeid = 200) t1 join (select companyid tablename employeeid = 300) t2 on t1.companyid = t2.companyid join (select companyid tablename employeeid = 400)...
Read more

java - Retrieving data from database using jsp (Hibernate + Spring + Maven) -

hello, guys. i'm trying retrieve data oracle database, nothing appears on screen. here userslist.jsp <%@page contenttype="text/html" pageencoding="utf-8"%> <!doctype html> <html> <head> <title>spring mvc hello world</title> </head> <body> <h2>all users in system</h2> <table border="1"> <tr> <th>user id</th> <th>login</th> </tr> <c:foreach items="${users}" var="user"> <tr> <td>${user.userid}</td> <td>${user.login}</td> </tr> </c:foreach> </table> </body> </html> it seems ok. , here spring controller userscontroller.java package com.controller; import com.database.users; import com.service.usersmanager; import java.util.list; impo...
Read more