java - How to access metadata stored by Spark Streaming custom receiver? -


spark streaming provides ability create custom receiver, detailed here. store data received receiver spark, store(data) method needs used.

the data storing spark has properties associated it. spark receiver class, extended custom receiver, provides several store methods of form store(data, metadata), imply metadata/properties can stored data. code extract below shows how used method store data , metadata/properties.

public class customreceiver extends receiver<string> {      public customreceiver() {         super(storagelevel.memory_and_disk_2());     }      @override     public void onstart() {         new thread() {             @override             public void run() {                 try {                     receive();                 } catch (ioexception e) {                     restart("error connecting: ", e);                 }             }         }.start();     }      @override     public void onstop() {         // not needed receive() method closes resources when stopped     }      private void receive() throws ioexception {         string str = getdata();         map<string, string> metadata = getmetadata();         iterator<string> = arrays.aslist(str.split("\n\r")).iterator();          store(it, metadata);          if (isstopped()) {             closeconnections();         }     } } 

this stored data accessed, class, shown in following code extract:

private void testcustomreceiver() {     javadstream<string> custom = ssc.receiverstream(new customreceiver());      javadstream<string> processedinput = custom.flatmap(row -> {         return arrays.aslist(row.split("\\r?\\n"));     });      processedinput.print(); } 

which brings question: how can metadata/properties stored data in custom receiver accessed testcustomreceiver() method shown above?

i have tried searching through documentation , exploring javadstream object in debugger search metadata, no avail. or advice on matter appreciated, thank you.


Comments

Popular posts from this blog

javascript - jQuery: Add class depending on URL in the best way -

caching - How to check if a url path exists in the service worker cache -

Redirect to a HTTPS version using .htaccess -