java - How to access metadata stored by Spark Streaming custom receiver? -
spark streaming provides ability create custom receiver, detailed here. store data received receiver spark, store(data)
method needs used.
the data storing spark has properties associated it. spark receiver class, extended custom receiver, provides several store methods of form store(data, metadata)
, imply metadata/properties can stored data. code extract below shows how used method store data , metadata/properties.
public class customreceiver extends receiver<string> { public customreceiver() { super(storagelevel.memory_and_disk_2()); } @override public void onstart() { new thread() { @override public void run() { try { receive(); } catch (ioexception e) { restart("error connecting: ", e); } } }.start(); } @override public void onstop() { // not needed receive() method closes resources when stopped } private void receive() throws ioexception { string str = getdata(); map<string, string> metadata = getmetadata(); iterator<string> = arrays.aslist(str.split("\n\r")).iterator(); store(it, metadata); if (isstopped()) { closeconnections(); } } }
this stored data accessed, class, shown in following code extract:
private void testcustomreceiver() { javadstream<string> custom = ssc.receiverstream(new customreceiver()); javadstream<string> processedinput = custom.flatmap(row -> { return arrays.aslist(row.split("\\r?\\n")); }); processedinput.print(); }
which brings question: how can metadata/properties stored data in custom receiver accessed testcustomreceiver()
method shown above?
i have tried searching through documentation , exploring javadstream
object in debugger search metadata, no avail. or advice on matter appreciated, thank you.
Comments
Post a Comment