java - How to handle multi-lines logs with spark streaming? -


i'm struggling find way parse multi-lines logs using spark streaming. created parser take array of string input parameter. when multi-line stacktrace found, loops on each line until reaches 'normal' line before processing it.

the logs injected via flume on kafka topic , received via kafkautils.createdirectstream.

when comes spark streaming, stacktraces might cut in middle on 2 (or more) distributed rdds. i'd lucky if not happen ...

my question is: reconstruct stacktraces cut before processing them?

should pre-process rdds , create new ones containing i'm waiting for? should reconstruct stacktraces via global buffer? should somehow play offsets? how exactly?

any ideas welcome.

thanx,

--mike


Comments

Post a Comment

Popular posts from this blog

android - net_scheduler holding wakelock -

i have observed on occasions, net_scheduler keeps acquiring wakelock repeatedly on short durations , prevents android device sleep. from logcat, not able obtain conclusive information why might happening. of gms services active. can suggest when there higher chance occur perspective of gms services. this seems coming gcmnetworkmanager , schedules network-oriented tasks, google play services able batch network operations across system. also if use more advanced scheduling firebase jobdispatcher , internally gcmnetworkmanager used.
Read more

sql - MySQL : Getting Entries from a many-to-many table -

i have following table, select records should have mentioned values. example select companyid should have 3 providerid 200,300,400. running query should retrun companyid 2. opposite of in . in advance ! companyid | providerid -------------------------------------- 1 100 2 200 3 500 4 600 2 300 2 400 7 100 do group by , select rows of 3 wanted employeeid values. use having make sure 3 different employeeid values. select companyid tablename employeeid in (200, 300, 400) group companyid having count(distinct employeeid) = 3 another approach double self join, 1 each employeeid. however, less flexible: select distinct t1.companyid (select companyid tablename employeeid = 200) t1 join (select companyid tablename employeeid = 300) t2 on t1.companyid = t2.companyid join (select companyid tablename employeeid = 400)...
Read more

java - Retrieving data from database using jsp (Hibernate + Spring + Maven) -

hello, guys. i'm trying retrieve data oracle database, nothing appears on screen. here userslist.jsp <%@page contenttype="text/html" pageencoding="utf-8"%> <!doctype html> <html> <head> <title>spring mvc hello world</title> </head> <body> <h2>all users in system</h2> <table border="1"> <tr> <th>user id</th> <th>login</th> </tr> <c:foreach items="${users}" var="user"> <tr> <td>${user.userid}</td> <td>${user.login}</td> </tr> </c:foreach> </table> </body> </html> it seems ok. , here spring controller userscontroller.java package com.controller; import com.database.users; import com.service.usersmanager; import java.util.list; impo...
Read more