Technology Challenges for Big Data

Technology Challenges for Big Data

 

 

Question - Technology Challenges for Big Data

--------------------------------------------------------------

 

Ingesting Streams at an extremely fast pace . This

relates to the velocity of streaming of big data over

the enterprise systems . This velocity of big data

generally relates to the torrential and fast streams of

data . In some cases the velocity of the data streams

may be very large and fast to be stored , but still then

the pace of the data inflow should be monitored

and this can be done by creation of special purpose

ingesting systems that could open multiple number of

channels for receiving , utilisation and consumption

of the data . These ingesting systems could be used

for holding data in queues from which business

applications can read and process data at their own

pace and convenience.

 

The second layer and second most important purpose

to solve for big data systems is thus managing the

challenge posed by velocity of big data . And to deal

with this issue , special stream-processing engines

have been put in place where all the incoming data is

fed into a central queueing system over the network

of big data machines . From this system , a fork

shaped system sends data to the batch storage as well

as to the stream processing directions . These stream

processing engines can do the work of collection

of high velocity big data and send it to the batch

processing systems who stream the incoming data in

multiple batches and redistribute the volume among

the batch segregation systems . A most popular

system for this type of work handling is apache spark

which handles the work of streaming applications .