Using the combine command in R for making data

 Using the combine command in R for making data

Technology challenges and solutions of big data - Variety

 Technology challenges and solutions of big data - Variety

 

Technology challenges and solutions of big data - Variety

===============================================

 

Description

To efficiently store large and small data objects and data formats

 

Solution

Columnar databases using key-pair values format

 

Technology Used

HBase , Cassandra

 

 

what are some technology challenges for big data

 what are some technology challenges for big data

 

what are some technology challenges for big data

 

•  Storing Huge Volumes of Big Data

The very first and foremost challenge over the technology challenges of big data relates to the storing of huge quantities of data . As of now , there is no such computing machine with a storage thatis as big and enough to store the ongoing growing volume of data which has been a big cause of issue for most of the organisations engaged in the business of big data . Therefore , one should first try to store the huge quantity of data in manageable & inexpensive machines and then with upscaling prices and costs one could strive for better infrastructure . However , as it is customary with machines , machines are prone to getting failure at any random point of time and with more and more machines over the big data ecosystem , the chances of failure at any time may also becomes high . Each of the participative and implementable machines in the list could fail at some point or another and failure of a machine could entail a

loss of valuable data stored over them .

 

• So the first function of big data technology thus should be to store huge volumes of data within them and that too without incurring a high cost to the organization , while also combatting the risk of data loss . So all big data systems distribute data across a large cluster of inexpensive machines connected with each other over the big data network . This process also ensures that all of the data within the system is made failproof by ensuring that every piece of data is stored on multiple machines which would guarantee that at least one copy of the data is available to all the connected storage machines . All the above-mentioned processes are made fail-proof with the help of Hadoop which is a very well-known clustering technology of big data . Hadoop's data storage pattern is called as Hadoop Distributed File System (HDFS). This technology HDFS is built on the pattern's of Google's Big File Systems , which is designed to store billions of pages and sort the pages

to answer user search queries .

Technology Challenges for Big Data

Technology Challenges for Big Data

 

 

Question - Technology Challenges for Big Data

--------------------------------------------------------------

 

Ingesting Streams at an extremely fast pace . This

relates to the velocity of streaming of big data over

the enterprise systems . This velocity of big data

generally relates to the torrential and fast streams of

data . In some cases the velocity of the data streams

may be very large and fast to be stored , but still then

the pace of the data inflow should be monitored

and this can be done by creation of special purpose

ingesting systems that could open multiple number of

channels for receiving , utilisation and consumption

of the data . These ingesting systems could be used

for holding data in queues from which business

applications can read and process data at their own

pace and convenience.

 

The second layer and second most important purpose

to solve for big data systems is thus managing the

challenge posed by velocity of big data . And to deal

with this issue , special stream-processing engines

have been put in place where all the incoming data is

fed into a central queueing system over the network

of big data machines . From this system , a fork

shaped system sends data to the batch storage as well

as to the stream processing directions . These stream

processing engines can do the work of collection

of high velocity big data and send it to the batch

processing systems who stream the incoming data in

multiple batches and redistribute the volume among

the batch segregation systems . A most popular

system for this type of work handling is apache spark

which handles the work of streaming applications .

 

 

 

 

variety of data - big data

 variety of data - big data

 

 

      Variety of Data

     

 

• Big Data is inclusive of the forms of data for all kinds

of functions and from all sources and devices . If

traditional data forms such as invoices and ledgers

were like a small store then Big data can be considered

as the biggest imaginable shopping mall that offers

unlimited variety .

 

• The three major kinds of variety of data are as follows :

 

01) FORM OF DATA :

Data types range in variety from numbers to text ,

graph , map , audio , video and many other forms with

some of the data types being simple and the othes

being very complex . There can be composites of

data that include many elements in a single file . For

example , text documents have graphs and pictures

embedded in them . Video movies have audio songs

embedded in them . Audio and Video have different

and vastly much complex storage formats than

numbers and text .Numbers and text can be more

easily analysed than audio or video file

 

02) FUNCTIONS OF DATA :

There is a lot of data that is being generated form human conversations , songs and movies , business transactions records , machine operations performance data and anew

product design data , old archived data , etc . Human

communication data is needed to be processed very

communication data is needed to be processed very

much differently form operational performance data

with different expectations and objectives . Because

of all these aspects , Big data technologies can be

used to recognise people's faces in ictures , compare

voices to identify speaker of the voice and compare

handwritings to identify the writer also .

 

03) SOURCE OF DATA :

Mobile phones and tablet devices enable a wide series of applications ( or apps ) to access data and also generate data from anytime and anywhere . Web access and usage and search logs are another new and huge source of data . Businesses and enterprise systems generate massive amounts of structured business transactional information .Temperature and pressure sensors on machines , and Radio Frequency (RFID) tags on assets , generate incessant and repetitive data .

 

 

Overall it can summarised as there are three forms of

data and data sources :

(i) human - human communications

(ii) human -machine communications

(iii) machine to machine communications

 

 

 

 

Big Data - Obtaining Data from Private Sources

Big Data - Obtaining Data from Private Sources

 

 

Big Data - Obtaining Data from Private Sources

===========================================

 

*  One can obtain data from private organisations such

as Amazon and Google

 

* Such kind of big MNC's and corporations maintain

huge databases with all sorts of useful information

 

* If the data obtained is supposed to be used for a

commercial setting , then one can pay some fee for the

data access or obtaining by downloading the content

to personal storages and servers

 

* An advantage of using data from private data

sources is that they are mostly high quality and

consistent , cleaner compared to those obtained from

public data sources and highly diverse data

 

 

 

What is meaning of the term Data Massaging

  What is meaning of the term Data Massaging

 

 

What is Data Massaging

====================

 

In scenarios when the data obtained from big data

sources may or may not be complete as proper tags

and headers have not been specified over the data .In

order to make the data better suited for machine

learning , some identifiers in form of tags and labels

need to be created which adds value and additional

relevant information to the data which makes it a right

candidate for processing for machine learning .

 

 

 

 

Understanding Big Data

 Understanding Big Data

 

Understanding Big Data

--------------------------------

 

Big Data is different from traditional data in every way i.e, space , time  and function . Quantitatively , Big data is more than 1000 times more than the traditional data and the data generation and transmission   speed is also of a great order . The forms and functions of Big Data are 10  times more diverse from numbers to text , pictures , audio , video , web  logs , machine data and many more .

 

Big Data is mostly over 90 percent unstructured data and it should also be dealt in a different way as well . Big Data carries with it huge opportunities to innovate and manage the entire life cycle of big data to generate , gather , store , organise , analyse and visualise the data .

 

 

Big Data can be examined and understood at different levels

 Big Data can be examined and understood at different levels

 

 

Big Data can be examined and understood

at different levels as:

 

01)  At a fundamental level , it is just another collection of large

        amount of data that is collected and that can be analysed and

        utilized for the benefit of business .

 

02)  On a higher level , Big Data can be seen as a special kind of

              data that poses unique challenges and offers unique benefits .

 

03)  At Business Level , data generated by business operations can be

       analysed to generate insights that can help the businesses to make

       better and faster decisions which can enable the business to scale

       bigger and generate even more data

 

 

 

Data Abstraction Concept

Data Abstraction Concept

 

         Data Abstraction Concept

          =====================

For a database system to be usable one user should

be able to retrieve data efficiently from it . The need

for efficiency has led designers of the database to

use complex data structures to represent data in the

database . Since many database system users are not

computer trained , developers ide the complexity from

users through several levels of abstraction which hide

the intricate details of the system but simplify users'

interactions with the system .

 

Physical Level Abstraction:

======================

The lowest level of abstraction describes how the

data are actually stored in the database system .The

physical level describes complex low-level data

structures in detail

 

Logical Level Abstraction:

=====================

It is the next higher level of abstraction on top of the

physical level and it describes what data are stored in

the database and it also describes what relationship

exists among those data .Thus the logical level

describes the entire database in terms of a small

number of relatively simple structures and as such the

users of the logical level does not need to be aware of

the complexity . Database administrators who decide

what information to keep over the database use the

logical level of abstraction mostly

 

View Level Abstraction

===================

This is considered as the highest level of abstraction

of the database and it describes only part of the database.

Even though the logical level of the database

uses simpler structures , complexity always remains

because of the variety of information stored over the

large database .The view level of abstraction exists to

simplify the interaction of the system with the users .

The system may provide many views for the same

database and for multiple categories of users as well.

 

 

 

steps to take over an analystical project during any data analysis - a short summary

steps to take over an analystical project during any data analysis - a short summary

 

A short summary on the concept of Star Schema of Datawarehousing

 A short summary on the concept of Star Schema of Datawarehousing

SQl Select Into statement - example

 SQl Select Into statement - example

SQL - Select statement with OR filter

 SQL - Select statement with OR filter

what happens when a model gets over-fitted in machine learning - a short summary

 what happens when a model gets over-fitted in machine learning - a short summary

What is a DBMS and why is it used

 What is a DBMS and why is it used

issues over file processing systems and environments - Difficulty in Accessing Data

 issues over file processing systems and environments - Difficulty in Accessing Data

Database system Applications and Usage

 Database system Applications and Usage