Data Convergence

What is Data Convergence

Data Architectures are becoming complex day by day as the complexity of the data is increasing. The complexity of the data arises from the fact that organizations today, use different data for different applications even if the source of the data remains the same. 

There is operational data serving the real time data needs and then there can be analytical data which is used for the machine learning algorithms. 

With the large amount of data coming in from various industries it becomes challenging as the overhead of maintaining separate stores can outweigh the benefits of the data being received.

By having all of the data in one location on the cloud, one can get real time access to the data for the operational and analytical needs. This leads to simpler data governance with low setup and maintenance cost and no unnecessary movement or migration of data. This is what Data Convergence is in a nutshell.

Haber's Approach

We receive a large amount of data from numerous customers through the eLIXA box setup in the plants across different industries. The eLIXA sends this data via IOT hub over to the cloud (Azure). The volume of data received from one of these boxes can range from 1-4 data packets sent per minute. This data is then ingested to the SQL Server hosted on cloud via a combination of Azure Stream Analytics and http triggered functions.

Simple is Beautiful

This data is stored in same location for all the users which makes it easier to pull the real time data for different customers on the dashboard efficiently and effectively. Since we have minute by minute data from n number of plants and n number of data points are being received, the complexity increases. If we go on segregating the data into different data stores that will bring upon a huge burden onto the resources. This will force us to spend many on acquiring the resources to fulfill the needs of our applications. Hence the cost goes high and high as more and more customers are added to the portfolio.

• Data Convergence helps in overcoming the problem of running multiple application in multiple silos.
• Scalability is another very important factor where data convergence simplifies things. With a single location for a data store the scaling becomes fast and easier without any hassle.
• Data gets more secured due to less number of routes available for unethical sources to get in the system. This makes us less vulnerable to attacks and helps in having maximum security over one location.
• Data can be backed up over different geological locations of the world for disaster recovery. Here as well data convergence removes the overhead of backing up multiple data stores since everything lies at one place.

Read More: AI in Post Covid Era

Advantages

• There is no need of connecting different data stores for the same data to be shown up on the dashboard for analytical purposes
• There is no need of maintaining separate ETL pipelines for the data being ingested
• The maintenance cost is less since there is only one data store which needs to be taken care of
• More emphasis can be laid on customer problems and solutions rather than solving the data problems
• The source of truth remains the same for all the applications. So no duplication and cleaning of data is required