Real-Time Processing of Data for IoT Applications

The Internet of Things (IoT) drives value across nearly every sector, spanning from manufacturing and logistics to retail and resource management. The IoT captures data from a network of connected “things” that includes drones, delivery trucks, medical devices, security cameras, and construction equipment.

While IoT sensors and devices collect tons of valuable insights, they also generate massive high-speed data streams that are difficult to process, analyze, store, and secure. IoT data is also highly perishable, and without the right tools, organizations miss opportunities to act on time-sensitive insights with the most potential.

In this article, we discuss how real-time data analytics and IoT applications come together to create new opportunities across a wide range of sectors.

What Does Real-Time Data Processing Mean for IoT Applications?

As IoT adoption continues to rise, organizations from every sector struggle to keep up with these massive datasets expanding at exponential rates. As a point of reference, IoT devices and sensors can capture gigabytes of data within a few hours—and that’s before you consider the data coming from your CRM, social media channels, financial reports, and so on.

At the same time, Big Data analytics, AI, and machine learning are evolving at a break-neck pace. By applying AI to IoT data management and analytics, organizations can quickly pull valuable information from these massive, heterogeneous data sets and respond to real-time conditions. Together, these technologies are driving game-changing innovations. For example, Big Data’s inherent characteristics are perfect for training AI and ML applications fast.

Those intelligent applications can then be used to automate processes, predict equipment failures, and detect security threats in real-time. In the case of fully-autonomous solutions, AI takes the wheel, relying on a connected network of IoT devices to guide the way.

With significant gains in autonomous driving at all levels, real-time analytics can support drivers with safety features like automatic braking, parking, and collision avoidance by transmitting data. While there are endless examples of what AI, advanced analytics, and the IoT can accomplish, they can’t deliver on those promises without the right tools.

Real-Time Insights Depend on Powerful Computing

Most of the IoT platforms in use today were designed to connect the various devices within a network and merge and process data streams from several heterogeneous sources. These platforms often address many of the challenges IoT presents—like storage, security, and interoperability—and they can integrate with data analytics solutions to provide valuable business insights. But because most data analytics solutions use a cloud computing architecture called Platform as a Service (PaaS), real-time data processing isn’t possible.

According to a recent Dell report, using cloud-based systems to process IoT data has several limitations, including security risks, latency, and missed opportunities to act on powerful, real-time insights. While IoT data streams themselves capture what’s happening in the moment, processing those data streams means sending them to the cloud for off-line analysis and processing, which can then be reviewed at a later time. You’re also working within a system where you send information to a remote location at a volume that may exceed network bandwidth and waste storage space and computing power on unusable insights.

The report found that while just 29% of participating firms have incorporated edge computing into their analytics strategies, 69% of respondents agreed that prioritizing edge computing for processing IoT data would help them achieve their primary business goals. However, it’s worth noting that edge computing alone won’t unlock the door to real-time data analytics. Technologies like 5G and WiFi6, IoT platforms like Kaa and AWS, event-driven architectures, and analytics tools like Kafka, Kinesis, Spark, Storm, Cassandra, and BigTable (all designed for processing continuous streams) are converging to enable real-time Big Data analytics.

The Convergence of IoT and Big Data Analytics

The convergence of IoT, Big Data, and AI-driven analytics presents a number of new opportunities for companies to create more competitive business models. According to Forrester’s 2020 Predictions, enterprise strategy is becoming a critical initiative for driving digital transformation. While the report mentions that interest in Big Data has waned over the past few years, innovations in AI and machine learning are driving renewed interest as they present new opportunities to process data and put it to good use.

At the same time, we’re seeing more affordable hardware, software, and sensors as well as emerging standards and best practices driving IoT adoption. As such, there’s a rapidly growing number of connected “things” (including audio, video, and images) capturing continuous data streams and metrics that measure machine functions, environmental conditions, and more.

Here’s a look at some of the ways leading companies are bringing these technologies together to drive value:

• Disney. Disney is applying advanced analytics to big IoT data and machine learning techniques to create personalized in-park experiences with a wearable, RFID-enabled MagicBand that collects customer traffic patterns, provides access to hotel rooms, and enables guests to charge purchases back to their room. Disney teams can apply these insights to several use-cases, including optimized park logistics that reduce long wait times for rides by redirecting guests to less-crowded parts of the park. Additionally, Disney might also use these insights to predict a guest’s favorite character and arrange surprise meet-and-greets.
• CPS Energy. In an interview with SAS, San Antonio-based CPS Energy discussed how they leverage data analytics to address multiple use cases—from leak detection to helping customers save money. The utility’s approach is all about gathering as much data as possible from smart meters, consumer usage habits, street lights, and other sources. This combination of event stream data, usage, and real-time anomaly-detection supports consumers in several ways. For one, the utility can use real-time analytics to detect leaks or outages and address those issues as they emerge. Long-term, CPS can use this massive amount of data to pull the psychological triggers that get customers to embrace energy-efficiency initiatives or provide products that help consumers.
• Primex. This example looks at how a company might view Big Data analytic solutions to solve a problem. According to a 2016 SQLStream case study, IoT company Primex decided to replace its legacy architecture with a more efficient, cloud-based solution. They settled on a serverless architecture that included Amazon Web Services (AWS) Lambda, Kinesis Streams, and CloudWatch. Initially, this was a practical choice, as Primex couldn’t maintain an open-source platform like Apache Spark. At the time, the company had over 150K connected devices deployed in the field, which meant they were processing over 67K AWS Lambda requests every five minutes. This system came with several challenges, as it wasn’t built to handle data on a massive scale.The report mentions one incident following a four-hour outage where it took 20+ hours for the system to process a massive backlog of sensor data. The organization also incurred high costs (around $565 a day), latency, and instability. By moving to SQLStream Blaze (a SQL-based platform for real-time stream processing), Primex reduced Lambda costs. The company charges clients a fixed rate, as opposed to AWS, which charges by the transaction. Additionally, the low-latency response times and the ability to ingest and process Big Data sets underscores the importance of selecting the right architecture to power your solution.
• Alibaba. Alibaba’s financial services arm, Ant Financial, uses real-time analytics and data sourced from prior online transactions (think bill-paying history) to evaluate potential borrowers. The Chinese tech giant’s real-time credit scoring system allowed small to midsize merchants to apply for microloans and access funds quickly without collateral. According to an IFC report, these online solutions allow more small business owners to participate in the economy. This is key because researchers found that 70% of female entrepreneurs have trouble securing loans within the traditional banking system, making it difficult to scale up or weather economic hardship.

The Role of Big Data Analytics in IoT

While the Internet of Things and Big Data are two distinct concepts, they’re becoming increasingly interconnected. In an IoT, you’ve got a massive network of sensors that collect an unprecedented amount of data from a variety of sources feeding into the broader Big Data landscape. The following is an example that demonstrates how much data just one of these devices could collect:

The Oura Ring is a device that is worn on a user’s finger and tracks the user’s sleep, temperature, and physical activity. The device captures data at a rate of 250 times per second.

To put this in context, if we were to pour water at a rate of a cubic foot per minute into Madison Square Garden, we would fill it in less than seven hours. That’s a lot of data! This data can include things like customer usage insights, sentiment analysis, sales metrics, and behavioral patterns, among countless other data sources. Together, Big Data and IoT create contextual insights that can be applied to improve products, services, and processes—and in turn, generate more revenue.

Big Data analytics platforms hold the key to unlocking this information by taking unstructured IoT data—such as foot traffic at a theme park, weather patterns, or patient health—and analyzing that information alongside other data sources to provide a holistic view of the situation. From there, platforms can organize that information into digestible insights that companies can use to optimize their processes. This means that environmental data from sensors, surveillance footage, log files, and geo-location data can join forces with social media and consumer behavior insights, to create a better understanding of your audience—bringing them to life in a way that marketing metrics can’t provide on their own.

How Do IoT and Big Data Impact Each Other?

According to Carrie MacGillivray—IDC Group Vice President of IoT, 5G, & Mobility—IoT is increasingly driving value creation across both the public and private sectors by enabling the exchange of information among people, processes, and the system of connected things. IoT applications generate raw data from sensors and devices, which is then collected in a centralized repository known as a data lake. These data lakes house IoT data alongside structured data such as transaction records and customer profiles and unstructured data from sources like social media, emails, and logs.

Big Data analytic platforms can then generate reports and visualizations from insights sourced from all data sets that feed into that data lake. In turn, this provides a big-picture view of how external factors like market fluctuations, trends, and environmental conditions impact what’s happening inside a business.

AI-based analytics tools also benefit from IoT data. By feeding high-volume, high-variety IoT data into AI applications, you can effectively train those systems to understand and make predictions from real-time data without human intervention, which can improve business outcomes over time.

Deriving Value from IoT Data

IoT and Big Data analytics are no longer stand-ins representing the promising use cases of tomorrow; they’re quickly emerging as essential tools for staying competitive right now. They give organizations the ability to extract value from IoT sensors and systems by analyzing IoT data with existing business tools and third-party data sets to bring more contextual information into the fold.

Then, the information can be applied to create better products, services, and experiences. However, organizations need to make sure that they have the infrastructure in place to support real-time data processing at scale, in order to get the full value from their investments.

[adinserter name=”Data Analytics CTA”]