Have you ever wondered how your favourite coffee shop seemingly knows what you want even before you order? Or how an ecommerce player like Meesho can make online buying a more personalised experience for its customers based on browser history? All that is done with real-time data insights – a technology quietly transforming customer experience.
It starts with a constant flow of information, like a never-ending stream of digital crumbs we leave behind with every click, swipe and purchase. By harnessing this data in real time, businesses can make lightning-fast decisions to deliver personalised, responsive services people often expect.
A recent study published by the MIT Center for Information Systems Research said that top-quartile real-time businesses (RTBs) saw more than 62% higher revenue growth and 97% higher profit margins than the bottom quartile. For context, RTBs respond immediately to opportunities and challenges by automating processes and enabling fast decisions at all levels using trusted and easily accessible data.
In today’s fast-paced business environment, the rise of the RTBs is inevitable. But their success depends on real-time data streaming that can unlock vital information about customers and markets.
Confluent, a data streaming pioneer with solutions built around Apache Kafka, researched the APAC market extensively and found that 64% of the data-streaming adopters have clocked 5x or more ROI after investing in this technology. Interestingly, the ROI has risen to 71% for Indian companies and 69% for Indonesian businesses.
“India is ahead of the curve when it comes to data streaming adoption and skill sets,” Will LaForest, global field CTO of Confluent, told Inc42 in a one-on-one conversation. He recently visited India for Confluent’s Data In Motion Tour 2024, the company’s regional in-person event that is making over 50 stops globally.
The company conducted a 2024 data streaming report globally in partnership with Freeform Dynamics and Radma Research. From this broader study, it has extracted and analysed data specifically from 1,424 IT leaders and senior stakeholders across the APAC region, covering Australia, India, Indonesia, Japan and Singapore for its Asia-Pacific Data Streaming Advantage report.
Sharing on the sly ‘how he had loved working with data way before it became cool’, LaForest discussed emerging data trends, how businesses embraced real-time data insights globally and the competitive advantage startups would gain from real-time data streaming. He also detailed Confluent’s road map for the Indian market. Here are the edited excerpts from the interview.
Inc42: You have travelled worldwide and interacted with customers from various industries. Given their varied challenges and needs, how do businesses adopt real-time data across regions?
Will LaForest: The technology is truly transformative, but data streaming is all about a shift in the mindset. For 60 years, we have been gathering data, putting it in a datastore, querying it and then using it. Now, instead of bringing questions or analyses to a database, you are inverting the relationship and working with a continuous data stream being delivered to the questions and analysis.
This conceptual change takes time to get used to but enables you to build scalable, fast applications and do analysis continuously on a constant data flow versus periodic batches. So, the first key factor is the change in mindset. Then, we can work towards cultivating the right skill sets to enable this process.
Inc42: How can startups use real-time data to stay ahead of rapidly shifting market trends and drive product evolution?
Will LaForest: Easy access to fresh, up-to-date data is a major competitive advantage for businesses in today’s rapidly evolving environment.
Startups have a critical advantage, though. They can build their technology stack with real-time data access from the very beginning without dealing with legacy systems and data silos that plague many enterprises. This data streaming first architecture gives startups a significant edge.
They can unlock critical insights by tapping into continuous, real-time data streams and act on those much faster than industry incumbents, which still have many batch processes they are relying on. This agility allows startups to evolve their products rapidly and stay ahead of fast-changing market trends.
For instance, a startup in the logistics or delivery space can use real-time data streaming to optimise fleet management, predict delivery times and provide a superior customer experience. Similarly, a fintech startup can leverage real-time data to detect fraud, make instant decisions and deliver more responsive services.
Ultimately, accessing fresh, high-quality data on demand is a force multiplier for startups, empowering them to innovate quickly, make better-informed decisions and outmanoeuvre larger, more established players. Embracing data streaming technology is crucial for startups looking to disrupt their respective industries and achieve rapid growth.
Inc42: How does real-time data make AI-driven applications more powerful and intuitive?
Will LaForest: Data streaming has long been important for AI implementations, as companies need suitable datasets to train their models. But now we are seeing a major leap – the ability to apply these AI models as quickly as possible against real-time data streams. That’s a game-changer.
Let’s take fraud detection, for example. As we know, batch processing in a data lake is slow compared to real-time processing. With fraud, it is crucial to do the inferencing as close to the data streams and as quickly as possible. If the AI model waits for data to be stored and then batch processed, it’s too late.
The immediacy makes companies more dynamic and responsive. For GenAI applications, data engineering has to be done at inferencing time, which requires fresh, up-to-date contextual data. So, if you are not using data streaming for GenAI, you are doing something wrong.
Inc42: What about Confluent’s plans and evolving roadmap to support India’s digital transformation?
Will LaForest: India is one of our fastest-growing markets, and the talent and engineering expertise we have here is a major asset. About 15% of Confluent’s employees are based in India. Our global headcount is now over 3,000, of which more than 400 are in Bengaluru, Chennai and Mumbai. Our office in Bengaluru is our biggest location outside Silicon Valley.
Our India operations are about five years old, and the goal is to continue scaling our presence and capabilities to support the rapid digital transformation underway in this country.
Inc42: Your recent study reveals insights into India’s data streaming landscape. Tell us more about the findings and their implications.
Will LaForest: Certainly! Let me share some industry data first. A recent Confluent study says 95% of India’s IT leaders with experience in data streaming reported 2x to 10x ROI from those initiatives. Also, 94% of the survey respondents from India prioritised data streaming as an essential IT investment in 2024, and it surpassed global figures. These statistics underscored the pivotal role real-time data streaming played in driving India’s digital economy.
Right now, Indian businesses are keen to invest in emerging technologies. While [cyber]security remains their top concern, data streaming and AI/ML solutions are high on their agenda. Per our APAC Data Streaming Advantage report, 85% of the Indian IT heads think GenAI will trend up in the next two years and 80% bet on data streaming. So, Indian businesses are ready to move ahead in a fast-evolving tech landscape.
Inc42: What excites you most about the future of data? How will that trend reshape global business strategy?
Will LaForest: I am particularly bullish on the rise of B2B data sharing through real-time data streaming. This has huge implications for industry segments like logistics, where time is money. I would say that the ability to do real-time B2B data sharing can be a game-changer. We also see this in financial markets, for instance NASDAQ provides market data as data streams. I expect this trend to take off in a big way as more businesses recognise the power of seamless, real-time data exchange.
Data products have gained a ton of traction recently, especially as these align well with the principles of successful data streaming. The fundamental idea is choosing the right dataset and making it easily accessible to help people innovate.
It also ties in with democratising data. Businesses today want more people to use their data instead of confining data insights to a small team of data scientists. Companies can break down data access and usage barriers by packaging data into well-defined, discoverable, and consumable ‘products’.
Data streaming’s capability to provide continuous, real-time feeds is a key enabler of this data product model. Rather than relying on static data dumps or batch updates, businesses can offer dynamic, self-service data products that users can tap into as needed. This way, data assets can deliver more value and accelerate innovation across organisations.
