According to Internet of Things true believers, the time is just around the corner when our cars, homes, appliances, TVs, PCs, phones and any other electronic or mechanical device in our lives will be spewing out data in all directions. That makes some sense, since IoT devices – at least those now envisaged – are designed for data spewing as they have minimal compute capacity presently.
Cisco estimates that already nearly 15 million connected devices comprise the nascent IoT, which will grow to 50 million by 2020. That sounds impressive until you realize it is less than 3 percent of the “things” on our planet potentially able to participate in IoT. Unfamiliar numerical terms such as zettabytes must enter our lexicon to describe the volume of data to be generated, consumed and analyzed.
What the IoT Data Wave Means for Big Data
The processing of the rivers of big data coming from today’s embedded sensors, telemetry, RFID chips, PCs, mobile devices, wearables, etc. already leaves 90 percent of these data in the dustbin. That is primarily because current big data hardware and software stacks are inadequate to manipulate it all let alone comprehend it.
Big data compute, storage and networking capabilities improve daily. However, even those enterprises on big data’s bleeding edge are today ill-equipped to handle the expected data flood gushing from the IoT let alone the larger Internet of Everything that Cisco tracks.
Even if IoT is realized in twice or thrice the time of most projections, then big data enterprises are going to be perennially behind the curve for the foreseeable future. The constant running to catch up will be the prime driver of the big data ecosystem beyond the next decade. If that does not kill big data, it will only make it stronger. Enterprises large and small will join the data mining gold rush if real-time analytics improve and a big data meta-architecture, as hinted at by Hadoop, emerges.
The Obstacles to a Happy Marriage between IoT and Big Data
Lack of Standards
Having to figuratively invent the wheel over and over again is the bane of any competitive industry. Without standards, IoT will struggle to reach escape velocity due to technology fragmentation. Standards must be in place for efficient access to “things”, consistent API interfaces, machine-to-machine communication, addressing privacy and security issues and lowering entry barriers to smaller, innovated players.
Closed or Inefficient Architectures
IoT is a game changer for big data architecture. All stakeholders are just now starting to recognize that dealing with IoT will require as much collaboration as competition.
The sheer magnitude of IoT data volumes dictate a layered hardware/software stack that is too gigantic, geographically dispersed and complex for a single enterprise or cloud providers. It begs for an ultra-distributed meta-architecture that step by step digests, absorbs and disperses unstructured data as it is collected, cleaned, normalized, correlated with other data, stored when necessary, deeply analyzed and presented. Along the way, vendors who today specialize in each of these processing layers will contribute via enormous arrays of small-scale data centers.
Analytics Capability Growth Rate
Above all else, business intelligence processing is the critical bottleneck to realizing the full potential of big data. The rate at which supporting analytics can improve is questionable without significant breakthroughs, but the search for data gold represents an immeasurable incentive. The deluge of IoT real-time data headed down the analytic pipeline will create even more pressure but is likely to engender even more opportunities for value extraction.
The Internet of Things is not an invention but a logical consequence of highly available, low-power, low-cost sensor technology and improvements in wireless connectivity penetration. Related technology improvements and cost-reductions in compute, storage and network hardware will complement the growth of IoT and make it something useful and valuable. And, finally, IPV6 is going to receive the appreciation it justly deserves.
All this power to generate, gather and process new, real-time micro-data is for naught, however, if it must be set aside awaiting analysis capabilities to catch up. Fortunately, although big data infrastructure and software are likely to be overwhelmed initially, that and analytic capabilities seem to have a bit of a head start. Increased collaboration among stakeholders, an effective, shared processing architecture and the inevitable analytical breakthroughs may just carry the day in the end.