In today’s data-first world, IT infrastructure is the foundation for strategic decision-making, with companies requiring larger quantities in shorter periods of time. This is putting the traditional data model – where data from systems like CRM, ERP and LOB applications are extracted, transformed and loaded (ETL) into the data warehouse – under pressure. The problem is compounded by increased data volumes from social apps, connected devices (IoT) and emerging sources of data.
The need to gather data from traditional, transactional systems, like ERP, CRM and LOB, and then integrate this data with social, mobile and connected devices has driven the adoption of big data storage technologies such as Hadoop. At Optimus, we’re finding more and more users demand predictive, real-time analytics to make use of their data, something that can’t be done with traditional data warehouse tools. Consequently, organizations are considering cloud-based solutions such as Azure to transform their data warehouse infrastructure.
Microsoft knows this, and are growing their solution portfolio accordingly. Below are four ways in which Microsoft Azure is adapting to meet the demands of today’s modern data warehouse.
1. Consistently High-Performance for all Volumes of Data
Microsoft is working to solve the problem of achieving high levels of performance for large datasets through MPP technologies, in-memory columnstore and optimizations on core query engine. In particular, Optimus is seeing SQL Server emerge as a leader in performance and scalability. SQL Server supports a large number of cores with complex vector instructions while holding terabytes of memory and contains local flash storage that provides high I/O bandwidth. When optimized for inherent parallelism and concurrency, it is not uncommon for users to outperform large distributed databases.
In one example, Microsoft and Intel teamed up to create a 100 terabyte data warehouse using a single server, four Xeon E7 processors and SQL Server 2016. According to the report, “The system was able to load a complex schema derived from TPC-H at 1.6TB/hour, and it took just 5.3 seconds to run a complex query (the minimum cost supplier query) on the entire 100TB database.”
2. Storing Integrated Data
Companies are looking for ways to store integrated – both relational and non-relational – data of any size, type and speed without forcing changes to applications as data scales.
Enter the Azure Data Lake Store. Data Lake makes it simple for everyone, from analysts to developers and data scientists, to access, add and modify data, regardless of its state.
Facilitating all of this is Azure HDInsight, a cloud-based Hadoop and Spark cluster. HDInsight lets your team create analytic clusters, manipulating data into actionable insights. In addition to a fully managed Hadoop service, Microsoft has included PolyBase in HDInsight, which provides the ability to query relational and non-relational data in Hadoop with a single, T-SQL-based query model.
3. Built with Hybrid Data Storage at the Core
While the cloud continues to gain popularity, companies are realizing that they still need to keep at least some information on-premises. Microsoft is acutely aware of this and has built Azure accordingly. Their data warehousing and big data tools are designed to span on-premises and cloud warehouses. Microsoft’s hybrid deployment is designed to provide the control and performance of on-premises with the scalability and redundancy of the cloud. Optimus is seeing users access and integrate data seamlessly, while leveraging advanced analytics capabilities, all through Azure.
4. Machine Learning and Big Data in Real-Time
Traditional advanced analytics applications use outdated methods of transferring data from the warehouse into the application tier to procure intelligence, resulting in unacceptably high latency and little scalability.
In contrast, Microsoft has transformed integrated analytics with machine learning in the cloud. The Cortana Intelligence Suite, coupled with R Server, can be deployed both on-premises with SQL Server and in the cloud with HDInsight. The resultant solution is one that solves for hybrid, scales seamlessly and enables real-time analytics.
There are many factors driving companies to consider an Azure Cloud data warehouse migration. To learn more, check out our e-Book, Building a Modern Data Warehouse on Azure.