Companies\u2019 accumulation of big data was on the rise before COVID-19. When nearly every industry shifted to virtual offerings, opportunities to collect and strategically leverage data and to engineer new data solutions boomed. In the years since, we have seen the marketing and sales chessboard overturned. Data, companies have come to realize, now infuses every solution, every sale, every customer interaction.<\/p>\n
Who is the gatekeeper of this power and potential? Data Engineers.<\/p>\n
Data engineers build the systems that collect, store and analyze companies\u2019 data assets.<\/strong> They construct the worlds that IT leaders and business execs probe to uncover insights and make business decisions. For thirty years or so, there were relatively few changes in the core tenets of data engineering. Those days are gone. Soon, we\u2019ll barely recognize the field.<\/p>\n Data engineering has evolved so quickly that the tools and techniques learned on the first day of university are obsolete by the time many engineers enter the field. Talent wars are underway for the brightest graduates. We are witnessing the dawn of astonishing technologies<\/a> that can propel organizations to exponential growth. Those that fail to adapt may lose top talent and fall behind.<\/p>\n This is the Great Data Disruption. Data engineers are sitting on a corporate fault line \u2014 and the cracks are showing, even if many execs can\u2019t see them yet.<\/strong> The rewards will be great for organizations that pay attention to the tremors and are prepared to adapt.<\/p>\n Data engineering encompasses all the tools and techniques used to turn data into business value<\/a>. In the past, that included four core skills:<\/p>\n Historically, data engineers have focused on one or two of these areas, with many specializing in a single tool. A major industry shift has been the growing expectation that data engineers should be adept in a multitude of tools and techniques. Today, data engineering skills should include, but are not limited to:<\/strong><\/p>\n The most important skill for data engineers today is the need to excel at quickly learning and applying new technologies. Companies won\u2019t laude the highest compensated engineers for their years of experience in a singular technology. The greatest reward will go to those who can assess an array of tech options and apply the best solution(s) for the business problem at-hand.<\/strong><\/p>\n The question is, will IT leadership be able to keep up with the pace of change?<\/p>\n IT managers have traditionally blanched at the idea of their developers exploring innovative tools and approaches. They have often viewed the pursuit of new technology as a diversion from productivity.<\/p>\n If mid- and upper-level managers hope to remain competitive, however, they will need to shake this old way of thinking. Here are three reasons why:<\/p>\n Guiding a data solution<\/a> through the innovation lifecycle, such as phases of product development, once meant years of uncertainty, uncountable people hours and massive investments. Today, innovation processes do not have to be so complicated.<\/p>\n What\u2019s changed? With cloud computing, it\u2019s now fast and relatively inexpensive to trial new tools and solution approaches.<\/strong> Many technologies are available ready-to-use with only a few clicks or pre-installed on containers. There is now an entire ecosystem of documentation and demo implementations available at no or limited cost. Open-source technologies often give direct access to the community of developers who created them.<\/p>\n In the past, engineers often dedicated themselves, either by choice or instruction, to one technology. Moving forward, the fastest way to advance your bottom line will be to have engineers explore and innovate on multiple technologies<\/a>. It will pay to support engineers\u2019 continuing education, send them to conferences, let them demo products, experiment with system disruptions \u2014 and see how quickly they can stitch together a better business approach.<\/p>\n Any classically trained data engineer would know the term \u201cKimball.\u201d This foundational model for data warehouse design was developed in 1996 by Ralph Kimball with co-author Margy Ross. Until recently, the Kimball was a widely known and often used data modeling technique \u2014 but now, it feels a lot like a dinosaur.<\/p>\n Why? You can now easily amp up processing power in the cloud<\/a>, meaning it\u2019s not necessary to optimize data models for compute.<\/strong> With tools like Power BI, Power Query and Analysis Services, analysts can explore data from any source. The data warehouse is no longer the only place to find useful data.<\/p>\nA Seismic Shift<\/h2>\n
\n
\n
Implications for IT Managers<\/h2>\n
1. Technology is streamlining innovation.<\/h3>\n
2. New models mean new opportunities.<\/h3>\n