Enterprise Data Warehouses, Are You Winning or Losing?
The biggest impact on marketing over the last 15 years has been the proliferation of data among new sources like digital and mobile that provide insight into behavior and social activity. For many companies, the challenge is the integration of databases to optimize understanding of the customer, assess value, target investments, and manage communications on products, prices, services and promotions so that you have the right offer at the right time in the right channel to the right customer.
Seems complicated? It is. When you consider sources of data and the fragmentation that still exists among data specialists, the ability to facilitate an enterprise warehouse is more important than ever. POS, e-commerce carts, e-commerce browsing, mobile apps and mobile browsing, in-store purchases and visits and geo-location of customers, products, prices, and services is a complicated mash of data, systems and standards. Smart companies seek integration and try to eliminate silos of data that prevent company performance from being optimal to the customer. For example, if I am in the market for a new computer, I might browse a web-site on a tablet or PC and the experiences are vastly different. The mobile site might capture more data including an app that is raiding my mobile device and taking all of my information about contacts, and other browsing behavior. This compares to a web-site visit where my specific browsing is logged and tracked with cookies and sessions and shopping cart behavior. Tracking my preferences on price, promotions, shipping, payment methods and more is an integral part of easing the purchase scenario. Also looking at the frequency of communications I receive and the messaging strategy, am I a consumer, a business, both, and how might I uses the new device socially, professionally and mixed. When databases for these systems are not integrated, then the communications all vary based on the information at hand, unifying the data is an essential task in order to improve the customer experience which will lead to greater utilization of your brand.
With the rise of Enterprise Data Warehousing, what is a small or medium sized business to do? The costs of warehousing is not trivial but it is not enormous either. The cost of personnel and software can be prohibitive even if you are a $20MM a year business. But the rewards are great so the question remains, how do you get it done?
Storage is ridiculously cheap. A terabyte of data used to be ” a lot” of data. Now TB hard drives are standard on virtually all serious personal computers and the proliferation of solid state drives (SSD) help increase speed and access to data. Likewise, RAM is cheap and while a TB of RAM may not be in your future, multiple GB of RAM on current processors is something to think about. The greater the RAM, the more your applications can run and store in-memory what they are doing which will lead to greater speed. How your data is disseminated in the data warehouse will also impact your processing. Highly normalized data structures have become unnecessary as tables have become larger and deeper and faster at the same time. Cube technology also enables data to be highly indexed and compressed so that it is made available very rapidly instead of watching your browser “think.”
This is an image of a server at CERN, the European Organization for Nuclear Research. CERN has been working on the Higgs Boson project and is now turning their considerable computing power toward the war on cancer. Massive databases and huge analytical processes can drive break-throughs in cancer research. Likewise, your datawarhouse can uncover customer patterns and provide ideas for new products and services.
Data Warehouse Structure
How data is structured and integrated is deeply important. The quest for the “universal customer id” is inhibited particularly in the United States because the universal id is a Social Security Number. But since storing SSN is risky many businesses opt not to do it. Telephone numbers and emails are so variable that you can mis-assign data to customers when they drop a phone number and get a new one, no one tells your data warehouse that has occurred. Likewise, people move, mobility in a country like the United States is highly accessible to all socio-economic levels so tracking by address is continually difficult. But in the end, it’s a combination of sources of data that enables your business to have a “universal id” that you can use to track your customers and integrate their data today, in the future and from the past. It’s amazing to me how many companies approach me with communication as if we have never done business together. Silos of information create the error of “I don’t know a damn thing about you, but will you buy our stuff, we are really good.” Building the structure to your database requires a universal id that you manage with great care and seek to integrate current, past and future data.
Data Warehouse Performance
So collecting data and storing data are integral parts of the warehouse. But when you wake up and have 15 TB of data all of a sudden your SQL queries are taking time that you simply do not have in order to react to the market. Staging of data is popular where current and previous behaviors and interests are aggregated and pushed forward to interface with customers and this data can be refreshed on a daily or weekly basis. But many businesses are turning to the cloud where the dynamics of pricing are making massive warehouses more affordable and accessible. Solutions like Google Cloud and Amazon Web Services enable businesses to turn computational servers on and off programmatically so that they are not paying for idle server time. Storage is now delivered on Solid State Drives and flash drives that allow data to be pulled instantaneously when properly indexed. Companies like Snowflake have taken this solution a step further and physically isolated storage from computational power to enable business to optimize spend on processing and to reduce hurdles in processing by enabling massive parallel processing in multiple instances.
Data Performance relies on computing power, speed of storage and speed of transfer. With the introduction of fiber optics, quantum computing and accelerated storage devices, the ability for business to push data into actionable information is rapidly increasing.
When it comes to business intelligence and getting data into the hands of the communicators and marketers and developers of products, prices and promotions, big databases can be as much of a hazard as a help. To improve performance, BI should be separated from the main data warehouse to create new rapid delivery structures including Column Index structures and Non Clustered Index structures that allow data to be indexed and stored and compressed in such a manner that queries move quickly to the intended data in the rows. Think of it as super-intelligent databases that predict what you are typing, kind of like typing your address into Google and Google fills in your complete address after a few characters are entered. Having data primed for analysis, aggregation and distribution is the key performance of your enterprise data warehouse. Emerging tools like SiSense are enabling such speed in dashboarding and reporting that the size of files is mattering less and the range of information is more the focus. Keep in mind – your BI provides the basis for your understanding of customer needs, wants, behaviors, life-cycle stages, social preferences, etc. – it has to deliver information for you to take the next and final step.
The proliferation of data has made it increasingly difficult and disadvantageous to try and use simple metrics to determine what a customer is likely to respond to. Instead, advanced analytics cluster customers into homogeneous groups based on geo-location, social behavior, product usage, pricing, promotion, browsing behavior, economic status etc. A customer profiling structure for management typically entails creating 6-8 meaningful clusters of customers with distinct behaviors. For example two customers both spend $1000 a year with you, one does it through a series of 10 purchases and the other does it through 2 purchases. Do you communicate the same way to both customers? Of course not – they behave differently. Further segmentation on social, lifestyle, economic status, access to products and service can all be directional in how to serve two customers who “spend the same” but spend very differently. As you build your enterprise warehouse, statistical analysis of data is going to overcome “gut feelings” and basic metrics. This is how you know you are on the right path, the predictive data becomes so much more accurate the more you “feed” the algorithms. So predictions based on one or two of your data sources versus 5-10 data sources will be completely different and lead you down the path of new products, highly targeted promotions, offers and prices and the path to being relevant to the customer because you “know” them. Think Amazon and their understanding of your Next Most Likely purchase. This comes from mountains of data on your browsing and buying behaviors, social status, economic ability, etc. The other big piece coming to advanced analytics are advances in artificial intelligence. Analytics today are executed and delivered through models that are built and deployed by humans and refined after results are learned and added to the model. AI is taking that to the next level by refining the model on an on-going basis using real-time data that is placed next to the consumer to customize their experience. This is the foundation of “BIG Data” and it’s a solution that is becoming more accessible to small businesses that want to engage their customers to earn higher sales and profits through total customization.
So enterprise data warehousing is here to stay. What you build and how you build it are integral to your future success and defending your brand and extending your brand to consumers and easing the consumption process. This is universally true for business marketing to consumers and businesses alike. Imagine the buyer in a business, he/she is a human, they have emotions, intelligence, experience and relationships. When you collect and aggregate data on your buyers you have profiles that emerge on their style of buying, what they respond to and do not respond to and even how to negotiate a business transaction. If you are a small or medium sized business, your enterprise warehouse is a work in progress and completely doable if you are proficient with data or you have the right partners to help you. You may be able to participate in aggregate enterprise warehouses where your data is partially mixed with other businesses so that you can leverage universal customer identification and gain insights into customer performance from experiences the customer has had with other businesses. The important thing to remember here is to look at what you are trying to provide your customers, your ability to invest in your customers, the lifetime value of a customer and the costs of your enterprise warehouse. The companies that know more, tend to sell more and to be more profitable.