Throughout January, we’re bringing you thought leaders of all types in our brand-new podcast, The Big Data Perspective. After all, 2017 is your year to become an expert on the big data world! Be sure to subscribe to our blog to get updates as soon as they’re published.
Today we’re featuring Mico Yuk, CEO of BI Brainz and the popular Analytics on Fire community, and author of Data Visualization for Dummies. She’ll provide us with her perspective on what’s happening in big data, and how that’s affecting the business intelligence world.
Transcript, lightly edited for clarity:
Andrew Brust: We’re moving towards a new era in data analytics and, of course, the business intelligence world has been watching avidly. I’m happy to have with us Mico Yuk, CEO of BI Brainz and the popular Analytics on Fire community and the author of Data Visualization for Dummies. She’ll be giving us her perspective on what’s happening in big data and in BI. I’m Andrew Brust and this is The Big Data Perspective. Mico, happy to have you here.
Mico Yuk: Hey Andrew, exciting times. This is way overdue.
Consolidation Between Big Data Analytics and BI
Andrew Brust: Awesome, yeah. I had the pleasure of being a guest on a podcast that you host, so I’m happy to have the roles reversed a little bit here. Makes it easier for me, I think. I’ve got some questions that are on my mind and just a sneaking suspicion that you’ve got some strong opinions about them. If it’s okay, I’m going to launch right into the first one: over the years we’ve seen a consolidation between big data analytics and BI. I’m wondering where you think this will lead us and what do you think BI veterans should do to adapt to this changing world?
Mico Yuk: You know, that’s a million-dollar-question, Andrew. You know, it’s interesting. You would think that big data analytics and BI should be in the same world but, ironically, they’re not. I think that where this is leading to is that the biggest silos that we have right now is BI legacy systems that don’t, to me, integrate very well into the big data technologies. When I say integrate, I’m not talking about the ability to connect to Hadoop or NoSQL storages or whatever the flavor the month is for big data. I think just integrate, as a whole, to most importantly provide the output, the level of advanced analytics that customers actually need to make sense of their data. My hope is that these will come together easier. They’ll enable the ability for more companies to take advantage of advanced analytics and then get real time insights, better performance management metrics.
Now the other part of this is, in terms of the people that are doing this is I also think that the skill sets that need to handle this also need to evolve to enable this, right. Technology is one end. Okay, make the technology speak, but in the people do too, to realize that they serve a unified purpose to the customer as opposed to really operating these siloed operations. That’s what I want to see. I want to see more integrated systems, easier to handle, not two separate conversations. It’s not Hadoop versus the data warehouse. It’s, we’re working together. We’re blending amazing data. Both old data, new data, is getting faster, better, easier and we’re able to provide better real time insight. That’s what I want to see.
Why Is There So Much Hype Around Big Data?
Andrew Brust: I’m with you there, and you know this. I’ve been puzzled by the segregation between the two communities because really, they’re looking at the same kinds of questions, the same challenges. Ultimately, if BI technology was a little stuck, then big data technology was really a response to that, to move things forward together, really. In general, keeping them in separate little cages just doesn’t make a lot of sense. Along with that, we’re constantly hearing about the promise big data holds. By the way, the sense of deja vu here is immense. We could swap out big data with BI going back fifteen years ago. We’re hearing about the promise. We’re hearing about the potential to increase profits, to improve processes, even to save lives. Clearly, that’s kind of hype-driven. I’m wondering, if we’re on the subject of hype, how you explain why the hype is there and what the industry can do to move beyond hype and start getting really constructive about putting this into play.
Mico Yuk: Well, first of all, Andrew, I kind of like the hype. I know it sounds really weird, but let me explain myself. I actually like the hype around big data because you know my background, and I brought up many times on our podcast, where I started as a senior research analyst, which was a data scientist. I took the statistics classes, which were, I guess, data science classes. No one actually knew what you did, so to be honest with you, I like the hype. However, I think that there is a lot of great opportunity, but I think the real issue is that we’re seeing in the industry in terms of response, is that while enterprises may have the money and technology, a lot of times you’re hearing, we’re not ready. Have you heard that before, Andrew? We’re not ready.
Andrew Brust: In some sense, I feel like I’ve been hearing that through my whole career, but go ahead.
Mico Yuk: Do you agree it’s still happening as 2016, even with all this new big data technology?
Andrew Brust: Absolutely, yes.
Mico Yuk: They mean we’re not ready from a skill set level. We’re not ready to take action on this data. We don’t necessarily trust it. We don’t understand it. Not ready is very loaded when it comes to enterprises. I think that even though there’s a lot of hype and potential with big data to do amazing things, a lot of companies aren’t ready and that’s something that really has to evolve. Back in the day, the issue was, we didn’t have the technology. We didn’t have in-memory. You had to do indexes. You had to do all these schemas and all this craziness to get your data. Guess what? Technology is there.
Here’s the problem, you brought it up. Mindset. People’s mindsets are not, to me, in a place today where they’re willing to establish and embrace the opportunities that big data brings. I really think it’s a couple of different areas, but if we could start to unfold what the term “not ready” means, which I’ve been hearing over and over again when it comes to big data, then I think we have some opportunities. It’s just exciting to see that, when you talk about big data, people get excited. It’s sexy. Here’s all the things we could figure out before it happens. It’s kind of like this crystal ball. This gypsy. I can create a universe. I can tell you what things need to be and I can tell you how to get there. Then you go great, let’s do it. Then it falls apart. We’re not ready. To be honest with you, I think it’s not big data. It’s not the hype. It’s really a mindset issue.
What’s Behind Failed Big Data Projects?
Andrew Brust: All right, well that may segue nicely into another question, but that other question may give me an opportunity to pursue a little follow-up from the one we just covered. Let me get that question out there and then let’s see if I can articulate the follow-up. The question kind of surrounds the issue of project failure. According to a survey published by Dimensional Research, although a hundred percent of participants acknowledged that data initiatives are important, the vast majority, and we can quantify that at 88 percent, have had failed projects.
First of all, what do you think is the reason behind these failures? I kind of feel like you’ve already unwittingly spoken to that. What can people do to encourage success? By the way, let’s just look at the premise here, especially as it pertains to the readiness that you were talking about. A hundred percent of participants acknowledged that data initiatives are important. To me, that sounds like readiness.
Mico Yuk: It does.
Andrew Brust: Meanwhile, almost all of them have had failed projects, so that seems like a lack of readiness. So what does readiness really mean? Can we get to the heart of this conundrum?
Mico Yuk: This is a really great question and I’ll get to where I think the solution comes in, but you brought up something before the call that really struck me and we were giggling. You said that, and I’m giving you credit for this because I think it’s powerful, you said that the idea about big data is it’s supposed to be the evolution from BI. We all know BI is stuck. The idea is that big data was supposed to come in and be the silver bullet, but here we are with, big data is 88 percent failure? BI is 80 percent. We’re back to the same fundamental problems.
Andrew Brust: Yes, we are.
Mico Yuk: You remember when big data came out, and you brought this up and I agree, it was like, this is it. We’re going to get away from IT and get away from BI. We’re going to get these new systems with NoSQL. You remember that, a couple years ago? I mean, it was exciting.
Andrew Brust: The euphoria was thick enough to cut with a knife, yes.
Mico Yuk: Right, and it was great. It looked all good and removed all the crazy barriers to IT and here we are again, at failure. It’s a good old saying. Same people, same results. I know that sounds really bad, but it’s a good lesson. You can change the tools. You can throw as much money as you want at it, but unless you change out some of the foundation … Maybe it’s the people using the tools. I’m trying to be nice about this. It’s not going to change. I think there’s a couple of high-level issues that we see, and I’ll kind of break them down before I get to solution mode. Things like waterfall and approaches, they don’t work, my opinion. Kimball, love the guy, great, built data warehouse, but I think it’s data no-house right now. Kimball is kind of becoming a bit legacy in itself.
I think from a BI standpoint, the failures translate over on a people standpoint. I already spoke about that. I think that the challenges remain the same. It’s that vendor Venn diagram, people, process, technology. This is my favorite line, Andrew. We’ll do something phenomenal and we’ll hear, “Can I get that to look like Excel?” It’s my favorite line.
Andrew Brust: It’s great that you mention that because, of course, Datameer’s user interface is very much modeled after that of a spreadsheet, so you’re vindicating us, even if you don’t realize it.
Mico Yuk: But I think you’re smart, because what you’ve done is, even though you’ve changed the back end, you’ve kept the front end the same. Because, as you know, what I see, when it looks different, that’s where fear comes in. Datameer is very smart and I’m sure you guys are very successful at doing that. I use that term, “I just want to see my report like Excel,” which I actually heard two days ago from one of our customers, just to explain to you where people are at when it comes to data.
I brought up one more thing as well, which is, to me, the number one issue with these projects and it’s the same thing to me with BI, though there’s other issues, is that data is subjective. I look at data. I see purple. You look at data. You see orange. I can try to convince you it’s purple, but you may see a little shade of purple, but you see orange. That issue, I’ll be honest, I have no idea how to remedy that. The thought process around that was if we have more data to support it, we’d be sure the answer was correct, but that doesn’t solve the human EQ problem which is, what’s going on in my brain to make me feel security that this is correct. I don’t want to get too deep into the weeds there in terms of the cognitive processes that are affecting us, but I truly feel that there is a whole different element beyond, necessarily, people on the surface of processes that have to do with data. There’s these ingrained mentalities around what you need to trust data and I think that is something that needs to fleshed out better. What does trust mean, whether it’s big data or small data?
Andrew Brust: It’s interesting because, to stick with your analogy there, even if we saw the same color, we’d probably have different favorite colors, which can probably color (no pun intended) our interpretation, influence our interpretation of what color we’re seeing. Then, if I can tie that back a little to something you were saying before, you were kind of alluding to. In the BI-only era, we had the issue of latency. We had the issue of the waterfall approach that you talked about, which is that we had to design a perfect star schema, be that for the data warehouse or for the OLAP cube, get everyone to agree with it, and only then could we move forward. That obviously doesn’t work well overall, and then where it does work okay is mostly going to be with the top-down centralized approach. Maybe what typifies big data and NoSQL, for that matter, is the idea that we can be less formal about that and we can get to the analysis. We can cut through the red tape and get to the analysis more quickly without having to obsess over the perfect schema and the perfect way to look at the data. Now that that problem’s solved, we did expect that the success rate of the projects would go up.
Mico Yuk: No.
Andrew Brust: If anything, it’s flat or it’s gone down a little bit. It seems like what we’re saying, is yeah, technology problems were part of the challenge, but I think what you’re saying is that social or, maybe I can say, political problems within the organization still have to be addressed. You still need consensus. You still need people to have a common goal and purpose and impetus behind a project. If people aren’t predisposed to thinking that way, then it’s hard to get beyond the proof of concepts and the skunkworks things and get this technology to just be part of the bread and butter of what the business uses to get its job done. First of all, do you agree with that, or is there a part you disagree with?
Mico Yuk: Yeah.
Andrew Brust: Second of all, what can we do to get there? It seems like we’ve been asking these questions now actually for twenty years or more. We’re getting old here.
Mico Yuk: I’m going to say something very crazy. Andrew, you know me, I love to dream. I’m a data visionary. My creative side of my brain is probably a little bit bigger than the analytical side in some sense. I truly feel that we have to get to the Google model. Let me ask you a question. When you go into Google and you type in an inquiry and you get the wrong results, what do you do next?
Andrew Brust: What do I do next? I probably ask the same question a different way.
Mico Yuk: Exactly, but you never assumed that Google was wrong, did you? Did you question their algorithms?
Andrew Brust: No, I probably questioned my own articulation of what I was searching for.
Mico Yuk: Exactly. BI is the exact opposite. When something goes wrong in BI, what’s the first thing that you do? You go, the number is wrong. Call Andrew. Let me see the data.
Andrew Brust: Yeah, blame the consultant.
Mico Yuk: Right, and so I feel that with BI and big data, the happy medium is to get to the Google model. You know why, because again, we trust and we assume we’re wrong, not the machine. That’s why I think the future of this and where this is going to evolve has to be this machine-learning concept. I know I have jumped the gun a little bit, but I truly feel it has to become easier and simpler and data has to teach itself, get smarter on its own, and just provide the answers. I think anytime we’re left up to subjectivity on that level, multiple human beings with different sides, number one, we’re never going to get to the truth. That’s why BI failed. That central source of the truth is never really the truth. It’s actually full of opinions and lies. Come on, BI, it’s a bunch of lies. Nobody believes it. In order for us to get there, I think we have to eliminate that subjective quotient. That’s my opinion.
Andrew Brust: That’s interesting, see. I think what you’re saying is people take Google or any search engine, really, or just the capabilities of it, people take it’s validity on authority.
Mico Yuk: Yes.
Andrew Brust: And nobody’s doing that with BI and probably big data, so we have to get to the point where that technology has the same level of trust and people take it on authority and give it the benefit of the doubt rather than attacking it on all surfaces where doubt may exist.
Mico Yuk: But Andrew, wouldn’t that be great? You get a number out of your big data analytics system or your BI system and instead of questioning it, you assume, maybe I didn’t ask the question correctly. I mean, I hope I’m alive when it happens.
How Can People Enter the Big Data Career Field?
Andrew Brust: I hope I am, too. That’s a nice North Star to have, although I have to think through how feasible it’s going to be to get there, but at least we’ve identified, you’ve identified a laudable goal here. That’s a lot better than just feeling defeated that there’s really, there’s no way out of the square we’re in to the next one.
If we’re talking about that mainstream trust in competency, I’m going to contrast that with someone rather at the other extreme, which is the notion that people, in order to work with data technology and get to the point of good results, have to be highly specialized and go through an awful lot of training. What are your thoughts about and around universities that are establishing data science and big data courses and indeed, majors and areas of concentration? Do you think that’s a good way for someone interested in the space to get up and running? Or maybe you have other ideas, given what you were just articulating.
Mico Yuk: First of all, the programs have always been there. They’ve just been in the corner of the science department. Let’s be real. Data science used to be called statistics. Do you agree?
Andrew Brust: I do. I think it was maybe even in the mathematics department, not necessarily just computer science.
Mico Yuk: Math and science department. Math and science is usually tied, right? The first thing is that this whole data science revolution is a great marketing exercise. It’s allowed universities to expand their program that has a lot of interest. I took statistics classes back ten years ago. I think it’s always been there. It’s just now it’s under a sexier marketing tap that’s very attractive to millennials and to employers. It sounds better. That’s number one.
Number two, do I think that university is a great way to go? Absolutely. But I also think that because of the drive in this profession and need for this profession, that there’s so much, like MOCCs, these mega online courses. There’s so many books available, there’s thought leaders, that honestly, if you wanted to get started, there’s Coursera, there’s so much resources today that you can actually sit down at any level, whether in BI or IT or on the business side, and really ramp up on your own. I can tell you from experience, 10 years ago, that’s something that didn’t even exist. No one actually knew, like when I did this, no one actually knew what SaaS was. We talked about it. I think university is a great way to go. Getting started in an online course is a good way to go. I also think that just getting your hands dirty is a great way to go.
Now, the one thing that I do struggle with, however, is that, having data science by itself in terms of like, a silo, that skill set by itself, to me, is a part of the challenge we’re having in this industry. This is a little bit touchy because one should ask, well, do you think BI people should go to try to become data scientists, or should data scientists go to become BI people? Before I answer and people listen to this get ready to shoot me, Andrew, I’m going to put you on the spot. Which one do you think is easier?
Andrew Brust: I think it’s probably easier to get existing practitioners up on more recent skill sets, if we can get them past the psychological block that these new skill sets are so different and that they’re not worthy. I think that’s a big blocker.
Mico Yuk: I agree. Now, the reason I ask that again is because there’s this notion that these companies are just bringing data scientists and put them in some corner by themselves and somehow they’re going to pop out the next model that’s going to work. Then what ends up happening? They have to come over and beg to the same BI well that we do. You’re not going to get away from our systems. You’re one more guy in the line going, “Can I get the module? Can I get some data?” At the end of the day, we all have to feed from the same wells. There has to be that new breed of professional that comes out of these programs. I honestly don’t care what resourcing programs. What I want people to focus on is that there has to be a new BI hybrid professional who comes into the market, or is bred into the market or evolves into the market from what we have today, that can handle both sides. That is the missing gap.
Andrew Brust: That’s easier said than done, but I’m with you.
Mico Yuk: But is it?
Andrew Brust: Well, I think, again, I think it’s mostly psychology. I think the industry spends a lot of time telling people of one technology generation that they’re outdated. I think it’s going to be a lot more sensible for everybody, and I don’t just mean this in a touchy-feely way, that you should just be nice to people. I think that if we want to get past labor shortages and skill set issues and so forth, the constructive way forward is to uplift existing people who are in your organization, who already understand the technology context and already understand the business. Basically, they’re eighty percent of the way there. Jeez, help them get the next twenty percent to pick up a new skill set and then keep moving, rather than saying, well, we have to rip it, replace it, bring in brand new people and get them all institutionally acclimated from scratch. I don’t understand that. It seems like a masochist exercise to do that.
Mico Yuk: I don’t understand why we’re now treated like how we built the functional layer in BI. Remember when it used to be IT, just get requirements. Get it from the business. Now most companies have what we call the functional layer, which is a superuser who came from the business side who understands the technology. Scientists in BI need that exact same thing. If there’s a silo today, I think that the BI team needs to treat the data scientist like a customer and hopefully convert them into a superuser to get that middle layer, and then I would hope that the functional aspect right now today in BI has those people coming and inbreeding to take advanced analytics to the next level.
Sorry, I have this diagram in my head. I know that sounds a little bit convoluted, but that’s what I think needs to happen. They’re not over here. They’re our customer, and then they become that functional layer for us. They inbreed, take it to the next level, and they can better help us serve the customers. That has to happen soon. Did I confuse you?
Andrew Brust: No, you didn’t. You’re asking, and so am I, by the way, for an awful lot of common sense. Unfortunately, we seem to get caught up in politics that preempt the common sense from reigning supreme. I hope we can have some reform and get past that, because ultimately, I think we can get answers to the questions if we’re asking the right ones. Sometimes the premise of the questions we’re asking seems all confused.
Mico Yuk: But don’t you agree that would be the easiest way to inbreed them, Andrew, is to bring the data scientists first as customers to BI teams, then integrate them into the functional layer to make that smarter? You could say, look, we have BI now, which is the rear-view mirror, and then we have the front-view mirror, predictive PA, which is this advanced layer, and make them be a part of that delivery mechanism to the customer. It can’t be us versus them, Andrew. That’s not going to work.
Andrew Brust: Yeah, and by PA, by the way, you mean predictive analytics, right?
Mico Yuk: Yeah, I’m sorry, sorry for the cliche terms.
Andrew Brust: I’m always the acronym police. Although, if you switch it over, then I tend to be the offender. Yeah, I think what happens is when there’s new technology in a new scope that it engenders this idea that there needs to be a new elite of people of who focus just on that, and well, I think we’ve already talked about how destructive that can be. You’re right. The more efficient way about this would be if people with new skill sets were embedded and integrated with existing organizations and groups rather than saying we’re going to start our own country, because that seems to be the cause of a lot of inefficiency. We’ll see.
Mico Yuk: It’s competitive. I mean, we’ve walked into these companies and when you ask them what’s going on, oh, the CEO just hired three data scientists and they’re over there doing this so we need hurry up and get our predictive tools up here on the BI side so we can provide this service. It becomes a very, to me, unproductive competition.
Predictions About the Future of BI and Big Data
Andrew Brust: Totally agree. Segues into my last question, which begins with the statement that you’re very passionate about the BI and big data world. That passion’s been quite on display in this conversation. If you were to channel that passion towards prognosticating a little bit, what do you see, what do you foresee happening in BI and big data this coming year in 2017 and then the next several years. If you feel really frisky, maybe you could even talk about a full decade from now, but just as far into the future as you feel is useful talking about. What do you hope for and what do you expect?
Mico Yuk: Sure. I actually have five bullets and some of them are very selfish, Andrew, just again, because of my background from data science to BI and my frustrations. I’ll start with the first one. I want to see predictive analytics go mainstream. I don’t want it to be a little group in the corner. I don’t it to be these unicorns who are able to embrace it because they’re younger, faster, better. I want it to become mainstream. I see that being something that needs to happen very, very quickly.
On a selfish note, I also want the ability to access and use data to be as simple as being able to open up my iPhone and open a app. I don’t even want to do single sign-in anymore. I’m tired of that, okay. I want, literally, take my lazy nature around data and to be able to conform it to my needs rather than me having to do a bunch of work to get it to where I need it to be. I want massaging of data to disappear. I also want data, and this is going to be crazy sounding, I want it to just model itself, Andrew. I just want answers. I said it earlier. I just want answers.
That leads me to my next one where AI needs to take off. Artificial intelligence, to me, has to be and is the answer to a lot of these problems and data failures. The problem is, it’s been around for a long time. You have some good stats from the 1960s, and it’s never really taken off, but I truly believe AI and all the facts that come with it, I mentioned machine learning earlier, are truly the future of data, where we’re only focused on getting answers and let’s focus on what it takes to massage that data to try to get the answer that we feel, subjectively, is correct. We have to eliminate that subjective part of the equation. We have to eliminate, to me, the human manipulation. Sounds completely crazy, but I truly believe that’s the only way that they’re going to start to see real returns on this stuff.
Andrew Brust: It’s interesting to me that the data industry talks about how powerful machine learning is and predictive analytics is and how it’s going to change so many things we do and automate so many things. At the same time, we refuse to be introspective around our own challenges and say, hmm, if I have predictive analytics but I also have this task of cleansing the data and shaping the data and prepping the data, and that makes me stuck, why haven’t we connected those dots and said, okay, let’s actually apply our own technology on our own problems and get the data to model itself, as you were saying. It’s funny you mention that because I was thinking that just yesterday as I was going through a bunch of predictions from a bunch of industry leaders. They’re all talking pie-in-the-sky stuff about machine learning and then they’re also talking about the need for data scientists and how they spend 80 percent of their time on the data. Without any irony, they talk about both of those things together without talking about how one can address the other, so thank you very much for calling that out.
But I’m going to push you a little bit. You’ve talked about what you wish for. What do you expect we’ll have in the next year, and five years, and maybe ten years out?
Mico Yuk: I expect that there’s going to be a lot more data scientists, but to be honest with you, because again, some of this is controversial … I am putting myself on the line here to say I, Mico Yuk, look forward to a new breed of hybrid professionals for BI and big data. I don’t know what the professional is named. I didn’t want to come on here and start coining a new term, but I do want to say that I want kids that are coming out of school to be ready and thinking about advanced analytics the way that we think about an Excel sheet. It needs to be second nature, and that’s something brand-spanking new in your job. Remember, I made an analogy about like coaching a team. A new coach comes in. You bring in the rookies on the front layer. You have your veterans that maintain that second layer.
I mean, this is not a data scientist, though. I want to be very clear. This is very separate, a new breed of BI professionals where advanced analytics is the norm. I think that’s a start. It’s not new. It’s not different. It’s simply the way that we think. We always think ahead. We always think about modeling and getting smarter. Today, to me, these university and programs are not there. That, to me, is one pivotal part of the equation that’s missing, and it doesn’t exist today, in my opinion. That has to happen, and I feel that that’s going to happen soon.
Andrew Brust: That’s a great note to close on. I think is that you’re saying we shouldn’t fetishize the technology as its own specialization, but what we do need is to have a widespread literacy around it, just as we do now have that literacy around spreadsheets that, of course, didn’t really exist in the seventies and early eighties. We got that to be mainstream. We got that to be ubiquitous. It sounds like what you’re saying is we need to have that same ubiquity and readiness and literacy with analytics.
Mico Yuk: Absolutely.
Andrew Brust: I think that’s a great spot to end on, although I wish we could continue for another hour or two. We’re going have to conclude there. It’s always good to leave your host wanting more, and I hope we’ve left the audience wanting more. If we have, that’s a sign we’ve done our job well. Mico, thank you so much for being here. It was a pleasure.
Mico Yuk: No, this is fun. Obviously, we could go on and on, but this is great. Andrew, I hope we have a follow-up sometime next year again, to see where the industry actually is, because I’m trying to tell you, if we’re still talking about big data and hype, me and you need to find a different day job.
Andrew Brust: Excellent. All right, thanks again for coming.
Mico Yuk: All right, thank you Andrew. Awesome.