The ‘Sweet Spot of Big Data’ May Contain a Sour Surprise for Some

I’ve been observing a rather distasteful trend in big data for the enterprise market over the past 18 months that has reached the point of wanting to share some thoughts despite a growing mountain of other priorities.

As the big data hype grew over the past few years, much of which was enabled by Hadoop and other FOSS stacks, internal and external teams serving large companies have perfected a (sweet spot) model that is tailored to the environment and tech stack. Many vendors have also tailored their offerings for the model backed up with arguably too much funding by VCs, dozens too many analyst reports, and a half-dozen too many CIO publications attempting to extend reach and increase the ad spend.

The ‘sweet spot’ goes something like this:

• Small teams consisting of data scientists and business analysts.
• Employing exclusively low cost IT commodities and free and open sources software (FOSS).
• Targeting $1 to $5 million projects within units of mid to large sized enterprises.
  
• Expensed in current budget cycle (opex), with low risk and high probability of quick, demonstratable ROI.

So what could possibility be wrong with this dessert—looks perfect, right? Well, not necessarily—at least for those of us who have viewed a similar movie many times previously, with a similar foreshadowing, plot, cast of characters, and story line.

While this service model and related technology has been a good thing generally, resulting in new efficiency, improved manufacturing processes, reduced inventories, and perhaps even saved a few lives, not to mention generated a lot of demand for data centers, cloud service providers and consultants, we need to look at the bigger picture in the context of historical trends in tech adoption and how such trends evolve to see where this trail will lead. Highly paid data scientists, for example, may then find that they have been frantically jumping from one project to the next inside a large bubble with a thin lining, rising high over an ecosystem with no safety net, and then suddenly find themselves a target of flying arrows from the very CFO who has been their client, and for good fundamental reasons.

As we’ve seen many times before at the confluence of technology and services, the beginning of an over-hyped trend creates demand for high-end talent that is unsustainable even often in the mid-term. Everyone from the largest vendors to emerging companies like Kyield to leading consulting firms and many independents alike are in general agreement that while service talent in big data analytics (and closely related) are capturing up to 90% of the budget in this ‘sweet spot’ model today, the trend is expected to reverse quickly as automated systems and tools mature. The reaction to such trends is often an attempt to create silos of various sorts, but even for those in global incumbents or models protected by unions and laws like K-12 in the U.S., it’s probably wise to seek a more sustainable model and ecosystem tailored for the future. Otherwise, I fear a great many talented people working with data will find in hindsight that they have been exploited for very short-term gain in a model that no longer has demand and may well find themselves competing with a global flood of bubble chasers willing to work cheaper than is even possible given the cost of living in their location.

What everyone should realize about the big data trend

While there will likely be strong demand for the very best mathematicians, data modelers, and statisticians far beyond the horizon, the super majority of organizations today are at some point in the journey of developing mid to long-term strategic plans for optimizing advanced analytics, including investments not just for small projects, but the entire organization. This is not occurring in a vacuum, but rather in conjunction with consultants, vendors, labs and emerging companies like ours that intentionally provide a platform that automates many of the redundant processes, enable plug and play, and make advanced analytics available to the workforce in a simple to use, substantially automated manner. While it took many years of R&D for all of the pieces to come together, the day has come when physics allows such systems to be deployed and so this trend is inevitable and indeed underway.

The current and future environment is not like the past when achieving a PhD in one decade will necessarily provide job demand in the next, unless like everyone else in society one can continue to grow, evolve and find ways to add value in a hyper competitive world. The challenges we face (collectively) in the future are great and so we cannot afford apathy or wide-spread cultures that are protecting the (unsustainable) past, but rather only those attempting to optimize the future.

In our system design, we embrace the independent app, data scientist, and algorithm, and recommend to customers that they do so as well—there is no substitute for individual creativity—and we simply must have systems that reflect this reality, but it needs to occur in a rationally planned manner that attacks the biggest problems facing organizations, and more broadly across society and the global economy.

The majority seem frankly misguided on the direction we are headed: the combination of data physics, hardware, organizational dynamics and economics requires us to automate much of this process in order to prevent the most dangerous systemic crises and to optimize discovery. It’s the right thing to do. I highly recommend to everyone in related fields to plan accordingly as these types of changes are occurring in half the time as a generation ago and the pace of change is still accelerating. At the end of the day, the job of analytics is to unleash the potential of customers and clients so that they can then create a sustainable economy and ecology. 

What should be the role of government with regard to disruptive IT?

This discussion I started in ‘Disruptive IT’ group at LinkedIn may be of interest:

What should be the role of government with regard to disruptive IT?

On role and title of Chief Data Officer

I left an extensive comment on a discussion surrounding the role and title of Chief Data Officer (CDO) over at Forrester Blogs by Gene Leganza, so thought I would share it here on our own blog (below).

~~~~

Gene

CDO reminds me of CKO more than CIO — and also suffers some of the same challenges as head of BI in the blog by Boris in the job description. Most of the arguments are valid until we take the entire organization into view and that’s where I see problems.

Anytime this discussion on a new officer comes up it tends to rise from the desires, need to have more influence in the org, and aspirations of the specialist (and their ecosystem) rather than the need of the organization. Similar to a strong EA for example, what typically matters is whether the CEO (and board) is competent or not, paying attention to how the organization functions or not, and whether the culture and relationships are collaborative, combative, or even apathetic.

Unfortunately, one impact we have observed with the CIO and a few CKOs was confusion over the title officer by the entire organization, including those holding the title, when they suddenly became obsessed with their own power rather than service to others. Whether BI, EA, or CKO, in some cases we observed quite strong individuals who were driving critical value to the organization, but reporting to an infrastructure CIO who didn’t understand much at all about business or organizations, or worse in a few cases simply concerned with protecting their own turf rather than the mission of the organization and customers.

Generally speaking I think it’s wise to allow the brand of the individual rise up in the organization based on functionality, knowledge, and contributions to the organization–rather than yet another title ending with officer. Indeed, often has been the case when a person with a more humble title has had more impact, even in aggressive and highly competitive cultures, particularly when they are wicked smart and wise.

One problem rarely discussed is that corporate officer in many companies has real meaning in terms of authority, responsibility and legal accountability whereas in most cases the job description title of officer does not, creating confusion internally and externally (I am thinking now of one giant tech company in particular where titles have been a disaster to everyone but the CEO who hands them out).

I personally prefer scientist over officer and I’m not terribly fond of scientist (except when used properly to describe a true scientist) due to the disconnect in meaning and culture between science and business that is too often manifest in organizational dysfunction. I see functional roles more of a master craft person who may lead a small team, but is more interested in the value of their work than career aspirations to one day become a CEO, or even to lead large numbers of people where internal and external politics tend to rule. In fact it’s been my experience that the strongest functional people do not have any desire to hold the title of officer.

So my fear based on previous experience and observations is that in the rare highly functional organization the role and title will also function well–regardless of what we call it, but in the norm it may do more harm than good, in which case the stronger professional will likely either avoid the organization to begin with or move on at the earliest opportunity. .02- MM

New Year Review 2013: The Importance of Triple Win Ecosystems

Looking Back on 2012

Among the positive trends we observed in 2012 include relatively strong continued adoption of richly structured data across all major industry clusters. A significant portion of senior managers have just recently engaged in an attempt to seriously understand how best to optimize structured data for their organizations and partners, with an exceptional minority now experiencing big aha moments. Until recently semantics was limited primarily to R&D, and analytics was restricted to a very few people in the organization with a fairly limited scope. A growing minority are finally connecting the dots and incorporating semantics into their ‘big data’ analytics strategy.

We changed a few global strategic plans with our Kyield pilot presentations in the past year as organizations began to realize that what seemed futuristic a few years ago is now executable in near real-time. However, what is still missing is a competitive ecosystem that could be called upon to serve the specific needs of customers in a triple win manner.  As is often the case with emerging technologies– individuals, small teams and companies engage due to a combination of motivating factors that includes independence, compensation, and to change the world for the better, but in order to achieve much in the enterprise market we need fully functional ecosystems.

A combination of assets must be offered in a highly efficient and credible manner, including very experienced management, superior technology, deep technical talent, appropriate financial structure and professional services, which is usually far more than even the largest companies can provide on their own. So job one for the emerging semantic enterprise community is to learn how to work together towards building a functional ecosystem, which means creating and offering mutual value while serving customer needs in win/win/win scenarios.

We held discussions with a few companies interested in partnering who underestimate the risk they face and overestimate their power, some of whom apparently don’t understand how to form mutually beneficial partnerships. In Kyield’s partner program we will only consider triple wins. And BTW, don’t expect anything but obstacles thrown in the way from the trillion dollar global integration machine, which is a mature, highly sophisticated ecosystem with unlimited funds that is threatened by independent standards and lower TCO. Customers have a special responsibility to assist the emerging semantic ecosystem as they are the primary beneficiaries of the technology and have a very important strategic interest in helping to ensure that the semantic ecosystem becomes more viable and sustainable.

Expectations for 2013

An interesting event recently occurred at a leading bank that could be telling for the near future with respect to semantics and data standards in 2013 and beyond. The bank recently announced large layoffs that included 25% within IT. The bank previously invested heavily in internal proprietary systems and are now moving towards standards where they expect to see a significant ROI with speculation that other leading banks will follow. Banks were late to semantics and obviously some proved that their proprietary risk management and governance systems were systemically dangerous, but we’re seeing a positive trend now. These layoffs may sound like bad news, but we should remember that the primary role of leading banks is to lend modestly leveraged capital to others, which creates far more jobs in a more diverse manner than when employed internally at a bank for proprietary systems and integration work. This is an important trend toward higher efficiency in a more economically sustainable manner thanks largely to adoption of independent standards. When combined with proper governance and advanced analytics throughout the financial system, this trend can become powerful indeed.

The big question in 2013 more broadly is to what degree the EU and U.S. can get their long-term fiscal house in order. The macro economic environment can easily dominate technology adoption and business creation. There is a direct relationship between balanced federal budgets, job creation, and sustainable economic progress. Clearly the U.S. culture is in a deep state of denial relative to sustainable economics reflected in a severely dysfunctional political system, and it’s having a strong negative impact even for those who offer an exceptional ROI.  In the case of Kyield, for example, the macro environment has resulted in some customer prospects in otherwise strong organizations freezing new projects until political and fiscal uncertainty is resolved, which in turn of course deters any prudent entrepreneur, investor or other decision maker from taking risk that may have seemed rational in the past in a much different macro economic environment. Since those in the health management business of elephants don’t have the luxury of being trampled more than once, Kyield plans to continue with our lean organic hybrid approach in partnership with customers and partners and leave speculative build outs to those who enjoy rolling the dice. 

I am confident that in 2013 we’ll see the competitive gap continue to widen between those who adopt more advanced, efficient, intelligent systems, and those who don’t, especially new systems that reduce lock-in and improve interoperability resulting in a sharply lower TCO. There is a strong compounding effect that occurs with adoption of rich data standards containing more integrity with advanced knowledge systems and real-time analytics, and it’s becoming increasingly more difficult to deny as organizations that master this technology pull ahead of competitors. Smart adaptive people working with smart adaptive computing is a powerful combination not to be denied.

To good health in 2013 for you, family and organization. – MM

Yield Management of Knowledge for Industry + FAQs

I decided to share this slightly edited version of a diagram that was part of a presentation we recently completed for an industry leading organization. Based on feedback this may be the most easily understandable graphic we’ve produced to date in communicating the Kyield enterprise system. As part of the same project we published a new FAQs page on our web site that may be of interest.  Most of my writing over the past several months has been in private tailored papers and presentations related to our pilot and partner programs.

I may include a version of this diagram in a public white paper soon if I can carve out some writing time. If you don’t hear from me before then I wish you and your family a happy holiday season.

Kind regards, MM

Kyield Partner Program

Five Types of Relationships

Kyield Partner Program: An Ecosystem for the Present Generation

We’ve been in various discussions for the past couple of years with potential partners, finding many interests in the industry to be in direct conflict with the mission of the customer organizations. Most of our discussions reflect a generational change in technology underway that is being met with sharp internal divisions at incumbent market leaders that result in attempting to protect the past from the present and future. The situation is not new– it is very difficult for incumbent market leaders to voluntarily cannibalize highly profitable business models, particularly when doing so would threaten jobs, so historically disruptive technology has required new ecosystems (MS and Intel in the early 1980s, Google in the late 1990s, RIM, Apple app stores in the 2000s, etc.).

So due to the platform approach to Kyield and the disruptive nature of our technology to incumbent business models, and resistance to change in industry leaders–despite pressure from even their largest customers, we determined quite some time ago that it may require building a new ecosystem based on the Kyield platform, data standards and interoperability. The driving need is not just about the enormous sum of money being charged for unproductive functionality like lock-in, maintenance fees, and unnecessary software integration–although this is an ever-increasing problem for customers, or even commoditization and lack of competitive advantage–also a very serious problem, rather as is often the case it comes down to a combination of complexity, math and physics.

Not only is it not economically viable to optimize network computing in the neural network economy based on legacy business models, but we cannot physically prevent systemic crises, dramatically improve productivity, and/or expedite discovery in a manner that doesn’t bankrupt a good portion of the economy without data standards and seamless interoperability. In addition, we do need deep intelligence on each entity as envisioned in the semantic web in order to overcome many of our greatest challenges, to execute advanced analytics, and to manage ‘big data’ in a rational manner, but we also need to protect privacy, data assets, knowledge capital and property rights while improving security. Standards may be voluntary, but overcoming these challenges isn’t.

So we’ve been working on a new partner program for Kyield in conjunction with our pilot phase in attempt to reach out to prospective partners who may not be on our radar that would make great partners and work with us to build a new ecosystem based not on protecting the past or current management at incumbent firms, but rather the future by optimizing the present capabilities and opportunities surrounding our system.  We hope to collectively create a great many more new jobs than we could possibly do on our own in the process–not just in our ecosystem, but importantly for customer ecosystems.

We’ve decided for now on five different types of partner relationships that are intended to best meet the needs of customers, partners, and Kyield:

• Consulting
• Technology
• Platform
• Strategic
• Master

For more information on our partner program, please visit:

http://www.kyield.com/partners.html

Thank you,

Mark Montgomery

Founder, Kyield

Converting the Enterprise to an Adaptive Neural Network

Those tracking business and financial news may have observed that a little bit of knowledge in the corner office about enterprise architecture, software, and data can cause great harm, including for the occupant, often resulting in a moving van parked under the corner suite of corporate headquarters shortly after headlines on their latest preventable crisis. Exploitation of ignorance in the board room surrounding enterprise computing has become mastered by some, and is therefore among the greatest of many challenges for emerging technology that has the capacity for significant improvement.

The issues surrounding neural networks requires total immersion for extended duration. Since many organizations lack the luxury of time, let’s get to it.

Beware the Foreshadow of the Black Swan

A recent article by Reuters confirms what is perhaps the worst kept secret in the post printing press era: Many Wall Street executives say wrongdoing is necessary: survey. A whopping 25% of those surveyed believe that in order to be personally successful, they must conduct themselves in an unethical manner to include breaking important laws, some of which are intended to defend against contagion; a powerful red flag warning even if only partially true.

This reminds me of a situation almost a decade ago when I had the unpleasant task of engaging the president of a leading university about one of their finance professors who may have been addressing a few respondents to this very survey when he lectured: “if you want to survive in finance, forget ethics”. Unfortunately for everyone else, even if that curriculum served the near-term interests of the students, which is doubtful given what has transpired since, it cannot end well for civilization. Fortunately, in this case the university president responded immediately, and well beyond expectation, after I sent an email stating that I would end my relationship with the university if that philosophy was shared by the institution.

For directors, CEOs, CFOs, and CROs in any sector, this latest story should only confirm that if an individual is willing to risk a felony for his/her success, then experience warns that corporate governance rates very low on their list of priorities. Black Swan events should therefore be expected in such an environment, and so everything possible should be planned and executed to prevent them, which requires mastering neural networks.

Functional Governance: As simple as possible; as complex as necessary

Functional governance and crisis prevention in the modern complex organization requires deep understanding of the organizational dynamics embedded within data architecture found throughout the far more complex environment of enterprise networks and all interconnected networks.

Kyield Enterprise Diagram 2.7 (protected by Copyright and U.S. Patent)

Are you thinking what I think you may be thinking about now? In fact adaptive neural networks in a large enterprise is quite comparable to the complexity found in brain surgery or rocket science, and in some environments even more so. The largest enterprise neural networks today far outnumber comparable nodes, information exchanges, and memory of even the most exceptional human neurological system. Of course biological systems are self-contained with far more embedded intelligence that adapt to an amazing variety of change, which usually enables sustainability throughout a complete lifecycle—our lives, with little or no external effort required. Unfortunately, even the most advanced enterprise neural networks today are still primitive by comparison to biological systems, are not adaptive by design, and are subject to a menagerie of internal and external influences that directly conflict with the future health of the patient, aka the mission of the organization.

So the next question might be, where do we start?

The simple answer is that most organizations started decades ago with the emergence of computer networking and currently manage a very primitive, fragmented neural network that wasn’t planned at all, but rather evolved in an incremental manner where proprietary standards became commoditized and lost the ability to provide competitive differentiation, yet are still very expensive to maintain. Those needing a more competitive architecture have come to the right place at the right time as we are deeply engaged in crafting tailored action plans for several organizations at various stages of our pilot program for Kyield enterprise, which is among the best examples of a state-of-the-art, adaptive enterprise neural network architecture I am aware of. We’ve recently engaged with large to very large organizations in banking, insurance, biotech, government, manufacturing, telecommunications, engineering and pharmaceuticals in the early stages of our pilot process.

Tailored Blueprint

Think of the plan as a combination of a technical paper, a deeply tailored use case for each organization, and a detailed time-line spanning several years. In some ways it serves as sort of a redevelopment blueprint for a neighborhood that has been locked-in to ancient infrastructure with outdated electrical, plumbing, and transportation systems that are no longer compatible or competitive. Most have either suffered a crisis, or wisely intent on prevention, while seeking a significant competitive advantage.

The step-by-step process we are tailoring for each customer serves to guide collaborative teams through the conversion process from the ‘current architecture’ to an ‘adaptable neural enterprise network’, starting with the appropriate business unit and extending throughout subsidiaries over weeks, months and years in careful orchestration according to the prioritized needs of each while preventing operational disruptions. Since we embrace independent standards with no lock-in or maintenance fees and offer attractive long-term incentives, the risk for not engaging in our pilot program appears much greater than for those who do. In some cases it looks like we may be able to decrease TCO substantially despite generational improvement in functionality.

Those who are interested and believe they may be a good candidate for our pilot program are welcome to contact me anytime.

New technology section @ kyield.com

New technology section @ kyield.com

We made extensive changes to kyield.com including a new section on our technology with new diagrams and descriptions that were not shared previously.