First, I am sure like many other authors in this space I need to define Big Data in the context of analytical processing to make it clear what we are talking about.  Big Data is a marketing term and not the best of terms at that.  A new reader in this market may well assume that this is purely about data volumes. Actually this is about being able to solve business problems that we could no solve before.  Big data can and more often than not include a variety of ‘weird’ data types. In that sense big data can be structured or poly-structured (where poly in this context means many).  The former would include high volume transaction data such as call data records in telcos, retail transaction data and pharmaceutical drug test data.  Poly-structured data is more difficult to process and includes semi-structured data like XML and HTML and unstructured data like text, image, rich media etc. Graph data is also a candidate.

From the experiences I have had in working in this area to date, I would say that web data, social network data and sensor data are emerging as very popular types of data in big data analytical projects.  Web data includes web logs and e-commerce logs such as those generated by on-line gaming and on-line advertising data.  Social network data would include twitter data, blogs etc. These are examples of interaction data which is something that has grown significantly over recent years. Sensor data is machine generated data from  ’An Internet of Things’. It is something we have only seen the beginning of in my opinion as much of it remains un-captured. RFIDs are probably the most written about of sensors. However these days we have sensors to measure temperature, light, movement, vibration, location, airflow, liquid flow, pressure and much more. There is no doubt that sensor data is on the increase and in my opinion it is something that will dwarf pretty well everything in terms of volume.  Telcos, utilities, manufacturing, insurance, airlines, oil and gas, pharmaceuticals, cities, logistics, facilities management and retail…..they are all jumping on the opportunity to use of sensor data to ‘switch on the lights’ in parts of the business where they have had no visibility before.  Sensor data is massive but we don’t want it all – it is the variance we are interested in.  Many Big Data analytical applications are/will emerge on the back of sensor data. These include analytical applications for use in:

• Supply chain optimisation
• Energy optimisation via sustainability analytics
• Asset management
• Location based advertising
• Grid health monitoring
• Fraud
• Smart metering
• Traffic optimisation
• Etc., etc.

Text as I already mentioned is also a prime candidate for big data analytical processing. Sentiment analysis, case management, competitor analysis are just a few examples of a popular types of analysis on textual data.  Data sources like Twitter are obvious candidates but tweet stream data suffers from data quality problems that still have to be handled even in a big data environment. How many times do you see spelling mistakes in tweets for example.  

There is a lot going on that is of interest to business in big data but while all of it offers potential return on investment, it is also increasing complexity. New types of data are being captured from internal and external data sources, there is an increasing requirement for faster data capture, more complex types of analysis are now in demand and new algorithms and tools are appearing to help us do this. 

So why is analytics on big data so important – or is it?

There are several reasons why big data is attractive to business. Perhaps for the first time, entire data sets can now be analysed and not just subsets. This is now a feasible option whereas it was not before.  So it is making enterprise think can we go down a level of detail? Is it worth it? Well to many it most certainly is. Even a 1% improvement brought about by analysing much more detailed data is significant for many large enterprises and well worth doing. Also schema variant data can now be analysed for the first time which could add a lot of valuable insight to that offered up by traditional BI systems.  Think of an insurance company for example. Any insurer whose business primarily comes from a broker network will receive much of its data in non-standard document format. Only a small percentage of that data finds its way into underwriting transaction processing systems while much of the valuable insight is left in the documents. Being able to analyse all of the data in these documents could offer up far more business value that could improve risk management and loss ratios.

At the same time there are inhibitors to big data analysis.  These include finding skilled people and a real lack of understanding around when to use Hadoop versus when to use Analytical RDBMS versus NoSQL DBMS.   On the skills front there is no question that the developers involved in Big Data projects are absolutely NOT your traditional DW/BI developers. Big Data developers are primarily programmers – not a skill often seen in a BI team.  Java programmers are aften seen at big data meet ups.  In addition, the analysis is primarily batch oriented with map / reduce programs being run and chained together using scripting languages like Pig Latin and JAQL (if you use the Hadoop stack that is).

Challenges with Big Data

There is no question that big data offers up challenges. These include challenges in the areas of

• Big data  capture
• Big data transformation and integration
• Big data storage – where do you put it and what are the options?
• Loading big data
• Analysing big data

In the context of BI, this turned the first generation collaborative BI tools on their head and said rather than take collaboration to BI it is far more effective to take BI to collaborative platform where the range of collaborative tools available offers a lot more power. Lyzasoft was a pioneer of this new generation of modern social and collaborative BI technologies.  Also new releases of more widely adopted BI platform products are now being integrated with mainstream collaborative platforms such as Microsoft SharePoint and IBM Lotus Connections.  Even cloud based collaboration technologies from vendors like Google are getting in on the act.  Mobile BI technology is taking this further by allowing people to collaborate on BI from mobile devices.

However, I (and others) would argue that we are still seeing only one side of the coin here with respect to BI and collaboration. That side is the classic approach of formal integration of data from multiple sources into a data warehouse, the producing of intelligence and the publishing of BI artefacts (dashboards, reports, etc.) into social and collaborative environments where it can be shared with others, rated and collaborated upon for joint decision making. But what about innovation, what about when innovative business users want to experiment, get some data and ‘play’ with it in a sandbox environment to figure out what business insight might be useful or to figure out what new metrics that would be useful to the business? Do we not need collaboration here also?  Another probing question is whether this innovation should be ‘upstream’ from a data warehouse? In other words let them play with the data until there is consensus as to what is useful and then feed this into a more classic approach of data integration, storage, analysis and sharing. I am comforted by the fact that it is not only me asking this question. Others like my good friend Barry Devlin are also talking about the use of collaboration and sharing of business insight produced in an innovative environment. I know Barry will be speaking about this here. The point is that in my opinion ( and it is only opinion admittedly) there is a place for collaborative and social BI in an innovative sandbox environment where BI is not yet ‘hardened’.  We need this capability in many industries. I have come across it in both retail banking and in manufacturing for example. However, what must be controlled is the release of newly formed innovation into production. This is where governance comes in. Data governance would allow newly created metrics to be published in a business glossary to be used by multiple BI tools in a hardened production environment for example. Also at this point, new data sources may be declared to a more formal production DW/BI environment for data acquisition.  Therefore we have two sides to collaborative BI, the innovation cycle which needs to share ‘experimental’ information and elicit feedback from other as well as the more formal production BI/DW environment where well polished business insight is shared across the enterprise for people to use and act on.  One feeds the other, typically because innovators also need to collaborate with IT to take the innovation and move it into the mainstream environment.

Let me know what you are doing with social and collaborative BI. I would be grateful for your comments.

Adding Aster Data to the mix means that Teradata now can potentially integrate with Hadoop deployments in both directions rather than one-way as with the Cloudera partnership.  For example, organisations could access Hadoop (Cloudera’s CDH and other other offerings) from from analytical queries running SQL M/R on Aster Data nCluster or indeed I would assume in the future on the Teradata DBMS itself.

There is no question this is a good move for Teradata. It gives them columnar capability and  also Aster Data has a rich library of pre-built map/reduce analytic functions to speed up M/R development and these functions can be invoked from SQL M/R on nCluster.  I would have to assume that Teradata would also want to open the Aster Data IDE and the M/R functions up to Teradata developers to deploy these M/R functions inside of Teradata. That is a no brainer in my opinion. You would also have to say that this takes Teradata in-database analytics to a new level of depth opening up the door for more sophisticated analytic applications. While the Teradata/SAS partnership is a successful one but adding Aster Data will potentially give Teradata much more power in the in-database analytics area. This is an area that really matters in big data environments. It will also give them more to compete with against IBM whose acquisition of Netezza (particularly with its TwinFin iClass appliance) and SPSS has given IBM much more competitive muscle recently especially against Teradata and Oracle (Exadata).   Besides competing with IBM, Aster Data will also give Teradata much more to compete with against Oracle’s Exadata.  We will have to wait to see what HP does with Vertica.

In addition, with the Tsumani of sensor data coming over the horizon this acquisition will help Teradata move into the world of Sensor Data Analytics which, by the way, is a battle still to be fought for(see my blog on this from last year).  Aster Data will help Teradata in accommodating the onslaught of data being generated by organisations increasing the instrumentation of their business operations with sensor networks.  However in this space, adding a CEP vendor technology to the Teradata portfolio would be a good move as sensor data event correlations need to be acted upon BEFORE that event data is stored in a data warehouse.  CEP, Active DW and SQL/MR. Hmm… now that is a combination  worth having. It will be interesting to see what is offered across the family of Teradata Appliances and if Teradata decide to rollout nCluster on any of them.  I would also think that Teradata will make sure they carefully protect the Aster Data customer base if they bring the DBMS technologies together gradually.

My only question is who will acquire Cloudera who have partnerships with BI platform vendors and other appliance vendors. That acquisition would pull the rug from under a lot of players.

My take on this is that configuration management and workload management are critical to large scale EDWs making it to the Cloud. I am thinking more about PRIVATE cloud than public cloud at this point as their are already an abundance of DW/BI PaaS and SaaS offerings on public clouds outside the firewall. However on-premise private cloud deployment is still very young. There is no doubt that small data marts are already moving but large scale EDW are not. Why not? I believe the reason is simple – no one has any experience how to configure virtual resources to make production EDW (Data integration, DBMS and BI platform) maximise it’s use of underlying hardware.

The point I would make here is more that I have not yet seen any detailed best practices from any source on porting bare metal EDWs to a virtualised private cloud in terms of the number of virtual servers needed, virtual memory needed, how to divide up virtual network channels or optimise virtual I/O.  In addition there is little experience of what software to install on what virtual servers e.g. whether data integration, DBMS and BI platform should all go on the same virtual servers or on different ones and in what combination. Also, what happens in ELT processing versus ETL processing? Instead what I am seeing is pre-configured private cloud DW/BI appliances emerging where the large vendors are coming to market with appliances with the complex configuration work already done i.e. the configuration is already optimised for an analytic workload to get maximum value out of the hardware.

If you look at guidance documentation for virtualising RDBMSs, it is clear that to date at least, intricate knowledge of the underlying hardware is needed in order to set this up correctly. However there is another point here which is that the advice is to create virtual storage (all physical disks available as a huge storage pool visible to potentially all VMs), switch on self-tuning memory management and let multiple VM instances (each with their own DBMS instance) access the virtual storage. That is a virtualised shared disk architecture. Yet for years we have seen scalable bare metal EDWs built on shared nothing MPP architectures. There is no doubt that you could set up virtual resources on a private cloud to function as a virtual MPP architecture but who are you going to get to do that? Data centre cloud developers who need to second guess MPP DBMS? Or should we leave it to MPP based DBMS vendors who know how to make that work? What happens if a cloud administrator changes the configuration underneath a EDW? Does the DBA know? Will it impact the service?…what experience has a cloud developer got here? I would argue that it is very very early days. Only Teradata has any long term expertise in virtual MPP architectures for EDW with virtual resources such as Virtual AMPs having been around for well over a decade on their platform. Even then they ship their own hardware with it balanced out of the box and provide tooling to monitor every aspect of virtual resource in detail. Also no one has issued any guidance on balancing the right amounts of  virtual resources or the best virtual configurations needed to scale data integration or BI platforms etc. So I agree that there is no way EDW on Private Cloud at least is a done deal. Certainly not this year. We still have a lot to learn. As one CIO put it to me recently, “why would I port a 100TB platform on dedicated hardware to a virtual server environment?”. Indeed….why would he? The bottom line is that for now at least he should not. There is no hard and fast rule that says everything HAS to be virtualised.  I am not saying don’t do it. What I am saying however is that people will want proof before they start moving these kinds of systems off scalable bare metal hardware to go into virtual scalability.

We have also learned over the years that data integration often scales better using ELT rather than ETL. Pushdown optimisation is the secret here such that the data integration server leverages the underlying massively parallel DBMS to do the heavy lifting work. So, what happens if ELT pushdown optimisation is on an MPP architecture mapped on to a shared disk virtualised environment? Would that not impact performance? I would think so. Again…very little experience here compounded by the fact that cloud developers are not in the same teams as DBAs, data integration developers, BI developers…. So we are entering an era where the ground is being changed under our feet by developers not necessarily skilled in DW/BI system deployments. Is it any wonder that vendors shipping private cloud analytical appliances are getting attention?

With a data centre team configuring VMs and virtual resources and BI, data integration and DBA developers now NOT responsible for installation, I think we have a way to go before large enterprises will just ‘up sticks’ and move every part of a BI/DW system to the cloud. ‘Steady as you go’ is my advice

I have had concerns for some time about how data integration tools are going to step up to the challenge of big data.  We are already in the era were hundreds of Terabytes and even Perabytes are a reality in data warehouses.  Also the volume of data needed for web analytics is massive let alone the tsunami of data being emitted by sensors that is coming over the horizon (if that data ever makes it into a data warehouse we really are going to re-define large). There is no doubt that the future is constantly on the up in terms of volumes of data and the number of data sources that the businesses need to integrate data from.  We all talk about how important data warehouse appliances, columnar compression  and scalable MPP databases are to handle large data volumes. But what about data integration? There is not much point having the ability to manage big data in databases if we can’t get the data in there in the first place.  So data integration vendors have to step up to this challenge. Of course several already have.  Many products on the market have offered pipeline parallelism during ETL processing for many years.  We have also seen many vendors switching to an ELT model for better performance so that they can exploit parallel SQL in the target DBMS engine to deal with the problem. That of course makes the data integration engine dependent on the parallel DBMS. This is fine as long as the data integration workload can be fenced off and managed separately from query processing workloads by a DBMS workload manager.  But is that it? What about the fact that these days modern hardware consists of multi-processor multi-core systems. Can a data integration engine itself not exploit this without relying on a DBMS?  Is there any way to get more bang for your buck on this kind of hardware?

Step in Pervasive DataRush.  This new engine from Pervasive is designed from the ground up as an MPP data integration engine that can exploit every core on a multi-core, multi-processor server.  This means that you might potentially avoid the need to go to clustered hardware because you can scale up before you need to scale out. The DataRush architecture is shown below

Source: Pervasive Software

What is interesting about this is not just that it exploits multiple cores in the DataRush engine but  that it also has an analytics library and other plug-in modules. The one that caught my eye was the DataRush Recommender module.  It strikes me that this engine (which can also be extended to support user defined libraries) not only has the capability to integrate data in parallel but it can also analyse that data using analytical models (data mining models) at the same time.  Couple that with the DataRush Recommender module and we are bordering on complex event processing (CEP). It seems we just need a rules engine in there and also of a sudden we are into massively parallel CEP.  Given that data integration is already rules driven it certainly looks to me that this product could potentially go well beyond just doing integration in parallel as important as that need is.  Pervasive has also made their products available in a PaaS offering on the Amazon EC2 Cloud as well as offering them on-premise which means that they can integrate and clean data from inside and outside the enterprise.  Given that you can also embed the technology it is certainly worth a look.  I plan to cover it in more detail in my up and coming Enterprise Data Governance and Master Data Management class running in London on September 22-24

Also it is possible to re-brand and re-skin MicroStrategy Mobile to fit your own corporate branding look and feel standards.  Overall, the user interface appears very interactive and very natural.  To support alerting, MicroStrategy Mobile is also integrated with the Apple push notification service so that personalised alerts are possible.  Also data can be cached so that when alerted users touch “view data”, the data is already there.

Architecturally speaking, MicroStrategy Mobile sits on top of the MicroStrategy Intelligence Server to allow BI to be made available to mobile employees, customers, partners and suppliers. Therefore all the capabilities of the MicroStrategy platform are still there.  I particularly liked the fact that the same design tool can be used for both web browser and mobile device access. Pre-built templates are also available to get you started quickly.  Getting started seems therefore fairly straightforward.  You can download the MicroStrategy Mobile App from Apple and point it at your own server.

One thing that was not covered enough in my opinion was the Portable Documents capability.  This allows you to build up a library of dashboards and re-use them across projects.  The reason this is important is consistency.  We all want trusted and consistent BI. In addition re-use is better than re-invention every time and so you can gain both from a productivity perspective as well as from a consistency perspective.

Implications of Mobile BI on Traditional BI Environments
Looking at this announcement there are several implications on any BI environment when implementing Mobile BI.  This includes the fact that the number of concurrent users is likely to increase significantly.  If you intend to make mobile BI available to external users like customers then the concurrent usage could skyrocket. So your BI platform and the underlying DBMS need to be able to scale to handle concurrent usage.  64-bit support in a BI platform will help a lot as more data can accessed in-memory. This together with caching will make a significant contribution to performance when handling more users. If your customers are worldwide and you make mobile BI available to them then it is very likely that you will also push your BI system into a 7×24 hour operating environment. High availability therefore starts to become critical.   I also believe that Mobile BI will become a ‘workhorse’ in business operations with many employees making use of it.  Therefore the number of on-demand BI requests from mobile devices is set to increase. The upshot is that workload management is going to be needed both in the underlying DBMS and in the BI platform to handle concurrent user requests from mobile devices as well as the traditional reporting and analysis taking place on a BI system.    Also if your mobile dashboards on an iPad are accessing data via data federation, there are implications here too.  There are likely to be many more federated queries and so caching matters here also.

Conclusion
Overall then, MicroStrategy has entered the mobile market with a robust implementation that exploits native interfaces to mobile devices.  With other BI vendors already in the mobile BI market the only question is how much of a differential will it have over its competition.  I like the offering. It also covers Blackberry which is more than Oracle BI 11g (announced yesterday) does.  MicroStrategy customers will no doubt exploit it.

If you combine On-demand BI running on top of on-demand information you can quickly see where we are going.  In my opinion it is only a matter of time before we see lots of intermediate companies (maybe even PaaS BI and SaaS BI vendors) making BI available on-demand in something similar to an iTunes  store.  The point here is that the level of abstraction rises again such that it is the BI that is of business value while the PaaS BI or SaaS BI solution is almost forgotten about.  Think of it like this. Imagine selling intelligence on-demand for the World Cup. At $5 a subscription the fact that this is built on a specific BI PaaS or SaaS BI solution is almost irrelevant. The consumer doesn’t care, they just want the intelligence.  It is the insight that is of value. The trick is to make it really easy to share intelligence of value in the form or reports and dashboards or dashboard components once they are built so that others can consume them quickly and easily on a subscription basis.

So bring on the “BITunes” store where insight is available on-demand on a subscription basis.  You can apply this to BI services built on external data available on the public cloud as well as to BI services available inside the enterprise on a private cloud. Users can simply subscribe to intelligence available on-demand.  This is an iTunes model and what a model it is. The size of this market could be very very significant. I don’t think we are far off.

• What is Cloud Computing and why use it as a deployment option?
• Why Cloud BI? – What are the requirements for a public cloud or externally hosted BI system?
• Understanding what is on offer – The Cloud BI Marketplace
• Getting data into a cloud based BI system
• Managing access to cloud based BI systems and analytic applications
• Integrating cloud based BI systems with on-premise systems
• Pros and cons of deploying on the cloud?
• Getting started with Cloud based BI

Bear in mind that both public cloud and private cloud based BI were under discussion even though the hype seems all around public cloud or externally hosted BI systems.   Looking at these points it is the third bullet down that for me is the clear inhibitor to cloud based BI adoption.In other words the lack of understanding as to what exactly is on offer.  And there is a lot on offer. On the public cloud we have everything from plain Infrastructure as a Service (IaaS)  all the way through to Software as a Service (SaaS) based packaged analytic applications. On the private cloud several BI platforms are already running on virtualisation software such as VMware and/or Microsoft Hyper-V.  However there seems very little in the way of best practice advice on do’s and don’ts when it comes to deploying BI systems on a private cloud based virtualised environment.

In total I came up with 6 options, the last of which is simply where many of us are today i.e. BI systems not deployed on a cloud (whether it be public or private).  The options are as follows:

1.     Public cloud based IaaS for a BI system

2.     Public cloud or externally hosted BI/DW PaaS for building your own cloud-based BI system

• Multi-vendor or single-vendor BI PaaS offerings

3.     Public cloud or externally hosted SaaS BI packaged analytical applications

4.     Public cloud or externally hosted SaaS BI for operational reporting on cloud based operational data

5.     Private cloud based BI system running internally

6.     Dedicated hardware based BI system (this is what most companies have today)

Option 1 is simply subscribing to an IaaS vendor like Savvis, Amazon, Rackspace or GoGrid  where you pay as you use on hardware and systems software and then buying and deploying your own ETL, DBMS and BI software (assuming they have no restrictions on what they will support).  I am not sure that this is attractive enough on its own without a BI/DW Platform as a Service (PaaS) as well.

Option 2 is the BI/DW Platform as a Service (PaaS) option on public cloud or even externally hosted.  Here you find another choice however. Should you choose a multi-vendor DW/BI PaaS or a single-vendor offering.  An example of a multi-vendor option is the RightScale/Talend/Vertica/Jaspersoft PaaS offering on Amazon EC2.  A single vendor PaaS offering (of which there are several on offer) would be GoodData, or SAP BusinessObjects On-Demand. Others include Birst, Indicee and PivotLink.  A key question here is going to be “Is Data Integration included?”  Clearly in the multi-vendor offering mentioned there is an ETL solution such as Talend in the above example.   Data integration is very much file based with BI/DW PaaS vendors i.e. you upload files of data and then there is some processing of that data to load it into the PaaS DW/BI database.  Several single-vendor PaaS offerings give you only fairly lightweight data integration once data is uploaded.  Certainly not full blown ETL with built-in data quality that you might be used to in a data centre. In fact if you are looking for full blown DQ you are going to be disappointed in most cases.  The ‘get out’ clause is you can add your own script but what about metadata lineage and auditability once the script writer has left for a better job?  A vendor like SAP (mentioned earlier) does have ETL (SAP BuisnessObjects Data Integrator) available but only if you subscrible to their Advanced Edition of SAP BusinessObjects On-Demand (there are 3 editions on offer).  I was even more surprised to see that SAPs BI/DW PaaS offering uses Microsoft SQL Server as the database and not BW.  I would expect that to change to SybaseIQ fairly soon. GoodData on the other hand have refreshingly recognised that you may want to go beyond the data integration you get out-of-the-box on subscription and have gone the extra mile to provide pre-built integration with cloud based data integration tools such as Informatica Cloud, SnapLogic and Boomi. Therefore you can use these tools to integrate your data before passing the data sets to them. The alternative to all of this is to do the lions share of the data integration in-house before uploading data files.

Option 3 is a fast growing market with many relatively new vendors (e.g. Cloud9 Analytics, Rosslyn Analytics, Lixto) as well as traditional mainstream vendors e.g. SAS, IBM Cognos.  The attraction here is a pre-built solution ready to go. These will clearly appeal to small and medium size businesses (SMBs) and even lines of business in some large organisations.  While we see horizontal applications looking at Salesforce.com data, spend analysis and pricing (to name a few), I am predicting that vertical analytic apps on the cloud will appear.

Option 4 is simply using a cloud based reporting system on operational data typically from a cloud based transaction processing system such as Salesforce.com.  In fact it would seem that Salesforce.com is dominating this space. An example here is SAP BusinessObjects CrystalReports.com for Salesforce.

Option 5 is private cloud based BI systems. The largest private cloud based BI system I know of is IBM’s internal Blue Insight which is based on IBM System Z and IBM Cognos 8 BI.  An estimated 200000 IBMers are using this.  IBM have since launched the Smart Analytics Cloud, a private cloud offering for large enterprises based on the same technologies.  However it is still early days for BI deployments on internal private clouds. There appears to be more support coming from developer forums than vendors at present.  From what I can see, companies are taking a ‘toe in the water’ approach to deploying on virtualized environments. No doubt, confidence will grow over time.  However does everything need to move to private cloud? Many companies with very large EDW initiatives may be reluctant to move to private clouds until they prove their scalability and lower TCO.   This issue here is should ETL, DBMS and BI platform all be on the same virtual servers? Should each have their own virtual server configuration? What is that configuration? Can I adjust it? etc. etc. I don’t think there will be a mad rush to put a 100TB DW on virtual servers.  I do like the fact that vendors like Microstrategy have given this some serious consideration and have released a private cloud enterprise edition of Microstrategy 9.  MicroStrategy components are packaged as Virtual Appliances and tuned for expected load. These Virtual Appliances contain fully configured software components and the number of running virtual appliances can be adjusted to accommodate specific performance goals. This is a damn sight better than just saying to a customer “it’s up to you, just deploy it and you figure out the virtual server configuration”  What Microstrategy have done is to allow you to adjust the underlying assigned physical resources to satisfy performance demands and have made available administrative facilities to control virtualized MicroStrategy environment.

It is early days in Cloud based BI. I recommend looking at your requirements and then match the options available to your needs

I would be interested if any of you have experiences in this area. Do’s and Don’ts. What works, what doesn’t.  Please share them by placing your comments.

Follow me on twitter

Attending some of the case studies I found some fantastic applications of the use of sensor networks and RFIDs.  Healthcare with sensors all over hospitals with equipment and patients all tagged with RFIDs.  The return on investment in this case was fraud prevention on equipment (theft mainly) and process improvement for patients.  Another session I attended was one on monitoring stress in all the bridges in the US – over 700000 of them.  Some of the stats being quoted by the speakers were staggering.  “Well we are emitting, 3 events per minute from every sensor on a 7×24 hour basis. After 6 months operating like this we have over 20 PETABYTES of data”.  You read it right 20 PETABYTES.   A lot of the technical focus at the conference was on energy harvesting to prolong sensor battery life,  but the business message was clear as a bell.  Process optimisation, preventative maintenance and cost reduction comes from instrumenting business operations.  Manufacturing production lines, supply chains, product distribution, asset management.  You name it, they’re measuring it.

So I have to ask, where are all the BI vendors? Where are all the analytical DBMS vendors? Where are the CEP products, the real-time dashboards and predictive analytical models for automated analysis?  This is an operational BI gold mine.  Yet there are no mainstream vendors in sight bar IBM (at least someone there is switched on to what is happening).  The volume of data coming over the horizon from the adoption of sensor networks and RFIDs is nothing short of massive.  What is also clear is that this is already going on in enterprises and IT are blissfully unaware of it in the main.  Clearly IT BI professionals have got to get in touch with their Engineering colleagues and engineers have got to be made aware of mainstream data integration, analytical database and BI platform technologies as well as CEP software of course.  In my 29 years in the industry, I don’t think I have ever seen a chasm between IT and business not even explored never mind crossed.  Yet the value of CEP and mainstream DW/BI to this market is nothing short of enormous.   It is symptomatic of a young market heaving with engineers that has yet to be tied into mainstream IT to exploit far more robust software than is being used on this data at present.  What an opportunity. What a huge opportunity.  It most certainly is going to re-define large databases when we have to set them up for analysis of historical event data emitted by these devices.  CEP has to go there. CEP vendors have to get out of just being in the financial markets and wake up to a ton of data in motion being emitted by the growing number of devices.  An article I read recently said that Sensors empower an Internet of Things.  Well, those things are coming over the horizon emitting a Tsunami of data. It is time CEP and DW/BI vendors woke up an smelt the coffee and became aware of this rapidly growing market.  CIOs had better take heed too because they are going to have to integrate it into mainstream IT.

The one thing that stood out for me was the seismic shift towards collaborative BI.   As my friend Colin White so aptly put it in the analyst briefing, “Microsoft have brought BI to collaboration rather than collaboration to BI”.  This is an important point because what it is says is that there is little point adding collaborative features to a BI platform if these are not the services associated with a mainstream collaborative platform.  There is far more value in integrating a BI platform with the company collaboration software to tap into things like collaborative workspaces, presence awareness, unified communication, shared calendar etc. etc.  In Microsoft”s case this is of course the SharePoint product which has become viral in most organisations.

It is no surprise therefore that Microsoft’s BI initiative is built around 3 main components and not just SQL Server.  These are:

• Office
• SharePoint
• Microsoft SQL Server 2008 R2

Note that SQL Server 2008 R2 includes StreamInsight, Microsoft’s complex event processing (CEP) engine and Microsoft Master Data Services

While there we were take through an excellent demo to show the power of collaboration and what it can do when integrated with BI.  It even included the Microsoft Round Table device which although it has been available for some four years, was the first time I have actually encountered one.

What the demo showed me was the speed with which BI and BI ‘components’ can be spread among a community of users. My conclusion is that integration of SQL Server 2008 R2 with Sharepoint 2010 takes this to another level in that the rate that business intelligence can be shared it is almost ‘twitter speed’.  For those of you using twitter, you will know that as soon as something of interest breaks, re-tweets can spread it across masses of people in a matter of minutes.  This is the feeling I got during the demo.  It fuels mass sharing, mass reuse and mass development of BI applications and artifacts.  In particular reports and dashboards. It certainly fits with Microsoft”s vision of BI for everyone.

Several new features open up the flood gates for collaborative BI to share intelligence with other without the need for IT. For example,

BI reports can be managed by Sharepoint in document libraries. You can also preview reports before opening them up.

Also Microsoft is fueling development by business users on the back of what power users have done, thereby bypassing IT.  This is because there is now a capability whereby Microsoft ReportBuilder 3.0 can access PowerPivot workflows uploaded to SharePoint sites.  You can also export to Excel from PowerPivot.  Power users using PowerPivot (originally referred to as Gemini), can take data from different data sources (including newly supported Atom feeds), merge and join that data. Relationships between tables can be managed inside of PowerPivot.  PowerPivot power users can then create workflows that process this data and can upload these to Sharepoint sites.  ReportBuilder 3.0 (or any BI client) can then treat the PowerPivot workflow as a data source.  Not only that but ReportBuilder can create report parts which are sharable in a report part gallery do that other users can reuse them by simply dragging an dropping the report parts onto a new report for rapid development without having to know the detail underneath.

Hopefully by now you have got the picture – power users building their own workflows in PowerPivot, publishing them to SharePoint, other users using them as data sources in reports, report parts created, and a gallery of parts to be shared across a community of users.  Powerful stuff, and we are not done yet.

In Sharepoint 2010 there is a new site template called Business Intelligence Center.  What you can now do is create a new site in SharePoint using the Business Intelligence Center template. This template includes chart web parts and Excel services workbook access. It also includes a PerformancePoint library so that you can start building your dashboard very rapidly including access to reports and report parts. With is mechanism, Microsoft is opening up dashboard development to the masses and also allowing ‘social’ performance management whereby dashboards and/or dashboard components can be rated.  All this integrated with SharePoint and Office is in my opinion going to take self-service BI development to another level that it could easily have a ‘popcorn effect’ with masses of BI being produced rapidly and IT nowhere in sight.  There is no doubt that it opens up the flood gates for business innovation and sharing.  Personalised dashboard development using PerformancePoint Services 2010 integrated with SharePoint 2010.

A Question of Governance?

My only concern with this is the issue of governance.  What Microsoft have done is to put mass development in the hands of the business.  If you think upi have seen anything on self-service BI, just wait until SharePoint 2010, Office 2010 and SQL Server 2008 R2 move into production in your shop. You ain’t seen nothing yet.

However, I see very little with respect to data governance. What about business glossaries? What about metadata lineage?  In a world of increasing regulation and legislation to prevent corporate catastrophes, can anything be audited? Can it be tracked back to where the data come from? How has the data been transformed by the power users? iWhat does the data mean?  I have as yet seen little from Microsoft in the form of metadata management and data governance despite the fact that Master Data Services is also delivered as part of this SQL Server release.  While there is no doubt that this is coming (confirmed by the Microsoft guys I spoke with on the exhibition floor booth) my only fear is will be too late.  Will the horses have already bolted with self-service BI unstoppable and off down a track without lineage to help users know that the data is trusted.

Equally, scorecard and dashboard development is bottom up. Everyone (with authority) can create their own scorecards and dashboards rapidly but there appears to be no framework whereby these can be slotted into a multi-level  strategy management unlike say SAP with SAP Strategy Management.  So what is the answer? Is it all bets are off and just let the business figure out the best way to manage on the back of socially rated scorecards and dashboards?  What happened to business strategy?  Many companies set a strategy at executive level and want enterprise wide business strategy execution.   This latter approach is top-down.  What Microsoft is fueling is bottom up.  My opinion is we need both and not one or the other.

Freedom Versus Governance – A Delicate Balancing Act

It is pretty clear then that, setting aside the new SQL Server Data Warehousing Edition, this is very much a Collaborative BI release by Microsoft.  It is a major leap forward in what the business users can do for themselves.  We have two forces at work here.  Freedom versus governance.  We have to get the balance right.  Too much freedom and we could have chaos with no ability to audit what has been done or whether the BI is trusted. Too much governance and we put innovation in a straight jacket or kill it altogether.   All I would say is that IT had better get a data governance program underway soon to control data all the way out to data marts and cubes. If that is done then there is no doubt that the business can be empowered to innovate which is what should happen. Without a data governance program however, I think it is really going to be hard to get alignment with what the business is doing given the sheer speed of development that is now possible with this release.  Let’s hope governance, innovation and collaboration are a winning combination.