Posts Tagged ‘Business Intelligence’

« Older Entries

BITunes on the Cloud? – The Emergence Of Subscription Based On-Demand BI

Tuesday, June 29th, 2010

As I research more and more into the world of Cloud-based BI, it is becoming pretty evident where we are headed. In my opinion we are moving down the road to an iTunes model for BI.   Yesterday I spent some time with Actuate in London looking at their BIRT On-Demand platform as a service (PaaS) solution (which is very easy to use). It was only a matter of minutes before I was up and running with a Mashboard.  A few weeks back in New Orleans I used Dundas Dashboard to quickly build a dashboard from pre-built components. Similarly Microsoft SQL Server 2010 has the ability in ReportBuilder 3.0  to quickly build up a library of components that can be dragged and dropped into a report. The  more I use these products to understand their capabilities the more I see a similarity to what is happening in the information management world.  Looking at cloud-based data integration solutions like Boomi, Informatica and SnapLogic for example, you can see that what these vendors are trying to do is to create a development platform for Information as a Service. In other words you build data integration jobs and then make the results available on subscription such that companies can subscribe to information which is supplied to them by cloud based data integration workflows running on the net.   So now apply this idea to the BI produced on cloud-based PaaS solutions.  Once your reports and dashboards are built then the next thing people are going to want to do is to publish these artifacts as on-demand BI services assuming the intelligence is of business value to others.

If you combine On-demand BI running on top of on-demand information you can quickly see where we are going.  In my opinion it is only a matter of time before we see lots of intermediate companies (maybe even PaaS BI and SaaS BI vendors) making BI available on-demand in something similar to an iTunes  store.  The point here is that the level of abstraction rises again such that it is the BI that is of business value while the PaaS BI or SaaS BI solution is almost forgotten about.  Think of it like this. Imagine selling intelligence on-demand for the World Cup. At $5 a subscription the fact that this is built on a specific BI PaaS or SaaS BI solution is almost irrelevant. The consumer doesn’t care, they just want the intelligence.  It is the insight that is of value. The trick is to make it really easy to share intelligence of value in the form or reports and dashboards or dashboard components once they are built so that others can consume them quickly and easily on a subscription basis.

So bring on the “BITunes” store where insight is available on-demand on a subscription basis.  You can apply this to BI services built on external data available on the public cloud as well as to BI services available inside the enterprise on a private cloud. Users can simply subscribe to intelligence available on-demand.  This is an iTunes model and what a model it is. The size of this market could be very very significant. I don’t think we are far off.

Tags: , ,
Posted in Uncategorized | 1 Comment »

Cloud Based BI – Understanding The Options Is the Biggest Barrier

Tuesday, June 22nd, 2010

Last week I was in Munich to present at the annual TDWI (The Data Warehouse Institute) conference on “Business Intelligence and Data Management in a Cloud Computing Environment”.  It was a very well attended conference with some great speakers and sessions.  My session focused on the following:

  • What is Cloud Computing and why use it as a deployment option?
  • Why Cloud BI? – What are the requirements for a public cloud or externally hosted BI system?
  • Understanding what is on offer – The Cloud BI Marketplace
  • Getting data into a cloud based BI system
  • Managing access to cloud based BI systems and analytic applications
  • Integrating cloud based BI systems with on-premise systems
  • Pros and cons of deploying on the cloud?
  • Getting started with Cloud based BI

Bear in mind that both public cloud and private cloud based BI were under discussion even though the hype seems all around public cloud or externally hosted BI systems.   Looking at these points it is the third bullet down that for me is the clear inhibitor to cloud based BI adoption.In other words the lack of understanding as to what exactly is on offer.  And there is a lot on offer. On the public cloud we have everything from plain Infrastructure as a Service (IaaS)  all the way through to Software as a Service (SaaS) based packaged analytic applications. On the private cloud several BI platforms are already running on virtualisation software such as VMware and/or Microsoft Hyper-V.  However there seems very little in the way of best practice advice on do’s and don’ts when it comes to deploying BI systems on a private cloud based virtualised environment.

In total I came up with 6 options, the last of which is simply where many of us are today i.e. BI systems not deployed on a cloud (whether it be public or private).  The options are as follows:

1.     Public cloud based IaaS for a BI system

2.     Public cloud or externally hosted BI/DW PaaS for building your own cloud-based BI system

  • Multi-vendor or single-vendor BI PaaS offerings

3.     Public cloud or externally hosted SaaS BI packaged analytical applications

4.     Public cloud or externally hosted SaaS BI for operational reporting on cloud based operational data

5.     Private cloud based BI system running internally

6.     Dedicated hardware based BI system (this is what most companies have today)

Option 1 is simply subscribing to an IaaS vendor like Savvis, Amazon, Rackspace or GoGrid  where you pay as you use on hardware and systems software and then buying and deploying your own ETL, DBMS and BI software (assuming they have no restrictions on what they will support).  I am not sure that this is attractive enough on its own without a BI/DW Platform as a Service (PaaS) as well.

Option 2 is the BI/DW Platform as a Service (PaaS) option on public cloud or even externally hosted.  Here you find another choice however. Should you choose a multi-vendor DW/BI PaaS or a single-vendor offering.  An example of a multi-vendor option is the RightScale/Talend/Vertica/Jaspersoft PaaS offering on Amazon EC2.  A single vendor PaaS offering (of which there are several on offer) would be GoodData, or SAP BusinessObjects On-Demand. Others include Birst, Indicee and PivotLink.  A key question here is going to be “Is Data Integration included?”  Clearly in the multi-vendor offering mentioned there is an ETL solution such as Talend in the above example.   Data integration is very much file based with BI/DW PaaS vendors i.e. you upload files of data and then there is some processing of that data to load it into the PaaS DW/BI database.  Several single-vendor PaaS offerings give you only fairly lightweight data integration once data is uploaded.  Certainly not full blown ETL with built-in data quality that you might be used to in a data centre. In fact if you are looking for full blown DQ you are going to be disappointed in most cases.  The ‘get out’ clause is you can add your own script but what about metadata lineage and auditability once the script writer has left for a better job?  A vendor like SAP (mentioned earlier) does have ETL (SAP BuisnessObjects Data Integrator) available but only if you subscrible to their Advanced Edition of SAP BusinessObjects On-Demand (there are 3 editions on offer).  I was even more surprised to see that SAPs BI/DW PaaS offering uses Microsoft SQL Server as the database and not BW.  I would expect that to change to SybaseIQ fairly soon. GoodData on the other hand have refreshingly recognised that you may want to go beyond the data integration you get out-of-the-box on subscription and have gone the extra mile to provide pre-built integration with cloud based data integration tools such as Informatica Cloud, SnapLogic and Boomi. Therefore you can use these tools to integrate your data before passing the data sets to them. The alternative to all of this is to do the lions share of the data integration in-house before uploading data files.

Option 3 is a fast growing market with many relatively new vendors (e.g. Cloud9 Analytics, Rosslyn Analytics, Lixto) as well as traditional mainstream vendors e.g. SAS, IBM Cognos.  The attraction here is a pre-built solution ready to go. These will clearly appeal to small and medium size businesses (SMBs) and even lines of business in some large organisations.  While we see horizontal applications looking at Salesforce.com data, spend analysis and pricing (to name a few), I am predicting that vertical analytic apps on the cloud will appear.

Option 4 is simply using a cloud based reporting system on operational data typically from a cloud based transaction processing system such as Salesforce.com.  In fact it would seem that Salesforce.com is dominating this space. An example here is SAP BusinessObjects CrystalReports.com for Salesforce.

Option 5 is private cloud based BI systems. The largest private cloud based BI system I know of is IBM’s internal Blue Insight which is based on IBM System Z and IBM Cognos 8 BI.  An estimated 200000 IBMers are using this.  IBM have since launched the Smart Analytics Cloud, a private cloud offering for large enterprises based on the same technologies.  However it is still early days for BI deployments on internal private clouds. There appears to be more support coming from developer forums than vendors at present.  From what I can see, companies are taking a ‘toe in the water’ approach to deploying on virtualized environments. No doubt, confidence will grow over time.  However does everything need to move to private cloud? Many companies with very large EDW initiatives may be reluctant to move to private clouds until they prove their scalability and lower TCO.   This issue here is should ETL, DBMS and BI platform all be on the same virtual servers? Should each have their own virtual server configuration? What is that configuration? Can I adjust it? etc. etc. I don’t think there will be a mad rush to put a 100TB DW on virtual servers.  I do like the fact that vendors like Microstrategy have given this some serious consideration and have released a private cloud enterprise edition of Microstrategy 9.  MicroStrategy components are packaged as Virtual Appliances and tuned for expected load. These Virtual Appliances contain fully configured software components and the number of running virtual appliances can be adjusted to accommodate specific performance goals. This is a damn sight better than just saying to a customer “it’s up to you, just deploy it and you figure out the virtual server configuration”  What Microstrategy have done is to allow you to adjust the underlying assigned physical resources to satisfy performance demands and have made available administrative facilities to control virtualized MicroStrategy environment.

It is early days in Cloud based BI. I recommend looking at your requirements and then match the options available to your needs

I would be interested if any of you have experiences in this area. Do’s and Don’ts. What works, what doesn’t.  Please share them by placing your comments.

Follow me on twitter

Tags: , , , , , ,
Posted in Uncategorized | 2 Comments »

Oracle Enters The DW Appliance Market

Thursday, September 25th, 2008

Well now – has the inevitable happened? Larry Ellison, Oracle’s CEO finally recognised the value of the hardware/software combination in the business analytics marketplace. This market, pioneered over 20 years ago by Teradata and now thriving with many other players including Netezza, ParAccel, Datupia, ExaSol, Vertica and others, has now become a target for Oracle who have clearly had enough of competitors eating away at the Oracle database with lower TCO DBMS offerings optimised for analysis and reporting. With the recent acquisition of DatAllegro by Microsoft, IBM with its Balanced Warehouse and now Oracle entering the database machine market it certainly seems that the DW Appliance market is now becoming a hot competitive battleground.

The newly announced Oracle Exadata DW Appliance is jointly developed by Oracle and HP and will be sold directly by Oracle. The Exadata server runs the Oracle parallel server on Oracle Enterprise Linux. It has 8-HP Proliant DL360 G5 database servers, with

  • 2 quad-core Intel Xeon Processor E5430 (2.66GHz)
  • 32GB memory
  • 1-HP InfiniBand Dual Port HCA
  • 4-146GB SAS 10K hard disk drives
  • 4-24-port InfiniBand switches
  • 14-HP Exadata Storage Server Hardware–each is an HP ProLiant DL180 G5, with 2 quad-core Intel Xeon Processor E5430 (2.66GHz)
  • 8GB memory
  • 1-HP InfiniBand Dual Port HCA
  • 12-300GB SAS or 12-1TB SATA disk drives

My question on this announcement is given that HP are jointly in on the Exadata product offering with Oracle, what does this mean for HP’s own DW Appliance offering – the HP NeoView Appliance? This is also a parallel DBMS product that competes with Oracle. I assume that with HP playing in both markets (its own DBMS product on its own hardware plus the hardware behind the Oracle Exadata offering) that it is seeking to maximising the revenue it can take by covering all bases. Time will tell. In my opinion it is clear that with so many vendors now in the DW Appliance market it is going to take a lot more than just TPC-H benchmarks to get a differential. It certainly means customers will have to look closely at performance claims. Everyone will claim they are the fastest which could easily result in prospective customers demanding more to distinguish one vendor from another. For this reason I believe that analytic application appliances have to happen (analytic application pre-installed on a DW Appliance). Vendors who go deep on vertical analytic application appliances could carve out a very lucrative business when you combine this with the attraction of low TCO DW Appliance offerings.

Tags: , , , ,
Posted in Uncategorized | No Comments »

DatAllegro – The New Microsoft DW Appliance?

Monday, July 28th, 2008

Why is it every time I take a week off on holiday something major happens in the BI market? I am of course talking about Microsoft’s Announcement to acquire the data warehouse appliance vendor DatAllegro. The message on this is certainly obvious, the scaling of SQL Server. It’s a very interesting announcement. With the exception of Netezza who have done well here in the UK and in Europe, many of the DW appliance vendors have been struggling. Almost all of them have been chasing business in the same vertical industries that have high volumes of data (e.g. Telco, Retail, Financial Services). Now the one giant that people were wondering about in terms of parallel DBMS scalability, has moved. However we are clearly going to have to wait to see how well SQL Server scales in this kind of setup. Even prior to this announcement the myth that SQL Server would not scale beyond 1 Terabyte has long been proved incorrect however. I have certainly had clients running single instance SQL Server BI system databases at around 13-15 terabytes for a number of years now. No doubt there are larger configurations than that out there. However this announcement will certainly lift Microsoft customer confidence that Redmond are serious about offering a scalable SQL Server option on non-proprietary hardware that starts to compete with the parallel DBMS offerings of IBM, Oracle and Teradata as well as other DW Appliance offerings. Time will tell what will happen and how competitive this will be. SQL Server Integration Services (the Microsoft ETL tool that ships with SQL Server) will also have to scale however to get larger data volumes on to a parallel SQL Server. So we have to see what Microsoft will do here. This announcement also offers up an interesting option for Kalido who recently announced support to generate for Microsoft SQL Server.

Tags: , ,
Posted in Uncategorized | No Comments »

BI Vendor adoption of Web 2.0 Emerging

Wednesday, April 9th, 2008

Following my recent series of articles on Web 2.0 and BI on the B-Eye-Network, it is exciting to see BI products pushing their way into the BI market embracing web 2.0. Several vendors such as Information Builders and SAP (Business Objects)< have released BI Mashup tools recently but the one vendor that caught my eye is Antivia, a small Australian company based in Sydney that is really embracing communities and social networking with their product.

As web 2.0 edges towards becoming mainstream in the BI market (probably 2009 timeframe) expect to see more adoption of richer interaction in user interfaces and more collaboration capability to share BI for joint decision making. If you are already using Web 2.0 in BI applications and tools in your organisation please share your experiences on what works and what doesn’t.

Tags: , , ,
Posted in Uncategorized | No Comments »

Younger Generation Vendors Rock The BI Boat

Tuesday, March 25th, 2008

In 2008, some 18 years since I moved into the BI sector of the industry, you would think that this space would be exhausted. Yet here we are seeing more announcements from relatively ‘new generation’ vendors aggressively going after this market. I refer of course to the announcement yesterday between LogiXML and Vertica on integration between LogiXML’s Web-based BI platform with the Vertica Analytic Database. It seems clear that columnar data warehouse appliance vendors are climbing the popularity charts with Vertica, ParAccel and SybaseIQ all gaining ground. LogiXML is clearly also on a growth path with its web based BI platform. With consolidation still happening in the BI market, it would be no surprise to me if these kinds of partnerships ultimately go further in future but for now, companies should not assume that just because the software giants (IBM, Microsoft, Oracle and SAP) have made their moves into the BI market that the game is over. This kind of announcement really offers an attractive and competitive alternative.

Tags: , ,
Posted in Uncategorized | No Comments »

Dashboards and Excel Data – Business Value or Lipstick on Untrusted Data?

Saturday, March 22nd, 2008

Over the last year or so I have noticed a real surge in companies using or evaluating products to rapidly develop dashboards. In my consulting activities in this area, I have been amazed at the reliance on one particular primary source of data that users have latched onto in dashboard development. This is of course Excel data. While there is nothing unusual about Excel data, it is the trait that users almost ‘prefer’ to access Excel data (because they are familiar with Excel) that I find concerning. Many users seem to either just have these spreadsheets or are downloading data into Excel from a range of data sources including operational systems, flat files (perhaps supplied from some other department or system), data marts and data warehouses. Once data is ‘in the wild’ like this, it takes on a life of its own with people manipulating it and sending it to others via email attachment. It’s like data management just got left behind.

While Excel can never be ignored in any organisation, the increasing demand to analyse Excel data raises questions as to whether or not that data can be trusted especially if you have been sent this data in your email. It brings back the issue that has plagued many companies for years when it comes to Excel. Do you know where the data in the spreadsheet came from? How do you know you have the right version of the spreadsheet? Are spreadsheets managed? Is there other server side data sources that can be accessed from the dashboard tool that would give you more confidence in trusting the data?

With Office Excel 2007 increasing the maximum limit on the number of rows in a spreadsheet from 64000 to 1 million, my concern is that the increasing demand for dashboards will raise the likelihood of million row “spreadmarts” being created all over the organisation by business users rather than pointing dashboards at server side data in a BI system. Only time will tell however policy is clearly needed around spreadsheet management and dashboard development if we are to remain in control of the data and have confidence in it. I would be interested in hearing from many of you out there who are encountering this problem and what you are doing to manage it.

Tags: , , ,
Posted in Uncategorized | No Comments »

Giants Start To Turn Up The Heat on BI & Search

Monday, January 14th, 2008

Microsoft’s announcement that it is intending on acquiring FAST last week is certainly the first major shot fired in the enterprise search battle that could ensue among the software giants. However it is not so much enterprise search that interests me in this context but the investment that a vendor like FAST had made in pursuing the BI market. There is no doubt that Search and BI are going to be hot in 2008. Expect much more activity in this area in 2008 as these two exciting areas of technology increasingly collide. I will be researching the state of this emerging Search and BI market and will feed back to the B-Eye-Network later in 2H08 on it. Meanwhile it will be interesting to watch vendors like Autonomy , Endeca and even Google and their relationships with the software giants to see where this leads.

Tags: , , ,
Posted in Uncategorized | No Comments »

Predictions for 2008

Wednesday, January 2nd, 2008

Happy New Year to all of you! It’s that time of year again when predictions are made so I thought I would throw my thoughts into the ring for debate. So here goes:

  1. Consolidation in the BI market will continue now that we have seen the software giants make their moves in 2007. Oracle bought Hyperion, SAP bought Business Objects and IBM announced acquisition of Cognos. In order to compete with this, other vendors will consolidate to try to offer and alternative. So expect to see more mergers in 2008
  2. The cost of the BI platform will continue to drop amid pressure from the software giants (Microsoft in particular), and open source alternatives (e.g. Pentaho, Jaspersoft, Talend et al). The money to be made is in Performance Management and Data Management.
  3. Both Performance Management and Data Management technologies will separate from BI platforms (if they haven’t done so already) and become suites of tools in their own right
  4. The growth in the size of the data management market is set to continue as companies try to standardise of a suite of tools for enterprise data management (an enterprise data management platform) which includes an end-user business vocabulary tool, data modelling tool, data discovery and mapping tool, data quality profiling, data cleansing, data integration (consolidation, federation and synchronisation). This data management platform will be used for data replication, data warehousing, data migration, master data management, data synchronisation and on-demand data management services published in a service registry and available on an enterprise service bus (ESB) in a service oriented architecture (SOA)
  5. Complex Event Processing (CEP) will become mainstream in 2008 as companies try to analyse data and the business impact of events well before that data arrives in any kind of data warehouse or data mart. This is also known as business activity monitoring (BAM) except we are going to see monitoring of complex events (on the lookout for several events happening before triggering action) and not just single ones
  6. 2008 will be the year of massive growth in memory exploitation. We will see parallel query execution continue to run across multiple shared nothing nodes in MPP systems with multiple processors, and multiple disks (as is the case today in many parallel relational DBMSs). However the difference here is that we will see this happening against in-memory data on a massively parallel scale in 2008 and beyond. With the volumes of data about to climb higher, and demand for CEP on the increase, we need to access data in memory to respond more rapidly and keep performance optimal. Massively parallel memory is therefore inevitable and will arrive on the scene this year whether that memory be in a single cluster server or deployed over a grid in a virtual memory configuration
  7. Performance Management is set to grow with BAM, process management, scorecards, dashboards, budgeting and planning and Business Intelligence all being integrated into a component performance management suite (enterprise performance management platform). Performance Management platforms will sit on top of BI platforms but will also integrate with other enterprise infrastructure software such as business process management, portals, enterprise content management systems and live collaboration tools.
  8. Web 2.0 collaboration will push its way into Performance Management. In particular, socially networked performance management will start to appear so that end users can tag metrics, graphs and reports in order to organise BI and PM content. This user defined categorising of content via tagging is known as Folksonomies and is already heavily used on the public internet on sites like Facebook, MySpace, de.licio.us, Digg, Flickr, Jotspot etc. Now it is coming inside the enterprise and will be applied to BI and PM content as well as other unstructured content. This means that users can see other users’ profiles and the tags that they have used to annotate BI and PM content. From here it means that BI and PM ‘tag clouds’ will form showing popular BI and PM tems that lead to popular BI and PM content and metrics. Also by following BI and PM tags we will see the dynamic formation of BI social networks consisting of people within the enterprise that have similar interests in acting on BI to improve performance. People will also be able to share reports and collaborate with others (in real time – e.g. IM, threaded discussions etc.) in Web 2.0 collaborative workspaces. Wikis (group publishing) will also come together with BI so as to fuel rapidly forming BI and PM workspaces that will be of exceptional value to the business.
  9. Search and BI are set to explode into popular use in 2008 as search opens the doors to mass access to BI content from a userbase that is not comfortable with BI tools
  10. BI reports will be capable of being published in document management and records management systems
  11. Master data management market size will continue to grow as companies try to wrestle with the complexity of their data and get it under control. Information and data architects will continue to be in demand with demand for such professionals potentially outstripping supply
  12. Companies will have to invest again in data modelling and data modelling skills. There is no doubt that standards here are dropping, many companies still have no data modelling tools at all and also too few people are skilled in good data modelling practices.
  13. Data management professionals will start to come together into integration competency centres so that people with skills in data cleansing, data integration, data modelling, master data management, enterprise content management, metadata management and ESB XSLT XML data translation are all co-located and can work together to solve the problem of enterprise data management
  14. Metadata management will become a mission critical issue if it is not already. Business users need access to business metadata to understand what data means and where it came from. Holding this metadata in spreadsheets is no longer acceptable. It must be made available to both end users and shared across multiple technologies. 2008 will see companies looking to act to solve this problem.

Well that’s all I have for now. Let me know your thoughts. I would be most grateful for your comments on any of this. Best wishes for a happy and prosperous New Year!

Tags: , ,
Posted in Uncategorized | No Comments »

IBM Finally Moves on BI by Acquiring Cognos

Monday, November 12th, 2007

Today IBM finally moved to plug the hole in its software product portfolio by announcing the acquisition of Cognos for $5bn USD http://www-03.ibm.com/press/us/en/pressrelease/22572.wss. This acquisition has been on the cards for a long time with several analysts including myself wondering how long it would be before we saw Big Blue move on Cognos (see my previous blog http://www.b-eye-network.co.uk/blogs/mt/mt.cgi?__mode=view&_type=entry&id=6976&blog_id=1) . This is almost a perfect fit for IBM given that it needed a BI vendor to provide tooling for reporting and analysis that had not much in the way of market share in the data integration space. The reason is that IBM is already the market leader in the data integration space with Information Server and does not want the ‘baggage’ of dealing with another product in that area. Cognos has never been a major player in data integration with its Data Manager product.

 

 

So you can expect IBM to immediately want to integrate Information Server with the Cognos product line. I also would expect that the WebSphere Metadata Server (bundled with Information Server) will be integrated with Cognos Framework Manager so that Cognos metadata ends up in the Metadata Server itself at some future release point. IBM also has had several joint development initiatives with Cognos and so there is a lot of integration already.


 

The only question mark will be in the area of performance management where Cognos has had a strong position for some time. IBM’s own Performance Management product IBM ActiveInsight will now have to come together with the Cognos Performance Management products. If this does not happen then the future of ActiveInsight as a product may well be in jeopardy. Time will tell.

 

So, the independent BI vendors are falling like nine pins as the software giants finally move in on the market. With Hyperion already falling to Oracle and Business Objects to SAP, it leaves only one large independent BI vendor left in the market – SAS. Of course SAS is privately owned and so will continue on that way until such time as its CEO (Dr Jim Goodnight) decides otherwise. MicroStrategy also remains independent also along with other smaller BI vendors. Expect further mergers and acquisitions to follow as the rest of the market consolidates to compete with the giants.

Tags: , , , ,
Posted in Uncategorized | No Comments »

« Older Entries