Recently I have noticed a lot of companies raising priority on Enterprise Information Management projects as it becomes clear that many companies realise that they do not have information under control. This includes both structured and unstructured data. It is clear that for unstructured data, enterprise content management, content authoring, tagging, search and taxonomy are all key. Also master data management has a part to play in deciding facets that can be used in taxonomies. Structured data needs to consider data naming and definitions, data modelling, data discovery, data mapping, data profiling, data cleansing, data integration, provisioning and data quality monitoring. There is a lot of work out there to do! The big question is how do you tie the two together. The secret is in MDM! What is your strategy for content authoring, content storage, content tagging, taxonomy, search, business glossaries, data integration ……Get in touch and let me know
Enterprise Information Management In Demand
August 12th, 2009Data Federation – Rapid Information Delivery in a Tough Economy?
April 1st, 2009Increasingly as I speak with my clients and CIOs I meet at various speaking engagements around UK and Europe, it is becoming clear that data federation can potentially offer rapid value to IT budget constrained companies that just can’t find the resources for another major database project. It may be that if you work in IT you are seeing increasing demand from business users for more reports requiring BI and non-BI information to help them manage their area of business responsibility in a more dynamic way.
In a recent paper by Jeremy Hope on Transforming Performance Management he states that “Most organizations want to adapt rapidly to changing events, but find that they are handicapped because of fixed budgets and poor forecasts. Adaptive organizations are able to respond more rapidly by switching resources dynamically to meet new threats and opportunities… “.
In order to do this there is no doubt that companies have to deliver information more rapidly irrespective of whether or not the data is in a BI system. What many cannot afford is going through a formal time consuming process of data warehouse change to bring in all information necessary. Data federation is capable of sourcing data from several places one of which would of course have to be a data warehouse or data mart. But with increasing amounts of valuable information residing outside BI systems (especially on the internet) it seems that data federation has a role to play as a delivery platform rather than having to change data models, ETL processes and creating another cube or relational data mart. My expectation is that over the coming year we will see an increase in demand for data federation software.
Several BI tools have been shipping data federation software as part of the BI tool bundle for some time to allow quick delivery of integrated information. Note that we are not talking about virtual data warehouses – on the contrary, data federation software rapidly integrates historical DW data with other data sources (operational data, internet feeds etc.) to deliver higher value information more rapidly. Therefore I see data federation software as complementary to BI systems. If you would like more information on data federation please see this article on how it works. There is also a white paper on Maximizing Business Value from Data Virtualization that talks about patterns and best practices to get the most out of this software. If you have already purchased data federation software or are considering it, let me know.
Data Governance Losing Priority in Some European Countries
March 30th, 2009Having just got back from a presentation tour in mainland Europe, it seems that in the countries I have spoken that Data Governance came out with a thumbs down vote among CIOs present in my sessions. In particular in Belgium it would appear to be not on their radar. Having probed for feedback into what exactly is high priority among CIOs attending my sessions it is almost as if raw ‘survival’ is taking hold. In other words, any IT project linked to business survival in this tough economic climate will get attention but not much else. Customer retention, self- service, cost reduction/containment and growth are high on the list. One CIO explained to me that his company’s priority over the next 12 months was to allow customers to customise the products and services they offer much more in the future. Therefore in addition to offering their own product lines on the web, they would be integrating their e-procurement with many back end e-suppliers so they can buy ‘on-demand’ to match what a customer wants. This means they want to allow customers to create their own custom ‘package’ before buying on-line and will stretch beyond their own products to stand out from the crowd. It seems to me that data governance and data quality to some extent are taking a back seat in favour of investment that will keep the revenue rolling in. I would be interested in your feedback. Is data governance a high priority in your organisation?
Flexible Fact Tables – Best Practice or Rope To Hang Business Users With?
March 20th, 2009Blogging occasionally offers up opportunity to open up a good debate. So here goes! Over the last several years I have observed data models in many different BI systems across different vertical industries where so called ‘generic’ fact tables have been designed with only one ‘generic’ measure. The objective of the design approach is that the measure in the fact table is supposed to hold ANY metric. Often this ‘generic’ measure column is then accompanied by some kind of type field to indicate what the measure actually is (what it means) and some other attribute(s) to indicate the level(s) in various dimension hierarchies that the measure stored is associated with. This helps indicate the additive nature of the metric. Also if it is a monetary measure it may have a currency field and if it is a unit measure it may have a field to explain the kind of units used, e.g. centimeters, litres, cubic metres etc. The stated advantage of these kinds of approaches is flexibility. Adding new measures becomes easy to accommodate as no change to the design is necessary. It is a perfectly good argument and certainly appears widely practiced by designers.
When it comes navigating such designs to develop queries (or even generate them) it is often the case that IT professionals developing reports for the business can figure out how to use retrieve the information required (although even IT developers can struggle). However when it comes to business users developing their own ad hoc queries and reports I frequently see these users really struggling to navigate the ‘flexible’ design first trying to figure out what measures mean, if the measure(s) is/are additive and whatnot. More often than not I see this resulting in real frustration among business users who end up getting aggregations in reports wrong and then start to lose faith in their new BI system. Of course IT steps in to rescue the situation by building more snapshot tables, more materialised views etc. burying generic ‘complexity’ to make the job easier for the user. More often than not these users also often resort to switching back to Excel to hold data outside any data mart so that they can look at data in a form they understand.
Have you seen this in your organisation? If so I want your feedback. Is it the case that so called ‘flexible’ design techniques are rope for end users to hang themselves with? My question is this. What is the best way that you see to design fact tables so that business users become productive and can easily understand how to get at the data when building their own reports? I am not so sure that being so generic is of business value. Sure it is flexible. But is it usable? What use is flexible design if a business user cannot understand it and make use of all that valuable data? Is it not better to have multiple metric attributes in a fact table (if multiple metrics are needed) with each attribute name saying what the measure actually is? Let’s have your input!
Trusted BI Critical in a Tough Climate
January 27th, 2009It’s pretty clear that Business Intelligence is becoming even more strategic in this tough economic climate as companies seek to have greater insight to help them stay profitable and keep making money. By default almost it would seem that trusted intelligence has got to be there for BI to be reliable and support confident decision making. Therefore on the eve of the Data Governance conference which starts next week (Feb 2-5) in London I thought I would put in a plug for Enterprise Data Governance. This is a fast growing topic and requires organisational change, processes and technology to manage it successfully. Structured data needs to be formally defined and named (a shared business vocabulary) and BI systems (data models, BI tool business views, reports etc) need to be changed to reflect these commonly understood data definitions.
In addition, data in disparate systems needs to be identified and mapped to the common definitions so as to gather knowledge on how to turn disparate data into trusted data. At this point you can then make sure that data integration is done in a fashion that creates trusted data for use in BI systems. Overall it is critical that you build modular data integration jobs (e.g. 1 data integration job per dimension) so that you can re-use data integration ‘services’ if needs be to guarantee trusted data every time. These days of course, data quality is built in to many data integration jobs and it is important to strive for this. When you have achieved this your data integration tool should provide a valuable set of metadata to support lineage should a user need to track data back to where it came from. Once your trusted data is available then you can monitor it to keep quality high and take action if quality deteriorates over time.
I’d be interested in your thoughts on enterprise data governance. Let me know what you are doing in this space.
Some BI Ideas for 2009
December 24th, 2008So here we are at the end of 2008. Another amazing year in the area of BI. In the 18 years I have been specialising in this IT sector, I suppose you might wonder what else is there to do here. After all BI is a mature market, indeed many of my clients today could be classified as very mature users of business intelligence (BI). Some are on their second, third or even fourth generation of BI system implementations, with data warehouses and data marts, web enabled ad hoc query, reporting, and OLAP tools already deployed and well established across their user base.
Yet there is plenty more that can be done. A key question is how can companies with mature BI set-ups strengthen and evolve their existing investment? I still see lots of opportunity as we head into 2009.
There are several areas emerging to enhance and build on existing BI investment that can offer more value to a business. These include:
- Integration of BI with Information Management infrastructure for trusted data
- Integration of BI with Performance Management software to roll up metrics into higher level KPIs
- Capturing of additional insight from unstructured content (e.g. customer emails) and from external information on the internet (e.g. about market intelligence and about what people are saying about your products and services)
- Event driven and on-demand Operational BI – a hugely exciting area for 2009 to continuously monitor operations and deliver right -time BI in the context of process activities for continuous business optimisation
- Integration of Enterprise Search with BI to open up broader access to intelligence via a search interface
- Exploitation of appliances for lower total cost of ownership on specific workloads
- Integration of BI with social software and collaboration workspaces to facilitate sharing and exploitation of knowledge in a collaborative environment. This is particularly relevant for those of you wishing to exploit products IBM Lotus Quickr quickplaces as well as Microsoft SharePoint workspaces. Integrating BI here will become increasingly important in 2009
These are just some of the ideas I will be discussing in the coming year and in a tough economic climate BI has never been more important. I wish all of you the best for the holiday season.
Oracle Enters The DW Appliance Market
September 25th, 2008Well now – has the inevitable happened? Larry Ellison, Oracle’s CEO finally recognised the value of the hardware/software combination in the business analytics marketplace. This market, pioneered over 20 years ago by Teradata and now thriving with many other players including Netezza, ParAccel, Datupia, ExaSol, Vertica and others, has now become a target for Oracle who have clearly had enough of competitors eating away at the Oracle database with lower TCO DBMS offerings optimised for analysis and reporting. With the recent acquisition of DatAllegro by Microsoft, IBM with its Balanced Warehouse and now Oracle entering the database machine market it certainly seems that the DW Appliance market is now becoming a hot competitive battleground.
The newly announced Oracle Exadata DW Appliance is jointly developed by Oracle and HP and will be sold directly by Oracle. The Exadata server runs the Oracle parallel server on Oracle Enterprise Linux. It has 8-HP Proliant DL360 G5 database servers, with
- 2 quad-core Intel Xeon Processor E5430 (2.66GHz)
- 32GB memory
- 1-HP InfiniBand Dual Port HCA
- 4-146GB SAS 10K hard disk drives
- 4-24-port InfiniBand switches
- 14-HP Exadata Storage Server Hardware–each is an HP ProLiant DL180 G5, with 2 quad-core Intel Xeon Processor E5430 (2.66GHz)
- 8GB memory
- 1-HP InfiniBand Dual Port HCA
- 12-300GB SAS or 12-1TB SATA disk drives
My question on this announcement is given that HP are jointly in on the Exadata product offering with Oracle, what does this mean for HP’s own DW Appliance offering – the HP NeoView Appliance? This is also a parallel DBMS product that competes with Oracle. I assume that with HP playing in both markets (its own DBMS product on its own hardware plus the hardware behind the Oracle Exadata offering) that it is seeking to maximising the revenue it can take by covering all bases. Time will tell. In my opinion it is clear that with so many vendors now in the DW Appliance market it is going to take a lot more than just TPC-H benchmarks to get a differential. It certainly means customers will have to look closely at performance claims. Everyone will claim they are the fastest which could easily result in prospective customers demanding more to distinguish one vendor from another. For this reason I believe that analytic application appliances have to happen (analytic application pre-installed on a DW Appliance). Vendors who go deep on vertical analytic application appliances could carve out a very lucrative business when you combine this with the attraction of low TCO DW Appliance offerings.
Flood Gates on Operational BI on the Verge Of Breaking Wide Open
July 30th, 2008It seems that everywhere I look the number of vendors gearing up for Operational BI is just massing as if awaiting an onslaught on the market. Among them include HP, IBM, InforSense, Progress, Tibco, SL, ThinkAnalytics and many more. Key to this is event driven data integration, in-memory data, predictive analytics and rules engines. Vendors like SL have even released a data cache for in-memory analytics with its RTView product. Progress is also pushing with its Aparma product and long term pioneer ThinkAnalytics are also doing well. Clearly the giants are also moving to get all their pieces in play. IBM’s acquisition of German rules vendor iLOG looks like it could also be used in the world of on-demand and automated decisioning. Also SAP Business Objects Labs have released a prototype on event-driven BI and seem to be partnering with SPSS on predictive analytics.
It’s all heating up, yet in the UK it staggers me the number of companies that are not seeing the benefit of these emerging technologies. Some verticals would reap massive benefits by exploiting this technology. If you are interested in Operational BI drop me an email at info@intelligentbusiness.biz
DatAllegro – The New Microsoft DW Appliance?
July 28th, 2008Why is it every time I take a week off on holiday something major happens in the BI market? I am of course talking about Microsoft’s Announcement to acquire the data warehouse appliance vendor DatAllegro. The message on this is certainly obvious, the scaling of SQL Server. It’s a very interesting announcement. With the exception of Netezza who have done well here in the UK and in Europe, many of the DW appliance vendors have been struggling. Almost all of them have been chasing business in the same vertical industries that have high volumes of data (e.g. Telco, Retail, Financial Services). Now the one giant that people were wondering about in terms of parallel DBMS scalability, has moved. However we are clearly going to have to wait to see how well SQL Server scales in this kind of setup. Even prior to this announcement the myth that SQL Server would not scale beyond 1 Terabyte has long been proved incorrect however. I have certainly had clients running single instance SQL Server BI system databases at around 13-15 terabytes for a number of years now. No doubt there are larger configurations than that out there. However this announcement will certainly lift Microsoft customer confidence that Redmond are serious about offering a scalable SQL Server option on non-proprietary hardware that starts to compete with the parallel DBMS offerings of IBM, Oracle and Teradata as well as other DW Appliance offerings. Time will tell what will happen and how competitive this will be. SQL Server Integration Services (the Microsoft ETL tool that ships with SQL Server) will also have to scale however to get larger data volumes on to a parallel SQL Server. So we have to see what Microsoft will do here. This announcement also offers up an interesting option for Kalido who recently announced support to generate for Microsoft SQL Server.
Kalido – Its a Wrap For Microsoft
July 16th, 2008Yesterday Kalido announced the availability of Kalido Universal Information Director for Microsoft SQL Server which will speed up development of BI systems based on the Microsoft Platform. By using Universal Director (part of the Kalido Information Engine suite of technologies) it is possible to capture the metadata and automate the generation and population of Microsoft SQL Server® Analysis Services (SSAS) cubes with data from the Kalido Dynamic Information Warehouse Automatic generation of SSIS jobs, SSAS cubes needed speeds up Microsoft BI system development and enables rapidly constructed SSAS cubes to then be accessed from Microsoft Office Excel, PerformancePoint Server and SharePoint. Also 3rd party BI tools can access this data. Effectively this is like Kalido acting as a wrapper for the Microsoft environment.
Note that Kalido is supports automatic generation of SAP’s Business Objects Universes and IBM’s Cognos platform. If Kalido keeps adding heterogeneous BI platform support it could put itself in a powerful position with respect to being able to leverage common metadata to integrate BI systems built on multiple heterogeneous technology platforms across different lines of business in large enterprises. I have yet to determine if Kalido technologies can be deployed on the Eclipse platform. If so this would be even more promising. Also this announcement of Microsoft support may well also be welcomed by small and medium size businesses (SMBs) who struggle to afford, find and retain skilled resources to build BI systems. Price point will be all important in the SMB market however especially as open source BI offerings have also penetrated this market.