In this post, I take a temporary diversion from the “business side of business intelligence” to take a quick look at the uniqueness of Ab Initio deployment services. This was prompted by a business contact who was struggling to differentiate in a sea of IT service providers.
With the IT budgets under scrutiny and time-lines as tight as they have been in years, business and IT managers cannot afford to make mistakes when selecting a specialty implementation partner. No where is this more fitting than when considering retaining the unique skills of an expert Ab Initio practitioner.
Ab Initio is a unique and powerful general purpose computing platform that is most commonly applied to complex and large volume data integration projects typically associated with data warehousing and large-scale analytic data stores. While its tempting to think of Ab Initio as just another technology that any engineer worth his or her salt should be able to just “pick up”, consider the extensive background requirements that have been associated with the timely and successful implementations of this powerful platform.
While there are numerous factors to consider, below are several guidelines that will help direct you in your search for an Ab Initio Consulting partner.
Experience in complex situations: Ab initio’s impressive graphical development environment (GDE) makes it very easy to get started and begin using, but due to the complex environments in which it is typically found, it is difficult to master and deploy in an optimal manner (thus necessitating expensive re-design work later if not done well). Most ETL development is pattern-oriented and fairly routine. To ensure you are working with an expert, seek a partner who has demonstrated experience in doing something out of the ordinary. This could be any innovative application of the technology (data mining, conversion utillities, parsing engines, etc.) that sets them apart from the “let’s just bill hours” crowd. Some keys to more advanced skills include Meta-programming or molding application meta data at run time (a particular strength of Ab Initio), re-usability and user-defined components (to name a few).
Mastery of Ab Initio’s Standard Environment: Ab Initio’s standard environment establishes common methods and deployment practices as well standard naming conventions to ensure a uniform delivery approach and ease of maintenance throughout an organization. When seeking an implementation partner, ensure that they truly understand the standard environment (most developers use the standard environment, but the majority don’t truly understand it) and are prepared to enforce its developmental rigor throughout the project team. Failure in this point could possibly lead to dramatic leaps in maintenance time and money (e.g. having to reconcile 12 different ways of implementing Ab Initio within the same organization).
EME (Enterprise Meta Environment) utilization and administration: While not all organizations license the EME, if yours does (or is planning to), ensure that you are working with an Ab Initio consulting partner who has an intimate understanding of its function, structure, configuration standards and preferably, administration. The more comprehensive their knowledge, the more opportunity to leverage the resulting meta data (e.g. change control, where used, impact analysis, etc.) down the road.
Mastery of parallel processing, high performance computing and ETL job performance: Ab Initio is not a kitty car, and as such, you don’t want a toddler heading up your deployment. You’ve most likely invested in Ab Initio (and the surrounding database, network and server stack) because you have critical, high-volume data requirements that mean millions of dollars to you. Seek a partner well grounded in the deployment of parallel processing and high performance technologies and applications with the maturity to know what’s at stake. Regarding Ab Initio applications themselves, check references to know that design with performance as a key consideration (e.g. proper partitioning, number of components, etc.) is a prominent attribute. I recall several situations where a senior engineers improved Ab Initio ETL job performance 10, 20, even 100X by leveraging proper design techniques. Think of the impact to batch windows, data availability, service level agreement compliance and user satisfaction.
Deep understanding of associated environments and technologies: Ab Initio does not run in a vacuum (or the cloud yet for that matter). In fact, Ab Initio is typically found with a cadre of other high performance technologies (e.g. database, network, servers, etc.) that enable the highest possible processing performance. Ensure that your Ab Initio consulting partner has relevant experience in the technologies and techniques associated with the related technologies including databases (e.g. Teradata, Oracle, DB2 UDB, Netezza, etc.), Unix/Linux (including Korn Shell Scripting).
Mastery of data warehouse and data integration fundamentals: A smart engineer once explained to me that “a fool with a tool is still a fool”. He meant that its not enough to just understand the technology, but its application within the environmental and business context is equally if not more important. Key considerations should include their knowledge of data architecture (logical and physical data modeling), ETL architecture (best practices for data acquisition, staging, loading and updating the data warehouse), master data management (reference data), meta data, general application development and ability to support downstream analytic uses (reporting, business intelligence, other analytics).
Excellent fundamental consulting skills and partnership style: All of the tools, techniques and talents listed above certainly provide a fundamental starting point for considering an Ab Initio consulting partner. Equally important is the general “personality” of the firm you are considering for the project. Are they easy to do business with? Do they have integrity (i.e. do their actions match their words)? Are they out to help you solve your problem or just to bill hours? How is their communication? Their commitment to your success? Do they provide you direction and a road map? Methods? (with a Small “m” as we want to avoid methodology “dictators”), and do they provide clear documentation helping you articulate your own organization-specific methods of deployment? Be sure to be just as comfortable with their business approach as you are with their technology prowess.
Explain to vendor management that price is not the only consideration: Yes the economy is slow and budgets are tight, but you have most probably invested millions in your high performance data warehouse environment and are no doubt anticipating reaping the maximum return on investment. Implementing Ab Initio in a sub-optimal way, will not only hinder the benefit of speed, data availability and nimble business decisions, but could literally add millions more to your investment in human costs (re-work and maintenance) and hardware (e.g. servers, processors, memory, etc.). While tempting to seek body-shop and staff augmentation firm pricing, consider data warehouse and BI specialty firms who while are bit pricier, can help ensure an optimal and sustainable information delivery capability.
Selecting the right Ab Initio consulting partner can help you avoid mountains of re-work and ensure that your maintenance burden is as light, nimble, flexible and inexpensive as possible. More importantly, proper design and deployment of this powerful technology will help prepare your organization to leverage timely analytics to optimize its key business operations.
Best wishes in your partner selection endeavors.
Posted by Jeff Seifert 