Selecting software tools to support your business intelligence effort – or any IT project for that matter – can be one of the most important preliminary decisions. Yet whenever tight
schedules call for shortcuts, the software tool selection process tends to suffer time after time. Tools end up selected based on flimsy criteria, the promises of a charismatic salesperson, or
stories of success from customers who have different environments and needs. Down the road, when the project discovers that the tool doesn’t meet critical needs, workarounds must be developed
and the schedule is protracted.
So how does one pick the right tools for their organization and reduce solution risk – all the while staying on schedule? This article presents techniques for addressing all of these issues
throughout the entire selection process.
Don’t Go it Alone
As soon as the need for a software tool arises, identify the stakeholders of the tool. While it is often the case, don’t assume that the stakeholders of the project are the same as the
stakeholders of the tool. On one enterprise data warehouse project for a federal government agency, for example, the tools selected for the project would become the enterprise standard tools for
the entire agency. If any other IT project in the agency wanted to use a different tool, they would have to submit a waiver for consideration. Thus, it was critical to include voices external to
the project.
Select five to seven of these stakeholders who are available to participate on an integrated product team (IPT) – a selection board that recommends the software tools for the project. Try not
to add any more stakeholders because problems, such as finding time to meet or running into decision deadlocks, begin to increase in magnitude and frequency as people are added.
Prior experience with a particular tool is not required of IPT members. In fact, this can be desirable because they bring no biases to the table. Just understand that they will need an educational
briefing prior to beginning the process to address what the tools do, their alternatives, and their key differentiators. Augment the IPT with two or three members of the project who will be doing
the majority of the work associated with the tool selection process: the heavy lifters.
Set IPT Expectations Up Front
Hold a kick-off meeting with the IPT after the members are identified. Be frank about their time commitments, which actually can be quite low – just a few meetings over the course of a couple
of weeks. Describe the assignment of duties. The heavy lifters will do most of the research and candidate selection, whereas the IPT will be consulted at regular checkpoints to assess the heavy
lifters’ decisions.
Most people favor the idea of competition, and tend to want a “bake-off” between the top few tools. While a tempting practice, it adds unnecessary overhead to the process and draws out
the schedule. Instead, set expectations with the IPT that their job is to pick the best of the top tools for a proof of concept; and if that tool succeeds, that tool will be selected. Describe that
the process will employ a series of increasingly comprehensive analyses – “hurdles” that become more difficult for tools to clear as the process advances.
Market Survey
Markets for software tools tend to have a small number of dominant vendors with tools that meet most general needs fairly well, surrounded by a large number of niche vendors with tools that meet
specialty needs extremely well. Also take into account that software tool markets are dynamic – vendors often purchase competitors’ tools and some tools drop out of the market
completely, so the tools available three years ago may not be the same available today.
Thus, it is important to spend one day on this hurdle, generating a list of tools that appear to be within the desired tool’s class (e.g., OLAP, advanced analytics, geospatial). Don’t
spend any longer on the market survey, or tools on the fringe of the class will begin to creep onto the list of candidates. Web searches and technology research groups, such as Gartner, Faulkner,
and Forrester, can yield good information quickly for this hurdle.
Architectural Review
Identify the architectural limitations or technology standards of the project environment and compare them to the tools developed from the market survey. For example, must the tool be able to run
on a mainframe or a UNIX server? Must it be compatible with Oracle or DB2? On their websites, most vendors provide data sheets for their tools that describe the tool’s compatibilities and
minimum configuration needs, so many tools can be ruled out with a minimum of effort. Again, spend only a day on this activity – if this type of information cannot be found quickly, it may
suggest that other information availability problems may be forthcoming from the vendor. The result of clearing this architectural review hurdle is a list of tools that can be installed into the
project’s environment.
Requirements Review
The objective of this hurdle is to spend about three days reducing the number of tools for the demonstration hurdle to no more than three by comparing the tools to the project’s most critical
needs. Do not attempt an exhaustive requirements analysis and comparison at this point, as that will only protract the schedule. For example, if real time capabilities and ability to access
federated data are the most important requirements of a project’s data integration tool, examine these features closely.
Additionally, take a look at the vendor at this point. Are they a tool reseller? If so, consider looking instead at the original vendor, who obviously knows the tool best. Also, see if any recent
or current events cast doubt on the future viability of the vendor. If industry analysts believe it likely that a vendor will be acquired by another, you should be looking at the potential buyer as
well and their track record for supporting tools obtained by acquisitions.
Vendor Orientation
Hold an orientation meeting, either face to face or through a conference call, with the vendors to set expectations and ground rules. Only the heavy lifters need be involved during this meeting, so
do not invite the entire IPT. Have a sign-in sheet – vendors sometimes vary who they send, so this tool is a simple way to prove which vendor representatives were at which meetings and to
have contact information at hand.
Elements of discussion during the orientation meeting include:
- Provide the day (or two days) on which the demonstrations will occur. The days should be consecutive if at all possible. Secure each vendor’s agreement to the dates and times.
- Stress the need for timeliness, both in the demonstrations themselves and in responding to the IPT if questions are submitted to them. Tell them that because the schedule of the project cannot
be held back by tardy responses, a vendor’s timeliness and cooperation will be one of the deciding factors. - Provide the vendors with a list of the project’s requirements and encourage them to model their demonstration around those requirements.
- Limit the demonstrations to no more than three hours, including questions and one or two short breaks in between.
- Notify the vendors that they should supply their own equipment (e.g., LCD projectors, laptops). Discuss any parking or building security issues, if any apply.
Sample Business Problem
Tools have different technical architectures, user interfaces, and capabilities. To improve the IPT’s ability to compare and contrast the strengths and limitations of each tool, consider
asking each vendor to wrap their presentation around a common business problem – one that is relevant to the project.
Create a set of inputs and provide some context around those inputs. Ask each vendor to use their tool in the demonstration to prepare a set of outputs. With each tool using the same input, it
becomes easier to compare the outputs. Distribute the inputs of the sample business problem to the vendors at the vendor orientation meeting.
Consider this approach if the project has inputs that are readily available for distribution. If no inputs are readily available, this process is still useful, but may take a day or two to prepare.
Circle the Wagons
After each vendor’s demonstration, take a short break. Reconvene the IPT after the vendor has departed to discuss the tool. By debriefing immediately, the IPT can discuss the pros and cons
while the tool is still fresh in everyone’s minds. Have a group facilitator manage the evaluation process and record the results of the discussions.
Weighting vs. Non-Weighting
Many tool evaluations use sophisticated weighting schemes used to score each tool – the tool with the highest score being the best. But this approach takes time to develop the weights and to
get agreement by all the IPT members on those weights. Weights are also somewhat arbitrary because converting a qualitative notion of importance into a numerical value is not often repeatable.
Instead, consider saving time by using a balanced scorecard approach for evaluating the tools. This lets each IPT member use his or her own implicit weighting scheme. Select no more than ten
categories of differentiation, such as vendor cooperation, functionality, and ease of use. While circling the wagons, ask each IPT member to rank the tool in each category. Calculate averages for
each tool across each category if a quantitative means of determining a leading tool is desired. Consider all of the scores across the categories holistically and identify the leading tool.
Proof of Concept
Inform the vendor of the leading tool that no deal is complete until the tool has been installed into the project environment and the IPT confirms that the tool meets the critical needs of the
project. Ask them for help, as some vendors are eager to assist in proof of concept efforts by making their consulting staff available to the project free of charge – after all, the faster
the proof of concept completes, the sooner the sale can be recorded. Hold off on telling the other vendors involved in the demonstrations that a tool has been selected until the proof of concept is
complete. If the leading tool fails the proof of concept hurdle, the IPT will need to subject the next best tool to the process.
Elements of the proof of concept effort may include:
- Having the vendor provide training to the heavy lifters. Training classes are excellent means of determining tool strengths and limitations.
- Selecting a relatively complex use of the tool on the project and prototyping the tool against it.
- Reviewing the user documentation for any glaring limitations.
If there is sufficient doubt as to whether the tool can help bring about project success, consider expanding the proof of concept to the next best tool. Otherwise, inform the IPT of the decision
and commence with the procurement.
Summary
The risk of selecting the wrong software tools for a project can have an enormous impact on the success of a business intelligence project. Rather than cutting selection processes down to the bone
by relying on less-than-optimal approaches, consider these techniques to pick tools smarter without extensive schedule delays. In the end, tools are considered that may never had been considered
normally; stakeholders, having been involved throughout the process, demonstrate full buy-in on the tools selected; and the actual tool capabilities and limitations in support of the project are
revealed prior to commitment to the tool and before it is too late to make a change.