* Company B was unable to supply the element "Aggregate Value of Vehicles". This was a non-imperative element so no disqualification applied.
* Company A was not able to provide one of the lifestyle elements, Company B was unable to provide two, and Company C could provide none. All lifestyle elements were weighted with a 1 (non-imperative data) so no disqualification applied.
Looking at the scores alone, Companies A and C are virtually tied. However, while Company C had the highest accuracy for its elements, it also had the lowest match rate at 72% (a full 20% below Company A) and did not provide any of the lifestyle elements.
Company A, while having a lower accuracy rating, had the highest match rate (a high number at 92%), the highest number of elements returned per matched record, the highest value in data usability, provided a data dictionary, and had rankings of 2 in data freshness, delivery time and customer service.
The question for HomeBank at this time is whether to choose Company A's high quality scores and somewhat lower accuracy or Company C's lower quality scores and higher accuracy.
HomeBank decided it would have to look at the prices before making this decision. Quotes were reviewed for the full 3 million record file enhancement. HomeBank wished to process this file quarterly.
Company A Company B Company C
--------------------------------------------------------------------------------
Price/M + Processing $40.00 $33.00 $25.00
Quarterly Price $120,000 $99,000 $75,000
--------------------------------------------------------------------------------
Yearly Price $480,000 $396,000 $300,000
Making a Decision
Company C was clearly the low cost winner. In a year's time the total cost for data enhancement would be $180,000 less than Company A's price. To make a final decision the company analysts, the actual users of the data, were asked for their views.
When queried, the analysts building the models stated that the level of accuracy for the elements returned by Company A were within acceptable ranges and that the high match rate and number of elements returned were desirable.
Because they were using the data for modeling and not simply as a list file it was more important for them to have enough data to make their models perform properly. They were concerned that the low match rate of Company C would be detrimental to the accuracy of their models. They stated they would prefer Company A's data to that from Company C if given the choice.
Decision
Company A was chosen as the winner of the contract because of its match rate, depth of elements returned, and good scores in all other areas. The difference in usability was deemed sufficiently large enough to warrant the choice of the higher-priced data from Company A, particularly in light of the views of the company analysts and their use of the data.
Learning From HomeBank
It is important to review the fact that HomeBank decided to rank and weight its data needs and incorporate these into the test. If weighting had not been used, the results of the test may have come out differently.
It is not important that all tests be designed with weighting factors unless they make sense to your company and reflect your data needs. With respect to data testing there are no scientific answers that can be provided on the choice of weighting factors. Common sense must prevail over empirical methods.
Note as well that despite Company C having close to the same score as Company A with a lower cost, Company A's data was chosen. Test scores are good for comparisons and will help you to evaluate the results of the test but should not necessarily be used as the sole determinant of the outcome. Just as IQ tests would be inappropriate as the only factor for hiring new employees, considering only the data test scores without other factors would be a mistake.
