Test Reliability - Employment Testing

Validity and reliability are two terms that go hand in hand in any discussion of personality assessment or other forms of employment testing.  It is important to understand the difference between the two terms.  Validity basically refers to research that provides evidence that a test actually measures what it is supposed to measure.  Reliability is more of a measure of consistency.

If you took a tape measured the length of your desk and found it to be 60 inches long, you would expect it to measure 60 inches again tomorrow.  But what if it measured only 56 inches when you measured it the following day?  You would then have to find out what went wrong with the measurement.  A reliable measure of the desk's length would always have to yield the same measure.  If it did not we would say that the measure is unreliable.

If a person takes a personality assessment today and scores high in a trait like dominance, and we retest that same person six weeks from now, we would expect the score to still be high in dominance.  If however, the individual scored low in dominance, we would have to conclude that the measure was inaccurate and unreliable.  The measure would be of little value to us.  That is why we need to know that any type of employment test is reliable before we begin using it.

If a measure (a trait) on a personality assessment was reliable, we would expect to see nearly identical scores on a retest.  The smaller the variance the more accurate the measure.  The reliability of a measure is based on a statistic called a correlation coefficient which ranges from +1.00 to -1.00.

The correlation coefficient measures the strength between two variables.  When a coefficient approaches plus or minus 1.00, we have a very strong relationship, with +1.00 representing a positive relationship and -1.00 representing a negative relationship.  A correlation coefficient of 0.00 would indicate no relationship.

The most commonly used method of determining reliability is through the test-retest method.  The same individuals are tested at two different points in time and a correlation coefficient is computed to determine if the scores on the first test are related to the scores on the second test.

A high correlation coefficient would tell us that the individuals scores on the first test are very similar to their scores on the second test.  The significance and importance of reliability should be quite obvious.  Without consistency of measurement, there would be absolutely no benefit or value to be derived from the employment test.  A math test that could not differentiate between high math competency and low math competency would not be of any value.  This same logic is especially true regarding personality assessment because of the complexity of the measures or behavioral traits involved.

If you are testing your applicants with an employment test that does not have a fairly high level of reliability, the usefulness of the test results will be greatly diminished because there will be little or no consistency in the results.  People that score low in dominance may actually prove to be quite high in dominance.  You would be paying for and collecting data that would be absolutely useless from a historical standpoint.  At some point in time you will want to subject that historical data to deeper analysis to discover correlations that influence or more strongly predict job success or failure.  A reliable assessment can provide quality historical data that is well worth analyzing.

If want to use a pre-employment assessment to hire people for jobs that require certain behavioral traits and qualities, you certainly don't want to end up with pot luck, so by all means select a good quality assessment to do the job.  Even the most expensive assessments are dirt cheap when compared to the cost of a bad hiring decision.  With an unreliable instrument, you will be unable to relate the behaviors to any type of theoretical behavioral model because the rules will always seem to be changing.  Such a situation would make it very difficult to learn about the basic fundamentals of human behavior.

If the behaviors you observe are inconsistent with the personality assessment results, how are you ever going to be able to apply the results to any type of hiring pattern?  What is the benefit of such a system?  There are none.  This is important to understand because slick packaging and marketing tactics are very effective in today's world.

I was approached by a competitor at a business luncheon a few years ago.  He wanted to congratulate me on a large account that I had acquired.  We got to talking about the various employment assessments on the market.  He asked me about the assessments that I was using.  After a brief discussion of the various products, he then proceeded to tell me all about the "tests" that he had developed and was marketing quite successfully.

I asked him how long it took him to do the validity and reliability studies.  As it turned out, most of the assessments were developed in a very short period of time.  No reliability or validity studies were ever done!  Why would any company buy such products?  Mainly because they can be priced very cheaply.  Reliability and validity studies require a great deal of research to do properly, not to mention time and money.

You cannot tell anything about the quality of a personality assessment by looking at the test booklet or the assessment questions.  The value is in the quality and consistency of the measures.  Do not be fooled into thinking that two different assessments are of equal value just because they appear to measure the same personality traits or mental aptitudes.  There are major differences from one assessment to the other.

Even a test that is a pirated copy of another test publisher's assessment will probably produce totally different and unpredictable results unless the same scoring algorithms are used to score the assessment.  Test publishers closely guard those algorithms, so never let anyone convince you that two seemingly identical tests will produce the same results.  They may be as different as night and day.  By all means, ask questions about the history of the assessment.  Find out how long it has been on the market.  Inquire about the validity and reliability studies and find out if copies of those manuals are available for purchase.  Any reputable company will be happy to take your order and put one in the mail.

Return to Top

Site Map