Test Reliability - Employment Testing
Validity and reliability are two terms that go hand
in hand in any discussion of personality assessment or other forms
of employment testing. It is important to understand the difference between
the two terms. Validity basically refers to research that
provides evidence that a test actually measures what it is supposed
to measure. Reliability is more of a measure of consistency.
If you took a tape measured the length of your desk
and found it to be 60 inches long, you would expect it to measure 60
inches again tomorrow. But what if it measured only 56 inches
when you measured it the following day? You would then have to find out what went wrong with
the measurement. A reliable measure of the desk's length would
always have to yield the same measure. If it did not we would
say that the measure is unreliable.
If a person takes a personality assessment today and scores
high in a trait like dominance, and we retest that same person six
weeks from now, we would expect the score to still be high in
dominance. If however, the individual scored low in dominance,
we would have to conclude that the measure was inaccurate and
unreliable. The measure would be of little value to us.
That is why we need to know that any type of employment test is
reliable before we begin using it.
If a measure (a trait) on a personality assessment was reliable, we would
expect to see nearly identical scores on a retest. The smaller
the variance the more accurate the measure. The reliability of
a measure is based on a statistic called a correlation coefficient
which ranges from +1.00 to -1.00.
The correlation coefficient measures the strength
between two variables. When a coefficient approaches plus or
minus 1.00, we have a very strong relationship, with +1.00
representing a positive relationship and -1.00 representing a
negative relationship. A correlation coefficient of 0.00 would
indicate no relationship.
The most commonly used method of determining
reliability is through the test-retest method. The same
individuals are tested at two different points in time and a
correlation coefficient is computed to determine if the scores on
the first test are related to the scores on the second test.
A high correlation coefficient would tell us that
the individuals scores on the first test are very similar to their
scores on the second test. The significance and importance of
reliability should be quite obvious. Without consistency of
measurement, there would be absolutely no benefit or value to be
derived from the employment test. A math test that could not
differentiate between high math competency and low math competency
would not be of any value. This same logic is especially true
regarding personality assessment because of the complexity of the
measures or behavioral traits involved.
There is no value where there is no consistency
If you are testing your applicants with an employment test that
does not have a fairly high level of reliability, the usefulness of
the test results will be greatly diminished because there will be
little or no
consistency in the results. People that score low in dominance
may actually prove to be quite high in dominance. You would be
paying for and collecting data that would be absolutely useless from
a historical standpoint. At some point in time you will want
to subject that historical data to deeper analysis to discover
correlations that influence or more strongly predict job success or
failure. A reliable assessment can provide quality historical
data that is well worth analyzing.
If want to use a pre-employment assessment to hire people for
jobs that require certain behavioral traits and qualities, you
certainly don't want to end up with pot luck, so by all means select
a good quality assessment to do the job. Even the most
expensive assessments are dirt cheap when compared to the cost of a
bad hiring decision. With an unreliable instrument, you
will be unable to relate the behaviors to any type of theoretical
behavioral model because the rules will always seem to be
changing. Such a situation would make it very difficult to
learn about the basic fundamentals of human behavior.
The price of everything and the value of nothing
If the behaviors you observe are inconsistent with the
personality assessment results, how are you ever going to be able to apply the
results to any type of hiring pattern? What is the benefit of
such a system? There are none. This is important to
understand because slick packaging and marketing tactics
are very effective in today's world.
I was approached by a competitor at a business luncheon a few
years ago. He wanted to congratulate me on a large account that
I had acquired. We got to talking about the various employment
assessments on the market. He asked me about
the assessments that I was using. After a brief discussion of
the various products, he then proceeded to tell me
all about the "tests" that he had developed and was marketing
I asked him how long it took him to do the validity and
reliability studies. As it turned out, most of the assessments were developed in
a very short period of time. No reliability or validity studies were ever
done! Why would any company buy such products? Mainly
because they can be priced very cheaply. Reliability and
validity studies require a great deal of research to do properly,
not to mention time and money.
You cannot tell anything about the quality of a personality assessment by looking at the test booklet or the assessment
questions. The value is in the quality and consistency of the
measures. Do not be fooled into thinking that two different
assessments are of equal value just because they appear to measure
the same personality traits or mental aptitudes. There are
major differences from one assessment to the other.
Even a test that is a pirated copy of another test publisher's
assessment will probably produce totally different and
unpredictable results unless the same scoring algorithms are used to
score the assessment. Test publishers closely guard those algorithms, so never
let anyone convince you that two seemingly identical tests will
produce the same results. They may be as different as night
and day. By all means, ask questions about the history of the
assessment. Find out how long it has been on the market.
Inquire about the validity and reliability studies and find out if
copies of those manuals are available for purchase. Any
reputable company will be happy to take your order and put one in
Return to Top