Equity and Equality, Fairness and Bias: Making Connections in Credentialing

The topics of Equality and Equity relate to concepts of Fairness and Bias in credential program development and evaluation in ways that are not always clear, especially considering for the most part the former two are commonly considered societal issues, and the latter are more technical concepts that live in the credentialing world. I aim here to clarify the connections.

We read about Equality and Equity a lot these days, and in recent times there has been a very insightful graphic that has appeared in various social media outlets that explains the difference between the two. Both are noble approaches but seek to accomplish markedly different outcomes. On the one hand, Equality is about sameness, and it looks to promote fairness and justice by giving everyone the same thing. But this works only if everyone starts with the same. So, in the graphic, Equality works only if everyone were the same height. On the other hand, Equity is about fairness, and it’s about making sure people have access to the same opportunities. Sometimes our history and differences create barriers, and as the graphic shows, if we provide people with what they need to overcome barriers so all people have the same opportunity (to view the game, in this example), then we are being equitable. But how do these concepts relate to the world of credentialing?

Very simply, both concepts come into play. For example, licensure and certification bodies, like my clients, want to make sure that all candidates/employees have equal or the same learning and testing materials – that’s equality. But on a larger front, these credentialing organizations need to make sure that all groups of candidates, for example across geographic regions, languages, and socio-economic status etc., have the same opportunity to access the learning and testing materials – that’s equity. But, these concepts as explored are more social than technical. In the credential testing world, we have some important technical considerations to think about as well.

Two very important technical concepts in credentialing focus on validation of scores from associated tests: Fairness and Bias.  These terms are sometimes mistakenly used interchangeably, especially by the public (Zieky, 2006).  Test fairness is a value-laden judgement based on several factors, including moral, ethical, social and sometimes legal standards (Thorndike & Thorndike-Christ, 2010), and is often expressed in guidelines for fairness in testing (Zieky, 2006).  Tests that are unfair result in systematic differences in scores for predefined groups of examinees – that is bias.  So, test bias is a psychometric property of test scores; it is the quantitative evidence that supports claims of test unfairness (Furr & Bacharach, 2008).

Fairness, in the credential testing realm, is a type of validity evidence in which the principle rests that test results are based solely on the ability of candidates to provide safe, competent practice.  It is important to note here that not only examination items, but exam policies may also hinder the performance of pre-defined groups of candidates based on factors such as their gender, language, culture, ethnicity and disability. Any exam item or examination policy that systemically advantages or disadvantages groups of candidates is said to be unfair. Thus, test fairness is more akin to the social concept of equity.

On the left side of the graphic, the provision of crates to boost sightlines may represent the provision of equal access to learning material to boost test performance. But, if learners – potential examinees – start their learning in different places (or have different heights like in the graphic), the crates, or the learning material do not help everyone.  For example, if learning material is made available online only, then perhaps those prospective examinees with limited internet access would not be helped as much as those with ease of internet access. The provision of the online learning materials is equal but not equitable; therefore, it is not fair, and could lead to bias in test scores for those examinees with limited internet access. For everyone to benefit from the learning material, symbolized by the crates, prospective examinees must be given the learning materials in a way that addresses their needs – the right side of the graphic.

How may we identify an unfair or inequitable exam? Statistical comparisons of performance data of pre-defined groups of examinees (identified by gender, ethnicity, age and the like who should in theory have equivalent levels of ability) can help identify the presence of bias in the exam results.  The source of the bias then needs to be found and addressed. It is the responsibility of the credentialing body to provide evidence that results of exams are not biased against subgroups; ultimately this requirement is manifested first in test development and analysis guidelines that require fairness studies be conducted periodically (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014).

When I work with my clients, whether they be bankers, lawyers, IT professionals or accountants, I always provide them with guidance to promote equity in their credentialing programs, if not for the social and altruistic reasons, but for the simple reason that an inequitable or unfair program creates bias in results that seriously undermines the validity of any inference drawn from them.  These credentialing programs are too expensive to be threatened by avoidable unfairness; moreover, the cost on a societal level of unfair programs is even greater.

Graphic: http://interactioninstitute.org/wp-content/uploads/2016/01/IISC_EqualityEquity.png

Further Reading:

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing (5th ed.). Washington, DC: American Educational Research Association.

Furr, M. R., & Bacharach, V. R. (2008). Psychometrics: An introduction. Thousand Oaks, CA: Sage.

Thorndike, R. M., & Thorndike-Christ, T. (2010). Measurement and evaluation in psychology and education (8th ed.). Upper Saddle River, NJ: Pearson Education.

Zieky, M. J. (2006). Fairness review in assessment. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 359-376). New York, NY: Routledge.

Putting Evaluation Users First to Build Evaluation Capacity

Recently I had the opportunity to write a piece for the Canadian Government Executive Magazine, an electronic and print publication that reaches some 70,000 readers per month. Many of these readers are key players in evaluation in Canada, from program managers to evaluation sponsors and consumers.

Since we work predominantly in the private sector, many of the organizations we work with are deeply reliant on performance measurement metrics to gauge success on key initiatives. There is nothing wrong with this approach, but sometimes a different lens is necessary to gain a more fulsome look at program operation and outcomes. This evaluative lens is new to many of our clients, so we help them build the capacity to use the lens and ultimately use the data to make informed program-related decisions.

Program evaluation can be a valuable source of business intelligence for our clients and we are best-equipped with our experience and expertise to do so.

Click here to read  and let us know what you think.

Tips on Data Visualization: The Next Step on the Journey



I have posted previously about my Transformation in Data Visualisation and without that paradigm shift, I believe I would not be adding as much value to clients as I do now. This post is about the next step on the journey for me. I believe a lot of you are in the same boat, so the message of this post is aimed at helping you along, as we are in this together.

I am a very visual person, from learning style to my photography hobby, to how I see the world. So for years when I was writing reports, I was actually out of my comfort zone. Then when exploring Garr Reynold’s Presentation Zen (http://www.presentationzen.com/), the works of Edward Tufte (http://www.edwardtufte.com/tufte/), participating in Stephanie Evergreen’s data visualization workshop (http://stephanieevergreen.com), and reading the works of various other data visualisation bloggers, I realized that most people are out of their comfort zone when reading text, and visualisation of key messages, whether they be qualitative or quantitative, enhances the reader’s ability to internalize the information. Let’s face it folks, most people prefer pictures to text!

While many of my clients still request a full-bore (pun-intended) report, I have tried to introduce visuals and graphics wherever I feel the report needs them. I like to insert graphics into section heads to help guide the reader through a long report. I also like to use the rule of thirds or quarters to divide the page into balanced areas for text and graphics. My previous post on Data Visualization shows my very early, yet client-endorsed, attempts to do this.

I am in no way an expert in data visualization; I am like most of you reading this, just someone who sees the need to present information more graphically, and is learning how to do so. So based on the sources above, and those of some other cool openly-available sources like Piktograph (www.piktograph.com) here’s some of the key pointers I consider I make when I want to visualize information:


    • Make sure the sizzle doesn’t out-perform the steak: In the end, the key message(s) is/are always the most important, not the cool pictures.
    • Show more than tell: Text is important in a data visualization but should be used as a supplement.
    • Keep the message(s) simple: Don’t try to accomplish too much in a single data visualization.
    • Tell a story or have a natural flow to the visual: The reader should progress through the visual and understand the story you are trying to tell.
    • The data must be credible and verified: There’s no point in going to all this trouble if the data are not reliable.
    • Know who your audience is: Your design is most compelling to folks who can best relate to the story you are trying to tell.

Above is an example of an infographic a type of data visualization, I was asked to build for the Canadian Evaluation Society. The message or story is a summary of key facts from their most recent National Conference. It is aimed at their membership, so evaluators, and specifically those who are organizing the upcoming conferences in 2016 and 2017.

The infographic flows using a grid system: vertically in four sections and horizontally in three sections. Each vertical section gets at a finer grain of the data and thus finer tune to the message:


    • The first level is an overview, including an “eye-catching” title (insert smirk here);
    • level two summarizes the use and perceived effectiveness of various promotional media;
    • the third level include tidbits about the conference registrants; and
    • level four presents facts about the actual conference activities, culminating in its overall approval rating by attendees.


Hey, this is far from the be all and end all of creating infographics! But, it is an example of how a guy who’s core skills are not in data visualization is able to convey a one-page message from a much larger source document, and do so fairly easily following some simple guidelines.

Why Do We Conduct Job Analyses for Certification Exam Development?

In one word: Validity.

 

When working with my clients from professional accountants to bankers to IT specialists, whenever we are building new certification or updated certification exams, early on in the project execution we conduct a job analysis. Now, our focus is a little different than those in the Human Resources world who conduct job analyses, but our methods are pretty much the same. Our perspective as credentialing program consultants is to build the validity evidence for the scores associated with the given certification exam. In other words, we want to make sure that if examinees are credentialed as a result of a successful score on the exam, then those scores represent that examinees actually demonstrated the requisite knowledge/skills/attitudes that are required to perform a given job in a real-world setting.

 

“So how does that work?” you may be asking. Our job analysis helps us identify the content domains for the exam, and from these content domains, which are weighted to mimic the real world job’s duties and tasks in the form of an exam blueprint, we develop exam questions. That’s it in a very compact nutshell, and it’s a simplification as the intent of this post is not how we do job analyses. What we are attempting to do with our job analysis is build the case for content-related validity evidence.

 

The Job Analysis Slide (above) we use with our clients may help you visualize the flow of a job analysis. We want to ensure that there are strong links from the job to the exam.

 

Why is Validity so important? Validity is the trump card in all exam development, it is the strength of the links between the job and exam content,  and content-related validity is the type of validity evidence most closely tied to exam development. The Standards for Educational and Psychological Testing, our go-to resource in the educational measurement field, says essentially that “validity” refers to the degree to which evidence and theory support the interpretation of test scores, and to validate a proposed or actualised interpretation of test scores is to evaluate the claims based on the scores. For example, does a successful score on my client’s banking certificate exam mean the test-taker has demonstrated they have met the standards set by the bank to engage in the job/role of “x” in a real-world setting? If the exam was not built from a job analysis, then the exam scores would not really mean the successful examinee could do the job in the real world – the exam score would lack validity evidence to support that score inference.

 

The validity argument built to support the intended use of the exam scores is crucial to mitigating risk associated with the awarding of the associated credential. The organization awarding the credential, whether it be a technical certification or a completion certificate or a professional license must stand behind the meaning and intent of the credential. The only way in which they are able to confidently stand behind the credential is to have assurance that the exam content was an appropriate and representative sample of the types of things the successful examinee will have to do in the real world workplace. That assurance is based on a thorough job analysis at the front-end of the project. It is the job analysis that is the foundation of the content-related validity evidence for the inferences drawn from the exam scores.

 

Of course, content-related validity evidence stemming from a job analysis works really well for exams that test specific skills, where it is relatively easy to define content domains and draw representative samples from those domains to include on the exam. For more broadly defined areas of performance, content-related validity evidence is more difficult to gather, but still essential for any intended exam score inference.

 

Very simply, we want to create strong links from the job’s duties and tasks to the exam content, to contribute  validity evidence for a credentialing exam’s score-based inferences. This is why we conduct job analyses with our clients.

 

 

There are volumes written on validity and content-related validity, but for those who want a relatively quick and solid peek at some foundational literature on the subject, please refer to:

Kane, M.T. (2006a). Content-related validity evidence. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 131-154). Mahwah, NJ: Lawrence Erlbaum Associates.

Kane, M.T. (2006b). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: Praeger.

Raymond, Mark, & Neustel, Sandra. (2006). Determining the content of credentialing examinations. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 181-224). New York, NY: Routledge.

 

 

Birth of a Certification Exam

As a consultant, I work on various projects and when the work is completed, I move on to the next ones, often not being able to see the fruits of my labour, such as improved programs, new program growth and the like. But recently I had the tremendous pleasure of seeing the culminating product of a project I was fortunate enough to be a contributing part of.

My consulting business started out in 2008, and one of my first clients were the Society of Certified Management Accountants of Canada (CMA Canada). My team and I conducted some work on their national certification program’s assessments. Over the years, I worked on several smaller projects for them, all relating to their national assessments. In 2011, my work with CMA Canada was instrumental in landing a contract with CICA (the Canadian Institute for Chartered Accountants). All the while, I was hearing talk of “unification” between Canada’s three governing bodies for accountants: CICA, CMA Canada and the Certified General Accountants (CGA Canada).

In the early part of 2012, I was asked by CMA Canada and CICA to sit on the Assessment Development Task Force for a new, unified accounting body in Canada: the Chartered Professional Accountants of Canada (CPA Canada), which would replace the legacy bodies of CMA Canada, CICA and CGA Canada. I met a lot of smart and dedicated people with the common goal to create a leading-edge, robust assessment program to accompany a new education program, the Professional Education Program (PEP), for Canada’s new and unified accounting profession. After about a year and several working meetings, our task force’s ideas and structure for the assessment program started to become a reality. We developed a six exam system, culminating in the Common Final Examination or CFE. The CFE would, in the Task Force’s vision, culminate a rigorous, cutting-edge, multi-modal professional certification assessment program. To many professional accounting candidates, the CFE would replace the former UFE, or Uniform Final Examination.

The first exams in this new system, the Core 1 Exam went live in the Fall of 2013. Over the next year and a bit, I consulted for CPA Canada on documenting process and performance validity evidence for the new exams.  I also was honoured to be invited to conduct workshops on various psychometric topics, such as Item Analysis and Standard-Setting for CPA Canada’s new Board of Examiners (BOE) at their inaugural and second annual summer meetings.

Additionally, I co-presented our assessment and validation work on the assessment program at the Annual Exchange of the Institute for Credentialing Excellence (ICE) in 2014 with one of CPA Canada’s lead Assessment Principals, whom I worked closely with on  exam validation. Our work was lauded and validated by the professional measurement community there. It was an incredible feeling!

Over the years, CPA Canada has taken on more of the psychometric analyses of the PEP assessments, and my role has decreased, which to me is a sign that I did my job well, helping the assessment staff take their measurement expertise to another level.

Then this past Fall, I had another feeling of tremendous satisfaction, as it was like the CPA Canada Assessment Development Task Force’s baby was born! The first ever Common Final Examination was written by the first cohort of new Chartered Professional Accountants. There is an article on the inaugural CFE in CPA Canada’s membership magazine that can be found here.

As noted, I am somewhat distanced from the CPA Canada PEP assessments now, but have a tremendous sense that my work as part of the dedicated Task Force and with the exceptionally talented Assessment staff at CPA Canada over the years helped create something progressive, and more importantly for a measurement expert, something valid, ushering in Canada’s first wave of new Chartered Professional Accountants.

I thank CMA Canada, CICA and CPA Canada for the opportunity to be part of something so ground-breaking.

P.S. my work with financial professionals does not end with CPA Canada. I am currently working on projects for the Royal Bank of Canada and the Canadian Association of Insolvency and Restructuring Professionals.

 

 

Putting “Presenting Data Effectively” to Work: My Transformation in Data Visualisation

As a trained educator, I certainly see the intrinsic value in continuous learning. As an entrepreneur and practitioner in an evolving field, continuous learning through professional development (PD) is not only essential to survival; it offers opportunity for competitive advantage. I regularly participate in PD courses at industry conferences, looking to stay sharp and get on the leading edge.

read more

Do you Know the Difference between Accreditation and Certification?

There is a profound difference between “accreditation” and “certification” and those who work in the professions or training and development, especially, should be aware of this. I have seen more than a few times that organizations are offering “accreditation” to members or individuals who take their courses. For example, a professional association I belong to, as a resource, allows other organizations to post advertisements for training on their web site that may be of interest to our membership.

read more

Error in Testing (and Sports)

With the spring weather upon us and the start of baseball season, hockey season gearing up for playoffs and Champions League soccer I have seen the element of human error of the officials come into play, sometimes leading to outcome-deciding moments. In the testing industry or educational and psychological measurement, we understand that no test is perfect and the element of measurement error comes into play ALWAYS!

read more

Managing Evaluations in the Corporate Sector: Using a Multi-Step Method

Please refer to the AEA365 Blogspot for my short piece on managing evaluations and evaluation capacity building in the corporate sector.

http://aea365.org/blog/?p=7047

Also the slide deck for the presentation at AEA 2012 can be seen on this site:

http://www.rae-consult.com/wp-content/uploads/2012/10/AEA-2012-Presentation-Ali.pdf