OnPolicing Blog

Reliable, valid measurement using sound psychometric principles 

Creating, conducting, and interpreting surveys is more complicated than it may seem. Law enforcement agencies across the country are joining us in recognizing the importance of data-driven decision-making in improving public safety outcomes, reducing crime, and increasing community satisfaction with police services. But where do you get that data from? And how do you know you’re reading your results correctly? 

These questions highlight the importance of measurement. Measurement is a key component in determining what works, yet finding and tracking accurate measurements is an often overlooked or misunderstood part of the process. 

For instance, perhaps your agency facilitated a survey of residents that sought information about their support for community policing. After reviewing the survey results, you concluded that community policing is a function residents are interested in and should be the focus of your crime prevention efforts. 

Did your survey offer the right conclusion?  

What Are Psychometrics? 

Psychometrics examines the reliability, validity, and accuracy of measurement, specifically with regard to attitudes, behaviors, beliefs, perceptions, and other social phenomena. The conclusion you might draw from the community survey mentioned above is a central concern of psychometrics. In policing, psychometric tools can help evaluate officer suitability, mental health, training effectiveness, perceptions of organizational culture, wellness, and community sentiment, among others. 

Often when conducting a survey, the element of interest (i.e., what you’re hoping to learn from the study) is considered a latent variable: something that can’t be measured directly. For example, job satisfaction, community trust, or supervisory support: these are all psychological “constructs,” in that they’re comprised of various attitudes, predispositions, values, etc. They aren’t just one single measurable quality. Just as community policing efforts require many different components to be successful, these concepts are shaped by a combination of attitudes, values, and behaviors. 

Without considering the essential characteristics of measurement (reliability, validity, and accuracy), one could draw completely inaccurate conclusions or misinterpret findings, and this could lead you to implement policies and practices that fail or do more harm than good. 

Use of Police Department Subject Matter Experts May Not Guarantee Good Measurement 

Surveys are a typical form of measurement designed to assess these phenomena. However, there’s an art to crafting the right survey questions to ensure you’re learning what you intended to. 

Many social scientists, including those often commissioned to facilitate surveys, focus heavily on their overall ideas and theories, as well as the agency’s goals and interests. However, they often fail to address the accuracy of the instruments they’re using to gather information for evidence-based decision-making. For example, they may create an ad-hoc, or ‘on-the-fly,’ instrument that appears to measure what it’s supposed to, but this is a very poor and unscientific approach. If the survey or instrument is flawed, the interpretation and conclusions drawn from the responses will be flawed as well. 

Scientific journals are full of peer-reviewed articles on the effectiveness of policing tactics, strategies, and programs even though the authors may have provided little, if any, information about how their surveys or questionnaires were developed, how they know they’re measuring what they’re intended to measure (validity), and/or whether they consistently measure the phenomenon over time or across individuals (reliability).  

Alternatively, internal experts may create surveys or modify those developed by outside experts without considering the principles of psychometrics or sound survey development practices. At first glance, survey development may seem like an effort to gather individual questions and answers. But failure to consider the true meaning of each question, the responses provided, or what the responses indicate can lead to misinterpretation. So developing ad-hoc, on-the-spot questions or items is not ideal. Quality survey development is often an iterative process that requires thoughtful development, implementation, analysis, and refinement through multiple repeated steps, with a focus on psychometrics. 

Consider the example above regarding a survey of community policing in your jurisdiction. Ask yourself these questions: 

• How do we define community policing? Is it a program? A strategy? An approach? A philosophy? Or something else? 
• Is that definition universal, i.e., is it a generally accepted definition in the field?  
• Because community policing is a multi-faceted phenomenon, are we addressing the various components of community policing? If so, how do we ensure that we address all the elements and that each component is unique from the others?  
• Perhaps most importantly, how do we know that the individuals we’re surveying share the same understanding of community policing?  

Simply asking those surveyed if they support community policing isn’t enough. It’s similar to asking different questions of different people—you won’t get an accurate picture unless you clearly define and operationalize community policing. 

You could provide the surveyed population with the definition as you intend it, or clearly articulate questions that represent the phenomenon. For example, questions that tap into each component will ensure that those items accurately and adequately represent each component. To do the latter, you may need to gather information on whether the items or questions demonstrate reliability and validity. This often takes time, as well as field testing through piloting or gathering data across multiple studies. This is why using surveys or scales that have already demonstrated these strong psychometric properties, collected over numerous studies or over many years, are preferred to a home-grown survey in which you start from scratch. 

Survey Development in Policing 

Developing effective surveys is crucial to capturing reliable and actionable data. Surveys can be targeted at officers, staff, or community members, and must be carefully designed to ensure clarity, avoid common pitfalls, and provide sound psychometric characteristics.  

Common Pitfalls in Survey Development  

1. Double-Barreled Questions 

These are questions that contain more than one question in a single item, e.g., “Do you consider the local police officers in your community to be fair and trustworthy?”  

Fairness is distinct from trustworthiness, and if someone says yes, they may be saying yes to one, the other, or both. Similarly, if they say no, they may mean yes for one, but not for both, or they may answer in the middle because they have a high score for the former and a low score for the latter, leading to an inability to interpret the findings. Yet these types of questions are commonplace.  

2. Response Scale Directionality 

This is when you change the direction of response scales across questions.  

Many survey items are designed to establish the level of agreement, frequency, etc., (e.g., strongly disagree to strongly agree, or never to always (known as Likert scales)). The scale should read the same across the survey or at least within the subscale (a set of related questions within the survey). In the first question, the response scale may be a 5-point1, or similar, ranging from strongly disagree (1) to strongly agree (5). However, in the second question, it may range from strongly agree (1) to strongly disagree (5). This is likely to lead the reader to rely on the first scale throughout and therefore answer the opposite of what they intended.  

3. Biased or Leading Questions 

This is when the tone or directionality of the item may influence bias in the response, such as asking a leading question like “don’t you agree that…?” 

Questions like this can make the survey respondents feel they need to agree, which skews the response in the direction of agreement instead of using a neutral tone. Using “to what extent do you agree or disagree that…?” is a more suitable alternative. Another example of a biased tone could be asking a question using “juvenile delinquents” instead of a version that avoids personal judgment, like “youthful offender.” 

4. Insufficient Response Options  

This occurs when you fail to provide all possible responses, which can leave the reader feeling offended or lead them to believe the survey doesn’t apply to them, ultimately causing the respondent to discontinue participation. 

For example, the question to officers about rank may include the following responses: a) patrol officer, b) sergeant, c) lieutenant, or d) chief. However, in the jurisdiction, there may also be a rank of captain. The solution is to include all ranks (by asking the agency in advance) or offer an “other” option. Such oversights can lead to a biased sample or attrition in the survey for some ranks. 

5. Assumptions About Experience or Knowledge 

This is when you assume the respondent experienced something (an experiential assumption) or has certain knowledge (a knowledge assumption). 

For instance, an experiential assumption may be “when you have been injured on the job.” Some officers may not have been injured on the job; instead, for this type of question, it’s important to include a response option of “not applicable” or “I have not been injured on the job.” 

A knowledge assumption may be an agreement scale for the statement, “Predictive policing is good for our agency.” If the agency hasn’t provided training to all officers on predictive policing or defined its meaning in the survey, it’s possible that some respondents may not know what it means, but will select an answer anyway to avoid “sounding stupid” by not answering. The same is true for overly complex or wordy questions, where the question may get lost in the text. 

6. Negatively Worded or Double-Negative Questions 

While not always problematic, questions asked using an agreement scale (“strongly disagree”) can be confusing when the question is worded negatively.  

For example, the item “I don’t dislike my supervisor” is a double negative and may be exceptionally confusing if the response scale also includes a negative option, such as “strongly disagree” or “not typical of me.” Instead, simply word the question as “I like my supervisor” or “I dislike my supervisor” (which is “reverse coded’). The use of some reverse coded items is a good check on whether an individual is paying attention, rather than just answering the same option all the way down a column, e.g., always answering “somewhat agree” or selecting the midpoint on an odd number of responses (a practice known as central tendency error designed to hide one’s actual feelings from the surveyor). This doesn’t mean that a neutral response shouldn’t be used, however, as it may accurately reflect the respondent’s views, beliefs, or attitude. 

What Are the Best Practices for Community Policing Surveys? 

To maximize the value of your survey, ensure it reflects the following principles. 

Clarity: Use simple, concise, and direct language when drafting questions. Avoid technical jargon or ambiguous phrasing that may confuse participants. Clear questions encourage honest and accurate responses, facilitating meaningful analysis of community sentiments. 

Cultural Sensitivity: Design your survey with inclusivity in mind to resonate with diverse demographics. For instance, questions should reflect the unique concerns of African American communities and address specific cultural nuances. This approach ensures every group feels heard and respected, fostering trust and strengthening community engagement. 

Relevance: Tailor your survey to address the key concerns of your community, such as property crime, burglaries, domestic violence, or police misconduct. Additionally, include questions about broader issues, such as quality of life, community trust, and perceptions of police activities. Relevance ensures that results can be directly applied to inform policy decisions and community outreach efforts. 

Pilot Testing: Testing your survey with a small focus group allows you to identify potential issues, such as unclear questions or cultural insensitivity. These trial runs provide valuable feedback, improving the overall survey design before deployment. Pilot testing also helps refine the methodology to align with community needs and goals. 

Psychometric Integrity: Pay close attention to reliability and validity to ensure your survey accurately captures community attitudes and sentiments. Incorporate questions that measure consistent outcomes across diverse participant groups. This standard helps build credibility for the survey results while creating actionable insights for the Office of Community Oriented Policing Services and other stakeholders. 

Community Engagement: Use community meetings, outreach initiatives, and partnerships with local community groups to raise awareness about the survey and its goals. This can increase participation rates and build trust in the process. Active engagement demonstrates a commitment to listening and collaborating with the broader community. 

Conclusion 

Collecting accurate data from surveys isn’t as simple as it may seem. Relying on tested strategies and data collection methods, effective survey development approaches, and measures with strong psychometric properties (as previously demonstrated) ensures the reliability and validity of survey data, allowing for credible and informed decision-making. By balancing these methods and approaches, agencies can enhance police performance and well-being, better understand community concerns, and foster greater trust and accountability. 

The application of psychometric practices and thoughtful, effective survey design principles is indispensable for meaningful data interpretation and informed decision-making in law enforcement and scholarly work in criminal justice. 

Do you need help with surveying? NPI offers tailored assistance across a variety of services, including surveying. Please let us know how we can assist you.  

FAQs 

What Are Some Good Examples of Community Police Survey Questions? 

1. How satisfied are you with the police presence in your neighborhood? 
2. Do you feel comfortable reporting a crime or concern to law enforcement? 
3. How would you rate the response rates during non-emergency calls? 
4. Do you feel the police adequately address concerns regarding violent crime or property crime? 

How Effective Are Community Policing Surveys in Improving Police-Community Relations? 

Effectively designed surveys allow law enforcement agencies to identify areas needing improvement, strengthen community trust, and address existing concerns systematically. They also open the door to new initiatives tailored to community feedback. 

How Do Community Policing Surveys Impact the Relationship Between Law Enforcement and Residents? 

Surveys offer a platform for honest communication. By involving residents and acting on survey findings, police demonstrate accountability, fostering trust and legitimacy in their initiatives.