Building and Maintaining a Review Team

As your approach to fraud expands from damage control after the fact to proactive detection and prevention, you may find yourself hiring team members to work primarily on reviewing orders. What qualities make someone good at fraud review, and how can you ensure your team’s performance stays consistent?

The Qualities of a Good Reviewer

A few key skills help the best reviewers quickly and accurately separate good transactions from bad:

Inquisitiveness: The best reviewers tend to be people who want to understand why things work the way they do. This inclination leads to discerning underlying patterns that can tell a helpful story about a user or transaction. For example: digging into a shipping address to discover it’s the workplace of the cardholder, thus eliminating fraud concerns.

Open-mindedness: Because merchants work with only a limited amount of information about orders, underlying biases can lead to inaccurate results in reviews. The best reviewers remain open to the possibility that their assumptions about how to fill in gaps created by missing data were wrong.

Decisiveness: Exploring possibilities and reconsidering assumptions are both important, but at a certain point a reviewer needs to make a decision. Being able to come to a conclusion quickly while taking all the right information into account is largely a product of being confident enough to make a decision and move on without agonizing over it.

Monitoring Team Performance

The value of manual review is equal to money saved (or lost to cost) compared to what the outcome would have been for an automated review of the same transactions. This is a function of the reviewer’s accuracy (contributing to savings) and speed (reducing costs).

Performance metrics to look at:

The two are related, and so a business decision has to be made at some point about how quickly you want your team to go — with the awareness that quicker decisions will necessarily be made with less information. That said, reviewing anywhere between 10 and 20 orders per hour is typical if reviewers are spending some time gathering information before making a decision.

Accuracy can be harder to measure, but one method some merchants use is to audit reviewed transactions by reviewer. To do this, get a list of manually reviewed orders for a given period of time (say, a week), with the order ID, the name of the reviewer and the resolution (usually cancelled or accepted). Depending on how the data is stored, you may have to instead get this information by resolution instead of by order, but the underlying idea is the same.

Take a sample of resolved orders — for instance, 20 cancelled and 20 accepted orders — and check to see if the reviewer used good reasoning in each case. If you do this regularly, and speak with the reviewer about decisions they reached where the reasoning isn't clear, you can help your team achieve greater accuracy over time.

Writing a Desk Manual

One of the most valuable steps you can take to keep your review team’s performance consistent is to assemble a desk manual for your team to go by. Don’t feel like you need to have every possible scenario covered, but focus instead on broad guidelines: what criteria should cause an order to be cancelled, what follow-up steps should be taken whenever an order is reviewed, and so on.

Get feedback from your team on where they feel they could use more guidance, and incorporate this into your desk manual over time. This will help in onboarding new team members and will give everyone on your team a consistent set of guidelines they can turn to when unsure of how to resolve particular cases.