Getting Started with Manual Review

Whether you’re assessing orders before fulfillment or reviewing orders post-acceptance/rejection decision, follow these best practices when performing manual review of transactions.

Why review transactions at all?

Before getting started on specifics, let’s ask: What’s the point of manual review? If you have fraud tools in place, why have a human being look at transactions at all?

Even the best-designed system will occasionally flag legitimate orders as suspicious, and overlook the occasional fraudulent order. The only way to assess the shortcomings of a system is to review its predictions. This could be done by judging the system only by the misses that lead to chargebacks, but this approach would completely ignore legitimate orders rejected in error.

Even the best-designed system will occasionally flag legitimate orders as suspicious, and overlook the occasional fraudulent order.

Additionally, for transactions that are neither extremely safe nor extremely risky, gathering more information before making a decision is a path you may consider. Companies are using SMS and dedicated landing pages in order to provide a second layer of authentication before processing orders. This provides an automated and scalable way to challenge transactions you may not have the most confidence in, with the added benefit of providing additional information for chargeback representment.

There are certainly businesses for which delaying orders from fulfillment doesn’t make sense, especially when average order value is particularly low or in the case of digital goods when instant delivery is expected. Regardless of whether it happens pre- or post-hoc, businesses will always find value in reviewing transactions and tailoring their fraud practices to fraudsters’ evolving tactics.

Factors to consider

Whether to put orders on hold: Deciding whether to put some orders in hold until they can be reviewed depends heavily on the nature of your business. If orders are fulfilled only once a day or less (for example, weekday afternoons when you ship physical goods), holding risky orders for review makes a lot of sense. If, on the other hand, instant fulfillment is inherently necessary (as with rideshare trips, for example), or if nothing is lost by issuing a refund once you learn a transaction is fraudulent (as with access to an online game, for example), holding orders for review might not be the best solution.

Hours of operation: Rest assured that if you put any orders in a held state, your customers will contact you about them. As such you should prepare in advance for any inquiries and how you plan to resolve them.

Handling logistics: Will you handle these requests by email, phone, or both? What are your internal service level agreements (SLAs) for response times? Knowing the answers to both of these questions can help you adjust your thresholds for held orders accordingly.

Role of reviewer judgment: What overall standards do you expect reviewers to employ when deciding what to do with orders? Generally, you should rely on the judgment of the people making decisions, rather than giving them very strict “cancel the order when this, release the order when that” rules.

Rules, after all, are easily automated, and you should generally avoid using people for anything that could be automated. Instead, provide clear guidelines for what to look out for in general, and which way to lean in certain situations (e.g., if it’s under $X, don’t bother calling to ask about it). Additionally, providing direction on what data to assess for review from your case management tool can improve consistency.

Building a manageable review queue

Not every order belongs in front of a person. Send only the transactions your automated decisions cannot settle with confidence, such as orders in the middle of your risk range, first orders from new customers, or unusually high order values. Everything clearly safe or clearly risky should be accepted or blocked automatically.

Volume matters as much as routing. A queue that grows faster than your team can clear it becomes a hold on every order inside it, which frustrates legitimate customers and slows reviewers down. Prioritize by order value and age, and adjust your thresholds when a backlog builds. If you hold orders, set a target decision time that matches your fulfillment schedule so nothing sits past the next shipping window.

What reviewers should look at

Reviewers move faster and more consistently when they know which signals to check first. Useful starting points include:

• Whether billing and shipping addresses match, and how far apart they are
• The age and pattern of the email address, such as a brand-new address or a string of random characters
• Whether the device and IP location fit the shipping destination
• How many orders, accounts, or cards share the same device, address, or payment method (see velocity detection)
• What is in the cart, since gift cards and easily resold items attract more fraud
• The customer’s history with your business, including past orders, returns, and disputes

No single signal settles a case. Look for the combination, and document your reasoning in the case notes. Those notes help the next reviewer stay consistent and give you evidence if you need to dispute a chargeback later.

Measuring whether manual review is working

Manual review should get cheaper and rarer over time, not busier. A few numbers show whether it is working:

• Review rate: The share of orders sent to a person. A rate that keeps climbing usually means your thresholds are too wide.
• Approval rate of reviewed orders: If reviewers approve nearly everything, you are paying people to confirm what automation could have accepted. If they deny nearly everything, those orders may be safe to block automatically.
• Time to decision: Compare it against your SLAs and your fulfillment schedule.
• Chargebacks on approved orders: This shows how many fraudulent orders slip through review.

Feed reviewer decisions back into your fraud system so its predictions improve and fewer orders need a human next time.

Prevent all types of online fraud and abuse with a single fraud detection solution for end-to-end protection.

Frequently asked questions