Search

Risk-Limiting Audits

RiskLimitingAudits.org is maintained by a community of RLA experts. Our goal is to demystify the RLA and connect you to the experts and tools you need to run RLAs in your state.

Introduction

Democracy depends upon trust in the electoral process. In the United States, nearly every jurisdiction relies on machines to tally votes, so trusting the process means trusting the machines that scan and tally our paper ballots. Given the high stakes of our elections, and our adversaries’ demonstrated interest in them, what can we do to double-check the equipment producing the tallies? The best safeguard we have against hacked or otherwise faulty voting systems is a Risk-Limiting Audit (RLA).

History

First, a little background. For over 15 years, election administrators, statisticians, security experts, and the election integrity community have been testing innovative ways to conduct post-election audits. They conclude that RLAs are the most efficient way to gain certainty that the winner reported by the voting system is correct. Some errors are inevitable – paper gets smudged, dust particles may be read as voter marks. RLAs take all of those into account and give us confidence that the claimed outcome is, in fact, the correct outcome.

The RLA is efficient because of its insight that we do not need to manually recount every vote to gain certainty in the outcome of an election. Instead, the number of votes we need to manually count depends on the margin of victory in the contest we’re auditing. A typical state-wide RLA might require checking fewer than 1000 ballots or 40 batches to validate an entire statewide contest’s results.

In 2017, Colorado became the first state to implement rigorous RLAs, using the ballot comparison method and auditing one race in each of its 64 counties. Colorado’s experience helped to create the requirements for software that is used to run RLAs today. RLAs are now a standard part of Colorado’s election process.

In 2018 & 2019, Michigan, Georgia, Pennsylvania, Rhode Island, and Virginia began piloting ballot polling RLAs by working with experts in the field and conducting small scale pilots. Different counties and jurisdictions verified the process worked on every paper ballot tabulation system and was truly implementable in states with tabulation methods different from Colorado’s central count system. These pilots proved to be very successful and each state moved forward with statewide RLA pilots in 2020. In 2021 and 2022, most of these states moved to using batch comparison RLAs statewide instead of ballot polling. Some even layered their RLA with a new or old traditional audit requirement.

California, Colorado, Georgia, Indiana, Kentucky, Maine, Michigan, Nevada, New Jersey, New Mexico, Ohio, Oregon, Pennsylvania, Rhode Island, Texas, Virginia, and Washington have all passed legislation requiring or allowing for RLAs following elections or have administration pilot programs in process. Legislation continues to be introduced throughout the country.

Basic Requirements

The central requirement to running an RLA is that every participating jurisdiction uses paper ballots verified and cast by the voter (or a voter-verified paper audit trail) and organizes those paper ballots so the ballots can be located and identified after the election. If your state’s jurisdictions use hand-marked paper ballots or ballot marking devices that create a paper ballot that the voter reviews before voting, you’re most likely ready to go. If your state instead uses Direct-recording Electronic (DRE) equipment with no paper trail, then you will need to upgrade your state’s equipment before you are able to perform any meaningful audit.

Types of RLA

There are three main flavors of RLAs: ballot comparison audits, ballot polling audits, and batch comparison audits.

Ballot Polling RLAs consist of the retrieval of randomly selected ballots to confirm the outcome of the election. Ballot polling RLAs only require secure storage and accounting of paper ballots. The selected voted ballots are hand tallied, and the resulting manual vote counts are checked against the total vote counts.

Batch Comparison RLAs consist of the retrieval of randomly selected batches of ballots. A batch may consist of all the ballots cast in a precinct, on a specific ballot scanner, or smaller tabulated batches. Those batches are hand tallied and counts are compared to the reported results, batch by batch, to confirm the outcome of the election. Batch comparison requires vote counts by batch.

Ballot comparison RLAs consist of very precise retrieval of randomly selected ballots. Each ballot is compared to the voting system’s cast vote record (CVR) to confirm the outcome of the election. Ballot comparison requires CVRs from the voting system, and that each ballot is stored in the exact order it was scanned. The imprinting of a unique identifier (in no way connected to the voter) is recommended. At this time, ballot comparison audits are most feasible in central-count states like California, Colorado, Nevada, Washington, and Oregon.

In most other states it is simpler to start with a ballot polling or batch comparison audit. Fortunately, the workflow for each type is roughly the same.

Demystifying the RLA

The first questions state officials have about RLAs regard their complexity and cost. So we’ll start there.

Steps to Complete an RLA

In the description below, we’ll assume the state is running the RLA, but a local jurisdiction can run an RLA independently of the state if it so chooses.

  1. The first step is to select which contest(s) to audit. This choice depends on a number of factors, including: the margin of the contest, whether all the ballots for the contest are able to be audited, which jurisdictions are participating in the audit, etc.

  2. Next, select the risk limit for the audit (usually between 5 and 10%). The risk limit is the largest possible chance that the audit will not correct an incorrect election outcome, if one exists. For example, a 5% risk limit means that in at least 95% of cases the RLA will correct the outcome of an election, if the originally reported outcome is wrong. The lower the risk limit, the more ballots need to be manually examined in order to complete the RLA.

  3. Based on the type of RLA and the details of the contest (contest margins, votes cast, etc.), RLA software is used to calculate how many ballots or batches of ballots need to be audited.

  4. Each local jurisdiction participating in the audit provides its ballot manifest. You can think of a ballot manifest like a shipping manifest - a list of all the containers where ballots are stored after the election and the number of ballots in each container. During the RLA, the manifest is used to randomly select ballots or batches for the audit and locate them. For ballot comparison audits, each local jurisdiction must also provide its cast vote records; for batch comparison audits, each local jurisdiction must provide its candidate totals by batch file.

  5. The state enters a random seed into the audit software’s pseudorandom number generator to randomly select a list of ballots for audit. There is often a public ceremony to generate the random seed - and ensure it is actually random. For example, Colorado rolls twenty, ten sided dice in a public ceremony to create the random seed.

  6. Each local jurisdiction retrieves the designated ballots. The local jurisdiction’s audit board manually enters the voter’s selections into the audit software.

  7. The software determines whether the audit is complete (risk measurement falls below the risk limit) or whether additional ballots need to be examined.

  8. The state initiates any further auditing indicated by the software and then finalizes the audit report, which the audit board signs.

Cost

The costs to run a risk-limiting audit are a small percentage of the overall cost to run an election. In other words, an RLA is quite affordable and, compared to older post-election audit methods, provides a greater degree of certainty that the outcome of the election is valid.

A state running an RLA for the first time may wish to employ a consultant experienced in RLA administration to help the state run the RLA and define ongoing support processes. The software to support RLAs is open source and can be self-hosted by experienced technical users. Hosting and maintenance are available from vendors. Additional expenses may include training local election administrators and staff, per diem pay for audit board members (who retrieve ballots, interpret them, and enter data), as well as any additional hardware costs related to conducting a ballot comparison audit.

Getting Started

If you are interested in learning more about conducting an RLA in your state, you should start by speaking with an expert.

Next, you’ll want to create an RLA working group in your state that should include:

  • State officials who will be responsible for running the RLA
  • County or local officials who are responsible for retrieving and inputting information for audited ballots
  • Representatives from the voting system vendor
  • Recognized statistics and auditing experts
  • Any other stakeholders who may add value or provide necessary buy-in

The goal of the advisory group is for members to become local RLA subject matter experts, make recommendations for RLA policy and rules, understand the technology requirements, outline the steps for conducting the audit, create a calendar of key dates, assist with training and pilot audits, and provide input on public communication plans.