Pilot Trials and Validation Statistics: Proving a Process Will Meet Its Permit Before It Is Built

Start from how compliance is judged

UK discharge permits for sewage works set numeric limits in two forms. Look-up table limits (the Urban Waste Water Treatment regime for BOD, COD and suspended solids) allow a permitted number of failures in a year that rises with the number of samples: for 4 to 7 samples one failure is allowed, for 8 to 16 two, for 17 to 28 three, for 29 to 40 four, and so on, with a separate upper-tier concentration that no sample may exceed. Percentile limits express the limit as a 95th percentile, so the test is whether more than 5 percent of samples over the period exceed it, with the same look-up logic applied to the sample count. Limits on ammonia and phosphorus under the environmental permitting regime are set this way, and nutrient permits increasingly add an annual mean limit, which is a different statistic again and is often the tighter one.

So the pilot must demonstrate the statistic the permit uses. A trial that reports an average effluent of 0.3 mg/L phosphorus against a 0.5 mg/L 95th-percentile limit has proved nothing unless the spread of results is also known, and a trial that reports a 95th percentile has proved nothing about an annual mean limit unless the distribution is characterised.

Effluent quality is log-normal: what that means for the numbers

Concentrations in treated effluent are bounded below by zero and have an upward tail, and in practice fit a log-normal distribution well. If the mean m and coefficient of variation CV are known, the parameters of the underlying normal distribution of ln(C) are:

Worked example. A pilot gives a mean BOD of 8 mg/L with CV = 0.5. Then σ² = ln(1.25) = 0.223, σ = 0.472, μ = ln 8 − 0.112 = 1.968, and C₉₅ = exp(1.968 + 0.777) = exp(2.745) = 15.6 mg/L. Against a 20 mg/L 95th-percentile limit the process complies with a margin; against a 15 mg/L limit it does not, despite a mean of only 8. Halving the variability to CV = 0.25 brings C₉₅ to 11.9 mg/L: for percentile limits, consistency is worth as much as performance, which is why equalisation and control are so often the real compliance measures. The ratio C₉₅/mean, about 1.9 at CV 0.5 and 1.5 at CV 0.25, is a useful rule for converting between a mean-based design target and a percentile permit.

At full scale the distribution will be wider than the pilot's, because the pilot ran under closer supervision with a steadier feed; a design margin on CV, not just on the mean, is prudent.

How many samples: tolerance intervals and confidence

A pilot that takes 12 samples and finds none above the limit has not shown that 95 percent of future samples will comply. For a claim about a proportion of the population, rather than about the mean, the tool is the statistical tolerance interval: an interval that contains a stated proportion p of the population with a stated confidence. For normally distributed data (which ln(C) is), the one-sided upper tolerance limit is:

Taking the example above with n = 12 pilot samples: the sample statistics in log space are ȳ = 1.968 and s = 0.472, and k₁ for n = 12 is about 2.7, so U = 1.968 + 2.7 × 0.472 = 3.24 and exp(U) = 26 mg/L. With only twelve samples the data can only support a claim, at 95 percent confidence, that 95 percent of results will be below 26 mg/L, well above the point estimate of 15.6. With 40 samples the same statistics give exp(1.968 + 2.1 × 0.472) = 19.3 mg/L, and with 100, 17.5. Sample number is not a formality; it determines what the trial can claim. A trial intended to demonstrate compliance with a tight percentile limit needs of the order of 40 to 60 samples over the range of operating conditions, and the regulator's own sampling guidance for operator self-monitoring applies the same logic in the other direction, setting the frequency needed to assess compliance at a given confidence.

Distribution-free tolerance intervals exist for data that will not fit a log-normal model, at the cost of larger sample numbers: to be 95 percent confident that 95 percent of the population lies below the largest observed value requires about 59 samples, and below the second largest about 93.

Designing the trial so the numbers transfer to full scale

1. Represent the duty, not the average. Run through the seasonal temperature minimum, the peak load, the wet-weather dilution and any shock the full-scale plant will see. A trial over a mild summer proves the summer.
2. Match the controlling dimensionless groups. For a biological reactor, sludge age and food-to-mass ratio; for a filter or contactor, hydraulic loading and empty-bed contact time; for a clarifier, surface overflow rate and, as far as the geometry allows, the Reynolds and Froude regimes. A pilot clarifier at full-scale depth but reduced area is right; one at reduced depth is not.
3. Scale what cannot be matched honestly. Wall effects in small columns, inlet energy dissipation and short-circuiting do not scale; a tracer test on the pilot and on the full-scale design (or its CFD model) shows whether the residence time distributions correspond.
4. Log everything continuously, not just the grab samples the statistics will use: flow, temperature, dose, pressure, dissolved oxygen, turbidity. A continuous record explains the outliers and is what a reviewer asks for first.
5. Pre-register the analysis. State in the trial plan which statistic will be reported, how many samples will be taken, which results may be excluded and why (instrument failure, documented upset) and the acceptance criterion. Deciding after the data are in is the fastest way to lose a reviewer's confidence.
6. Use accredited analysis for the determinands in the permit and report the analytical uncertainty, which at low phosphorus or ammonia concentrations is a material part of the apparent variability.

The special case: validating UV disinfection

Ultraviolet disinfection cannot be verified by sampling the effluent for the target organism in the way a chemical process can, because the pathogens of concern are rare and slow to assay. The accepted approach, codified in the US EPA UV Disinfection Guidance Manual and adopted in UK drinking water and reuse practice, is biodosimetry: a surrogate organism of known UV sensitivity (MS2 bacteriophage or Bacillus spores) is dosed into the feed of the full-scale or identical reactor across the range of flow, UV transmittance and lamp power, and the log reduction measured. A collimated-beam test on the same organism gives the dose-response curve, so each reactor result can be converted to a reduction equivalent dose (RED), the dose that would have produced that inactivation under ideal exposure.

The validated dose is the RED divided by a validation factor that accounts for the difference in UV sensitivity between the surrogate and the target pathogen, the uncertainty of the test, and the position of the lamps relative to the sensors. Each reactor then has a validated operating envelope of flow, transmittance and intensity within which it delivers at least the required dose (for example 40 mJ/cm² for a 4-log virus credit in drinking water practice, lower for Cryptosporidium, higher for adenovirus), and the control system holds it inside that envelope. A UV reactor with no validation certificate for the duty, or one operated outside its validated transmittance range, is unvalidated whatever its lamp power. Design detail is in the UV disinfection guide; the equivalent for reuse schemes, where log-reduction credits are assigned per barrier, is in the water reuse guide.

Reporting

A trial report should give, for each determinand in the permit: the number of samples, the mean, the CV, the fitted distribution with a goodness-of-fit check, the point estimate of the permit statistic, the tolerance limit at 95 percent confidence, and the operating conditions under which the data were gathered. It should separate the pilot's steady operation from documented upsets and report both. Where the trial supports a permit application or a variation, the compliance submission is only as strong as that statistical chain, and reviewers, including the Environment Agency's technical assessors, will check it. Wider permit context is in the effluent standards guide.

How to plan a pilot trial that will satisfy a regulator

1. Write down the permit statistic. Look-up table, 95th percentile, annual mean or upper tier; the trial must report that statistic, not just an average.
2. Choose the sample number from the claim. Use the tolerance-interval factor for the confidence and proportion required; 40 to 60 samples for a tight percentile limit.
3. Cover the duty range. Winter temperature, peak load, wet-weather dilution and shock loads; log operating conditions continuously.
4. Match the controlling groups. Sludge age, hydraulic loading, contact time, overflow rate and depth as at full scale; tracer-test the pilot.
5. Pre-register the analysis. Statistic, sample count, exclusion rules and acceptance criterion fixed in the trial plan before sampling starts.
6. Report with uncertainty. Distribution fit, point estimate, tolerance limit and analytical uncertainty for every permit determinand.