How to Measure the Cost of Not Knowing

The cost and the risk of discovering how your own work runs can be measured — directly and indirectly — and that measurability is both the proof of the idea and its value

The first four articles described a problem and a trap: information about how the work runs is missing, much of it can only be declared rather than found, those declarations are costly and arrive in a paradoxical order, and rebuilding to escape the whole thing is a blind bet. A fair objection at this point is that all of this is an argument, not a measurement — interesting, perhaps, but untestable.

This article answers that objection. The cost and the risk of acting on a disordered process can be measured. That they can be is the point: it is what turns the preceding claims from a story into something the world can confirm or refute.

The disturbance is the instrument

Start with the obstacle, because the way around it is the whole idea. The second article showed that you cannot inspect a disordered estate without changing it — asking who owns something forces a declaration, and the declaration alters the thing. That seems to make measurement impossible.

It does the opposite. If you cannot observe without disturbing, then measure the disturbance and what it costs. You do not have to complete the discovery — which the third and fourth articles say you can’t — to measure its price. You read the price off the partial gathering, because the gathering is already paying the cost and already touching the value. The very thing that made discovery paradoxical is what makes its cost observable.

What can be measured directly

The cost of discovery. This is plain bookkeeping: the hours, the elapsed time, and the spend it takes to surface the systems you did not know you had. An organization that believed it ran two hundred applications and finds six hundred has a number attached to finding the extra four hundred. Track it as you go, and if each further system is harder to find than the last, that rising curve is itself a reading of how deep the not-knowing runs.

The mix of what you find. As you gather, sort each fact into two bins. Some are retrievable — they exist, with an owner and a record, merely undocumented. Others are not facts yet at all — no owner, nothing to retrieve, a gap that can only be closed by declaring something into being. The share that falls in the second bin is a direct index of both cost and risk: the more of your estate that has to be decided rather than looked up, the more expensive and the more dangerous acting on it will be. Weighed against how interconnected those parts are — how far each declaration would ripple — it gives a usable estimate of what clarity will cost and what it puts at risk.

What can be measured indirectly

The value hidden in the unmapped parts cannot be counted exhaustively, but it can be estimated, by two moves that are easy to confuse.

Sample the value of what you just found. Take the newly discovered systems, sample them, and assess on the value chain whether the sampled ones carry value. That estimates the value of the whole newly-found set without examining every item — ordinary sampling, valid as long as the sample is representative.

Estimate what you still haven’t found. “We asked two hundred owners and eighty replied, so there may be more out there” is not a complaint about survey response — it is a recognised estimation method. Counting how much two independent sweeps overlap lets you estimate how much remains unseen; ecologists size unseen animal populations this way, and software teams use the same trick to estimate how many defects an inspection missed. The size of the still-undiscovered estate is estimable from the discovery rate alone.

Assumptions and sensitivity. The two sweeps must refer to a population that is effectively closed during observation, match the same items reliably, and have sufficiently homogeneous capture probabilities within each stratum. Positive dependence between sweeps tends to understate the unseen estate; negative dependence can overstate it. Report the overlap, the interval and a zero-overlap warning, then repeat the estimate by relevant strata and under plausible dependence scenarios rather than presenting a single total as ground truth.

def estimate_unseen_population(sweep1_found: set, sweep2_found: set) -> dict:
    """
    Capture-recapture (Chapman) estimator — a small-sample correction to the
    two-sweep method. Use only when the estate is closed during both sweeps,
    identities can be matched, and capture probabilities are sufficiently
    homogeneous within each stratum.
    """
    overlap = sweep1_found & sweep2_found
    n1, n2, m = len(sweep1_found), len(sweep2_found), len(overlap)
    if n1 == 0 or n2 == 0:
        raise ValueError("Both sweeps must contain observations")
    estimated_total = ((n1 + 1) * (n2 + 1) / (m + 1)) - 1
    variance = ((n1 + 1) * (n2 + 1) * (n1 - m) * (n2 - m)) / ((m + 1) ** 2 * (m + 2))
    found_so_far = len(sweep1_found | sweep2_found)
    standard_error = variance ** 0.5
    return {
        "found_so_far": found_so_far,
        "estimated_total": round(estimated_total),
        "estimated_still_unseen": round(max(estimated_total - found_so_far, 0)),
        "approx_95pct_interval": (
            round(max(found_so_far, estimated_total - 1.96 * standard_error)),
            round(estimated_total + 1.96 * standard_error),
        ),
        "zero_overlap_warning": m == 0,
    }

# Illustrative: two independent inventory sweeps of the same estate
# (not measured data — a worked example of the method).
sweep_a = {f"sys_{i}" for i in range(1, 121)}    # 120 systems found in sweep A
sweep_b = {f"sys_{i}" for i in range(60, 200)}   # 140 systems found in sweep B

print(estimate_unseen_population(sweep_a, sweep_b))
# -> {'found_so_far': 199, 'estimated_total': 275, 'estimated_still_unseen': 76}

Keep these apart: you sample the systems you have found; you estimate the ones you haven’t. Do not stretch the sample across that line, because the systems you cannot find may differ systematically from the ones you can.

The measure the organization gives you for free

There is one more reading, and it is the strongest kind — the sort an organization supplies through its own behaviour rather than through anyone’s opinion. Faced with systems no one understands, organizations do not simply delete them. That refusal can be treated as evidence that the organisation perceives a non-zero cost or risk in removing the unknown; it does not by itself identify or quantify that cost. It does not prove the hidden value is real — caution and the sheer effort of deciding to delete both play a part — but it shows, in revealed behaviour, that the cost-and-risk being measured is judged to be greater than zero. And it is falsifiable in the cleanest way: if organizations freely deleted their undocumented systems and nothing went wrong, the cost and the risk would be near zero, and this whole line of argument would collapse. They don’t, which is the point.

Why this matters

This is what makes the series more than a thesis. The cost is countable, the value at risk is sampleable, the size of the unknown is estimable, and the organization’s own conduct supplies a behavioural reading of the risk — none of it requiring that the argument be taken on trust. The remaining bet, and it is a real one to be tested rather than assumed, is that these measures predict where projects fail: that the estates with the most to declare and the most entanglement are exactly the ones where rationalization, automation, and discovery stall. That is a prediction, and it can be checked.

And none of it is an accusation. It measures a structural property of an estate, not anyone’s competence — which is what lets a practitioner read the number without flinching.

Methods drawn on: statistical sampling and overlap-based (“capture–recapture”) estimation, used in ecology and in software residual-defect estimation; and revealed-preference reasoning from the organization’s own behaviour.