Validity, Uniqueness, Novelty

Metrics mainly used for evaluating graph generative models on synthetic data.

This module provides VUN, a class for computing the Valid-Unique-Novel (VUN) metrics, which measure what fraction of generated graphs are:

  • Valid: Satisfy domain-specific constraints
  • Unique: Not isomorphic to other generated graphs
  • Novel: Not isomorphic to training graphs

By passing confidence_level to the constructor, you may also compute Binomial confidence intervals for the proportions.

Example 
from polygraph.datasets import PlanarGraphDataset, SBMGraphDataset
from polygraph.metrics import VUN

train = PlanarGraphDataset("val")
generated = SBMGraphDataset("val")

# Without uncertainty quantification
vun = VUN(train.to_nx(), validity_fn=train.is_valid)
print(vun.compute(generated.to_nx()))           # {'unique': 1.0, 'novel': 1.0, 'unique_novel': 1.0, 'valid': 0.0, 'valid_unique_novel': 0.0, 'valid_novel': 0.0, 'valid_unique': 0.0}

# With uncertainty quantification
vun = VUN(train.to_nx(), validity_fn=train.is_valid, confidence_level=0.95)
print(vun.compute(generated.to_nx()))           # {'unique': ConfidenceInterval(mle=1.0, low=None, high=None), 'novel': ConfidenceInterval(mle=1.0, low=0.8911188393205571, high=1.0), 'unique_novel': ConfidenceInterval(mle=1.0, low=None, high=None), 'valid': ConfidenceInterval(mle=0.0, low=0.0, high=0.10888116067944287), 'valid_unique_novel': ConfidenceInterval(mle=0.0, low=None, high=None), 'valid_novel': ConfidenceInterval(mle=0.0, low=0.0, high=0.10888116067944287), 'valid_unique': ConfidenceInterval(mle=0.0, low=None, high=None

polygraph.metrics.VUN

Bases: GenerationMetric[Graph]

Computes Valid-Unique-Novel metrics for generated graphs.

Measures the fraction of generated graphs that are valid (optional), unique (not isomorphic to other generated graphs), and novel (not isomorphic to training graphs). Also computes confidence intervals for these proportions.

Parameters:
  • train_graphs (Iterable[Graph]) –

    Collection of training graphs to check novelty against

  • validity_fn (Optional[Callable], default: None ) –

    Optional function that takes a graph and returns True if the given graph is valid. If None, only uniqueness and novelty are computed.

  • confidence_level (Optional[float], default: None ) –

    Confidence level for binomial proportion intervals. If None, only the point estimates are returned.

compute(generated_graphs)

Computes VUN metrics for a collection of generated graphs.

Parameters:
  • generated_graphs (Collection[Graph]) –

    Collection of networkx graphs to evaluate
Returns:
  • Union[Dict[str, BinomConfidenceInterval], Dict[str, float]] –

    Dictionary containing metrics. If confidence_level was provided, it contains confidence intervals as tuples (estimate, lower bound, upper bound). Otherwise returns only the point estimates.

Raises: ValueError: If generated_samples is empty