Sample release (Databricks-ready)

This is release r3 of the prototype visitor product inventory for 3 sample LVEPs: Cornwall, Greater Manchester and York & North Yorkshire. The snapshot date is 12 September 2026. It replaces ve-audit-sample-r2-2026-09-12; the change set lists what changed.

It holds 19,846 product records built from open data: OpenStreetMap, Wikidata, the National Heritage List for England and ONS. Hestia Forge made it for the VisitBritain/VisitEngland tender BTA2235 to show how the inventory could be delivered as tables that load into Databricks.

Release id: ve-audit-sample-r3-2026-09-12. The files are Parquet (snappy) and CSV (UTF-8), with a data dictionary, SQL DDL and a manifest with SHA-256 checksums.

Files

The manifest lists the checksums of the data files, the dictionary, the DDL and the Spark log. The checksums do not cover this page or its stylesheet.

Tables

product

One row per product record, with category, role, LVEP and local authority, location, website, opening hours, months open, checks and field-level sources.

product_attribute_provenance

One row per attribute value: the source record that supplied it, the method (tag, property or rule), the rule confidence where there is one (category only; fixed per rule, not calibrated), and the last edit or check date of that record.

product_source_xref

One row per source record merged into a product: source identifier and link, licence, retrieval and edit dates, how it was linked and the match score. This is the crosswalk from source ids to ve_product_id.

product_relationship

Relationships between products: the nearest visitor product of each heritage asset within 1,000 m, and the possible duplicate and same-name pairs kept separate for a reviewer.

product_lineage_event

One row per merge event, review candidate pair, same-name candidate pair, unit group, and id registry event (split, product merge, retirement).

product_change_set

What changed since ve-audit-sample-r2-2026-09-12, keyed on ve_product_id: inserted and retired products, and one row per changed column of an updated product.

lvep_membership

Which ONS local authorities each sample LVEP covers, with mid-year population. The membership is illustrative.

ve_product_id_registry.csv lists every ve_product_id ever published, the release that minted it, whether it is active or retired, and the id that replaced it.

The data dictionary gives each column's type, nullability, meaning, derivation and an example.

How to load in Databricks

-- 1. In Databricks, run ddl.sql (creates ve_audit.sample and the four tables).
-- 2. Create a volume and upload the four .parquet files to it.
CREATE VOLUME IF NOT EXISTS ve_audit.sample.release;
-- (Catalog Explorer: Upload to volume, or the CLI:)
-- databricks fs cp product.parquet dbfs:/Volumes/ve_audit/sample/release/product.parquet
-- 3. Load, parents first.
COPY INTO ve_audit.sample.lvep_membership
  FROM '/Volumes/ve_audit/sample/release/lvep_membership.parquet' FILEFORMAT = PARQUET;
COPY INTO ve_audit.sample.product
  FROM '/Volumes/ve_audit/sample/release/product.parquet' FILEFORMAT = PARQUET;
COPY INTO ve_audit.sample.product_source_xref
  FROM '/Volumes/ve_audit/sample/release/product_source_xref.parquet' FILEFORMAT = PARQUET;
COPY INTO ve_audit.sample.product_lineage_event
  FROM '/Volumes/ve_audit/sample/release/product_lineage_event.parquet' FILEFORMAT = PARQUET;
-- 4. Check against manifest.json.
SELECT count(*) FROM ve_audit.sample.product;  -- 19,846

The CSV files hold the same rows. They join array values with “; ”, so Parquet is easier to load. To read a CSV file in Spark, use header=true, multiLine=true and escape='"'.

Spark read check

We ran pipeline/spark_read_check.py on a local Apache Spark session (local[*]). It reads every Parquet file with spark.read.parquet, prints the schemas, counts rows and compares them with the manifest. It also reads each CSV file with a header, checks keys, and creates the tables from ddl.sql.

Spark version

4.2.0

Java version

21.0.12

Python and pyarrow

3.12.3, pyarrow 25.0.1

Row counts match the manifest (Parquet and CSV)

yes

ve_product_id unique, and every id active in the id registry

yes

Every cross-reference and lineage ve_product_id exists in product

yes

DDL tables created in local Spark and match the Parquet schemas

yes

All checks passed

yes

For the DDL check, local Spark has no Unity Catalog or Delta Lake, so the script substituted USING DELTA with USING PARQUET and ve_audit.sample. with a local database, ve_audit_sample_local. Local Spark rejected the informational PRIMARY KEY / FOREIGN KEY clauses (they need Unity Catalog), so they were removed for this check; their syntax was not validated locally. The CREATE CATALOG statement was not run locally.

Read the full Spark read check log (real output, with versions and timestamps).

Persistent identifiers

Use ve_product_id. It is a surrogate key: it is minted once, when a product is first published, stored in an id registry, and from then on only looked up, never recomputed. Every source record of the product is stored against the id (the product_source_xref table), so the id stays the same when a different source record becomes primary, for example when a BTA or LVEP listing arrives for a product first found in OpenStreetMap.

• Merge: when two products become one, the product keeps the id that holds most of its source records; the other id is retired with replaced_by and a product_merge event.
• Split: when one product becomes two, the part holding most of the old source records keeps the id; the other part gets a new id and a split event.
• Retirement: an id whose source records have all gone is retired, never reused.

In this release 19,846 of 19,846 products kept the id they had in ve-audit-sample-r2-2026-09-12; 0 ids were minted and 0 retired. The ids of release 1 were minted from the SHA-256 hash of the then-primary source record key; that formula is not applied again. The logic is in pipeline/assign_ids.py, with a unit test for a change of primary record, a merge, a split, a new product and a retirement.

prototype_id (for example CW-00001) is the number shown in the web prototype. It is renumbered at every build, so do not store it or join on it across releases.

Change set since ve-audit-sample-r2-2026-09-12

0 products inserted, 0 retired, and 72 updated (93 changed values). Changes in this release come from these pipeline corrections:

• Latent heritage rule for area designations: World Heritage Sites, registered parks and gardens and registered battlefields are now measured from their Historic England boundary (boundaries retrieved 13 September 2026, pipeline/fetch_nhle_areas.py and area_rule.py) instead of one representative point. A visitor product record inside the site or within 500 m of its boundary counts, so the check latent_asset is removed where one exists (for example the Cornwall and West Devon Mining Landscape, with East Pool Mine and Heartlands inside), and nearest_product_m is the distance to the boundary (0 inside). Other sources keep the 12 September 2026 snapshot; no other rule changed.

Opening months: stated and inferred

Stated means the source states the months (OpenStreetMap opening_hours month rules or the seasonal tag). Inferred means opening hours with no month rules, assumed to be open all year: an assumption, not evidence. The column seasonality_evidence keeps them apart.

Match precision test and same-name check

These files let anyone reproduce the matching figures: the 819 pairs, the same-name pairs, the figures and the script (it reads the raw source downloads, which are not published).

Identifier hold-out. 819 OpenStreetMap features carry the Wikidata ID of an item within 500 m. With the ID hidden, the name + distance + category rule scored every Wikidata item within 500 m. 690 pairs reached the auto-merge band and 688 of them chose the tagged item: precision 99.7% (Wilson 95% interval 98.95–99.92%). 115 went to review and 14 found no match; 98.0% were merged correctly or sent to review. Tagged features lean towards well-known places, so this tests the rule; it does not replace a human-labelled sample. Wrong auto-merges:

• Radisson (way/29450342) -> The Edwardian Manchester, A Radisson Collection Hotel (Q111368147), tagged Free Trade Hall
• Hyatt Regency Manchester (way/452432847) -> Hyatt House Manchester (Q111873746), tagged Hyatt Regency Manchester

Same name nearby, two rules. Text rule (names equal after removing stop words such as “the” and “St”, within 100 m, every pair): 161 pairs, 94 in the same category (candidate missed duplicates) and 67 in different categories (different products at one site). Release rule (names equal with stop words kept, within 150 m, not a review-band pair, names used 5 or more times in one LVEP skipped): 108 pairs, flagged same_name_nearby and listed as same_name_candidate events.

Merge and split events

The product_lineage_event table records how products were formed in this release:

• merge: 2,470 rows. A source record merged into a product, for example a Wikidata item linked to an OpenStreetMap feature by a shared identifier.
• review_candidate_kept_separate: 228 rows. Two records that scored in the review band (0.60 to 0.85) and were kept separate for a reviewer.
• same_name_candidate: 108 rows. Two records with the same name within 150 m that the matcher did not link.
• unit_group: 2 rows. OpenStreetMap units, such as numbered chalets, listed as one site.
• split: 0 rows; product_merge: 0 rows; retire: 0 rows. Id registry events against the previous release.

Licences and attribution

OpenStreetMap

Licence: Open Database Licence (ODbL) 1.0. Attribution: © OpenStreetMap contributors.

The OpenStreetMap-derived layer of this release is a Derivative Database of OpenStreetMap and is made available under the Open Database Licence (ODbL) 1.0, with the attribution © OpenStreetMap contributors. Rows and values in that layer are identified by product.osm_derived = true, product.primary_source = 'osm', the product *_source columns with value osm, and product_source_xref rows with source = 'osm'. If you publicly use an adapted version of this layer, the ODbL share-alike terms apply to it.

Wikidata

Licence: Creative Commons CC0 1.0. Attribution: Wikidata, CC0 1.0 (attribution not required; given as a courtesy).

Wikidata values (identifiers, descriptions, websites, classes) are dedicated to the public domain under CC0 1.0.

National Heritage List for England

Licence: Open Government Licence v3.0. Attribution: © Historic England 2026. Contains Ordnance Survey data © Crown copyright and database right 2026.

NHLE list entry numbers, names, designations, grades and locations are used under the Open Government Licence v3.0.

ONS local authority boundaries (May 2026), mid-year population estimates, LSOA (2021) population-weighted centroids and Census 2021 residents

Licence: Open Government Licence v3.0. Attribution: Source: Office for National Statistics, licensed under the Open Government Licence v3.0. Contains OS data © Crown copyright and database right 2026.

Local authority codes and names (boundaries used to assign records) and mid-year population estimates are used under the Open Government Licence v3.0.

Caveats

• Illustrative sample built for the VisitBritain/VisitEngland tender BTA2235 response. It is not the production tool.
• Open data only: OpenStreetMap, Wikidata, Historic England NHLE and ONS. It contains no BTA or VisitEngland data and does not follow VisitEngland's schema.
• Covers 3 of 38 LVEPs (Cornwall, Greater Manchester, York & North Yorkshire). The LVEP to local authority membership is illustrative; Cornwall excludes the Isles of Scilly.
• prototype_id is not stable between builds. Use ve_product_id.
• Matching and classification are rules only: no machine learning and no human review. Some merges may be wrong, and pairs in the review band are kept separate rather than resolved. Category confidence is fixed per rule, not calibrated.
• Opening months are stated for only 53 of 16,559 visitor product records (0.3%): the source states the months. A further 1,593 are inferred to open all year only because their opening hours have no month rules; the rest have no month evidence. By LVEP, stated / inferred: Cornwall 24 / 379 of 4,198; Greater Manchester 3 / 726 of 5,674; York & North Yorkshire 26 / 488 of 6,687. 34 records show months closed.
• Open data coverage differs between places and categories. Missing records are not evidence that a product does not exist.
• Values are as retrieved on 2026-09-12 and may have changed at source since.