Developer Notes¶
Repository layout (public):
src/include/openmeta/: public headerssrc/openmeta/: implementationsrc/tools/: CLI toolssrc/python/: Python bindings and helper scriptstests/: unit tests and fuzz targets
tests/public_metadata_fixtures.h provides compact, source-built public
integration fixtures for TIFF/DNG with a representative Nikon MakerNote,
WebP, AVIF, JP2, JXL, RAF, X3F, and CRW. They contain no copied third-party
files and can be reused by downstream smoke tests without external assets. The
RAF and X3F fixtures include declared embedded JPEG metadata and a synthetic
entropy interval for proving that positional scans do not fetch image data.
OpenMeta structure¶
Development/correction and learning lifecycle in 0.8.11¶
The six existing development/correction fields A40D-A412 and LearningOptOutIn 9287 now use translation contract 2. Exact valid clean native matches gain Dirty while retaining values, wire hints/counts, origin and endian provenance. Dirty matches remain unchanged. Complete ReplaceExisting deletion retains typed native intent even when a key is absent; clean tombstones are promoted and dirty ones reused. Promotions and deletion intents consume operation limits, and new intents also consume entry limits. Conflict, source-mask and rollback rules remain.
Selected sources still require explicit host exif_version 300 or 310; translation never infers or changes ExifVersion or processing history. These APIs do not validate a complete destination EXIF profile. All seven fields use selective TIFF/BigTIFF/DNG merge and removal in both safety modes, including restored snapshots. The shared native allowlist grows from 77 to 84. Existing/template DNG version retention, source-pixel filters, ABI 4, snapshot v1 and host-owned synchronization remain unchanged.
LearningOptOutIn uses checked 16-bit word conversion from native provenance to canonical little endian and then to the target TIFF byte order. Unrelated UNDEFINED payloads remain opaque. Canonical encoding validates learning values before copying them.
See Interpretation Status for the semantic interpretation matrix.
OpenMeta’s public architecture is organized around a small set of user-facing capabilities. Internally some of these split into more stages, but the public model should stay compact:
Area |
Purpose |
Readiness |
|---|---|---|
Decoding |
Find metadata carriers and decode EXIF, XMP, IPTC, ICC, Photoshop IRB,
JUMBF/C2PA, EXR, and related blocks into |
High, about 98-100% for the current target scope. |
Interpretation |
Normalize names and values, group entries by meaning, and classify
source-bound data such as RAW crop, exposure adjustment,
color/profile/source-color-transform evidence, RAW curves/linearity
metadata with descriptor-backed compressed-storage applicability,
lens-correction, sensor, BMFF brand/item-property
associations and rollups including |
High, measured about 99.85% for declared semantic targets. |
Query |
Find entries by exact name or semantic group, then expose normalized query candidates, structured interpretation records, and bounded cross-family concept resolutions, transfer hints, sensitivity, and conflict flags for crop/border/active-area, exposure/gain, color/WB/profile/source-color-transform, orientation, date/time, GPS, descriptive fields including contact/event/person/organization/product/ artwork/rights/license/release, editorial, accessibility, taxonomy, registry, image-region, document-identity, document-lineage, and document-history, technical-image, audio, and preview semantics, lens-correction, computational/thermal/stitch, and RAW/source-processing fields plus BMFF derived-image construction and tiled-image configuration evidence across standard and vendor metadata. |
High, measured about 99.77% for declared query targets. |
Fuzzy Search |
Optionally find misspelled, aliased, or near-match metadata names and property paths with bounded deterministic top-k ranking and explicit exact/alias/fuzzy provenance. |
High enough for the current milestone, about 80-85%; the standalone RapidFuzz-backed API, curated positive/adversarial quality gate, bounded ASCII contract, Python wrappers, Release/libc++ CI gate, and opt-in scaling benchmark are implemented. |
Creation |
Build fresh metadata entries through either logical fields or exact typed keys. |
High, about 82-87%; the logical API and thin Python wrapper remain simple, while the C++ generic v1 builder covers EXIF/TIFF/DNG-style tags, IPTC, custom scalar/indexed XMP, borrowed typed values, wire hints, capacity planning, and detached validation. Full RDF authoring and fresh ICC construction remain open. |
Editing |
Modify existing logical metadata entries while preserving valid surrounding structure. |
Medium-high, about 75-80%; the v1 logical add/set/remove transaction, deterministic occurrence handling, singleton conflict repair, dirty/tombstone behavior, provenance preservation, portable serialization, transfer visibility, and immutable thin Python wrapper are implemented. |
Transfer |
Move metadata between files using explicit compatible-file or rendered-image safety policies. |
High, about 85-90%; the primary writer contract, bounded EXIF/IPTC/XMP synchronization policy, compare-backed release gates, stable prepared handoff, and TIFF/DNG baseline are complete. |
Translation |
Project metadata between families through explicit bounded mappings, including strict paired IPTC creation-date projection to XMP and transactional reverse projection of edited XMP dates, common technical and capture fields, target-bound image geometry, descriptive properties, flat IPTC Core locations, and editorial fields into native EXIF/IPTC groups. |
Medium-high, about 82-87% for the declared scope; exact date parsing, four technical and five typed capture EXIF mappings, target-validated orientation and complete stored-dimension groups, 20 IPTC text/priority mappings with combined atomic writeback, primary GPS coordinate/altitude groups with exact rational precision and version handling, structured-location record selection and paired flat-XMP/IPTC reconciliation, GPS UTC date/time and navigation pairs, repeated-value reconciliation, exact rational conversion, safe ASCII/UTF-8 handling, tombstone propagation, resource limits, JPEG/TIFF round trips, and thin Python exposure are implemented. |
Writing |
Serialize metadata and write or rewrite it into target containers. |
High, about 78-84%; target-neutral deterministic TIFF/EXIF measure/write
feeds JPEG/TIFF, PNG/WebP, JP2/JXL, and BMFF wrappers. JPEG is strong,
TIFF/DNG have a bounded rewrite contract, and compact BMFF |
Adapters |
Thin integration layers for host APIs or format-specific ecosystems such as EXR, DNG SDK, LibRaw orientation mapping, and flat host exports. |
Medium, about 60-70%. |
Utilities |
Small standalone helpers such as capability queries, compatibility dumps, safety audits, tag-name lookup, version-value formatting, and orientation conversion. |
Medium, about 65-75%. |
The measured interpretation and query values use explicit target sets.
Fuzzy Search is not included in the 99.77% Query audit; its readiness is a
separate architectural estimate.
Unknown private numeric IDs and intentionally opaque payloads are tracked
separately rather than counted as failed semantics. The remaining measured
tail is primarily malformed, undefined, or incomplete metadata. Decoding
retains a 98-100% range because not every declared container lane has an
independent conformance sample set, even though tracked inputs have explicit
read outcomes.
Logical and generic typed Creation, detached validation, canonical TIFF/EXIF serialization, Editing, and current reverse Translation milestones are implemented in their public headers. Their contracts are documented in Metadata Creation, Generic Typed Metadata Authoring, Canonical Metadata Serialization, Metadata Editing, and Metadata Translation. The active sequence continues through Transfer, Translation, and Writing. Creation and Editing resume for multilingual alternatives, full RDF structured values, fresh ICC construction, structural block operations, and broader cross-family synchronization. Adapters and Utilities remain deferred. Fuzzy Search has the lowest priority and resumes after those stages. Its remaining work includes independently sourced quality expansion, designed Unicode/transliteration behavior, multilingual gates, and an optional immutable index for repeated searches over large stores.
Query results should expose both inspection-level matches and interpreted
candidates. A crop query, for example, may match separate
DefaultCropOrigin and DefaultCropSize tags, an ActiveArea rectangle,
vendor margin fields, or a raw integer array. OpenMeta should return the source
entries, confidence, value shape, match provenance, and any normalized
interpretation rather than hiding ambiguity behind a single value.
The first experimental C++ query surface is openmeta/metadata_query.h.
It returns both raw matches and normalized candidates for crop/active-area,
exposure/gain, white balance, color/profile, lens correction, orientation,
descriptive, and RAW-processing queries.
Crop queries include DNG crop tags, ActiveArea, Phase One/Leaf raw
geometry, Fujifilm RAF raw crop/zoom rectangles, Canon aspect/crop metadata,
Nikon Capture crop bounds, Sony panorama crop margins, and
crop/border-style XMP property paths. The non-crop queries expose per-entry
value candidates and reuse standard tag names, selected DNG tags, matching XMP
paths, canonical border-margin parsing, and vendor RAW-processing
classification where applicable.
They also append grouped candidates for related DNG color matrix/calibration/
reduction/forward matrix tags, DNG white-balance vector tags, and
lens-correction table groups. Color queries expose a distinct
color_profile semantic for EXIF color-space evidence, ICC header/tag
entries, XMP ICC/profile/color-space fields, and PNG profile text carriers.
Vendor-classified MakerNote/RAW fields can also form per-family grouped
candidates for white balance, color, raw-storage, sensor, computational,
thermal, stitch/panorama, source-processing, raw value curve, linearity-limit,
calibration-curve, and curve-control-point records. RAW-processing queries add
conservative groups for black/white levels, linearization tables, RAW value
curves, RAW linearity limits, RAW calibration curves, RAW curve control points,
CFA/sensor layout, source geometry, raw-storage identifiers, and source-private
processing buckets.
Exposure/gain concept resolution promotes exposure time, aperture, ISO,
exposure bias, exposure program/mode, gain, and raw exposure-adjustment records
into host-visible roles, with raw exposure adjustments kept unsafe for rendered
targets. Standard EXIF exposure program/mode and gain-control values and
selected Canon/Nikon/Sony/Fujifilm/Pentax/Olympus/Panasonic/Phase One/Kodak/
Minolta/Sigma/Samsung/Ricoh/Apple/FLIR/JVC/GE/Reconyx/Microsoft/Nintendo/
Sanyo MakerNote scalar print conversions are exposed as bounded labels when a
stable enum mapping is
available.
Version/firmware-like fields use exif_tag_numeric_value_format(...) and
exif_tag_byte_value_format(...) instead of enum-label lookup where the
value is a formatted payload rather than a closed numeric choice.
Current source-private aliases include camera-to-XYZ/RGB matrices, creative and
picture styles, film simulation, dynamic-range processing, optical/lens
correction, white-balance gains, and raw-development terms.
Grouped candidates use matrix_set, vector_set, and table value
shapes. Color matrix sets, white-balance vector sets, and lens-correction
tables are promoted only when the numeric payloads meet conservative minimum
shapes; other records stay visible as per-entry matches/candidates.
Semantic Query always uses deterministic tag, namespace, and name matching so
similar names cannot change classification. Raw matches retain exact_match,
fuzzy_match, and fuzzy_score compatibility fields, but tolerant
free-text ranking belongs to the separate Fuzzy Search API.
Python Document and TransferSourceSnapshot mirror this as thin wrappers
returning the same match/candidate dictionary shape.
Fuzzy Search is a separate optional stage layered over Query. See
Fuzzy Search for its complete text, ranking, resource, threading,
quality, and benchmark contracts. The
openmeta/metadata_fuzzy_search.h API searches decoded entry names and
property paths with caller-selected score and result bounds. Results are
ordered by descending score, then exact/alias/fuzzy provenance, then stable
entry id. Search is locale-independent and currently accepts ASCII query text
only; non-ASCII queries return an explicit unsupported status rather than
performing byte-wise similarity or implicit transliteration. The curated
quality gate covers common spelling errors, aliases, metadata-like adversarial
negative queries, deterministic top-k truncation, deleted-entry filtering, and
ASCII/UTF-8 boundary behavior. An opt-in benchmark confirms linear scaling and
supports keeping the scan path for ordinary per-image stores; a reusable
immutable index remains deferred for repeated GUI queries or aggregated
stores. Builds without RapidFuzz retain exact and semantic Query behavior.
For code that wants an iterable semantic record stream instead of raw query
matches, use openmeta/metadata_interpretation.h. It projects query
candidates into records with query class, semantic kind, normalized shape,
confidence, source entries, and normalized geometry/value arrays where
available.
For cross-family duplicated concepts, use openmeta/metadata_concepts.h.
It currently resolves orientation, date/time, exposure/gain, color/profile,
GPS, descriptive fields, geometry, lens-correction, and RAW-processing into
candidate lists with
candidate source entries, source families, preferred entries, normalized
compare keys, parsed date/time fields, date/time precision, timezone kind, GPS
altitude-reference state, distinct destination/shown/created coordinate roles,
structured-location scope, normalized descriptive language, additive
collection preference, structured record kinds and scopes for editorial,
rights, license, release, end-user, image-creator, image-supplier, and
image-asset, controlled-vocabulary, registry, image-region,
resource-reference, resource-event, manifest-item, version,
editorial-workflow, source-software, editorial-contact, technical-image,
audio-asset, preview-asset, and pantry records,
including non-equivalent IPTC taxonomy/workflow fields, content-location and
prior-envelope scopes, and editorial release/expiration date-time pairs,
independent policy
sensitivity,
canonical geometry origin/size/rect/margins,
normalized exposure values, full normalized value vectors for grouped
matrix/vector/table records, transfer hints, compatible and rendered safety
booleans, and same-role conflict flags. This is deliberately an
inspection/policy surface; host code still decides whether a conflict should
be shown, ignored, or corrected during editing/transfer.
Read-path coverage snapshot¶
Tracked HEIC/HEIF, CR3, and mixed RAW EXIF compare gates are passing.
EXR header metadata compare gate is passing for the documented name/type/value-class contract.
MakerNote support is broad and baseline-gated; unknown tags remain lossless.
EXIF + MakerNotes (code organization)¶
Core EXIF/TIFF decoding:
src/openmeta/exif_tiff_decode.ccCRW/CIFF decode + derived EXIF bridge:
src/openmeta/crw_ciff_decode.ccVendor MakerNote decoders:
src/openmeta/exif_makernote_*.cc(Canon, Nikon, Sony, Olympus, Pentax, Casio, Panasonic, Kodak, Ricoh, Samsung, FLIR, etc.)Shared internal-only helpers:
src/openmeta/exif_tiff_decode_internal.h(not installed)Unit tests for MakerNote paths:
tests/makernote_decode_test.cc
Internal helper conventions (used by vendor decoders):
read_classic_ifd_entry(...)+ClassicIfdEntry: parse a single 12-byte classic TIFF IFD entry.resolve_classic_ifd_value_ref(...)+ClassicIfdValueRef: compute the value location/size for a classic IFD entry (inline vs out-of-line), usingMakerNoteLayout+OffsetPolicy.MakerNoteLayout+OffsetPolicy: makes “value offsets are relative to X” explicit for vendor formats.OffsetPolicysupports both the common unsigned base (default) and a signed base for vendors that require it (eg Canon).ExifContext: a small, decode-time cache for frequently accessed EXIF values.MakerNote tag-name tables are generated from
registry/exif/makernotes/*.jsonland looked up via binary search (exif_makernote_tag_names.cc).
Interop adapters¶
export-only naming/traversal surface:
src/include/openmeta/interop_export.hexport-only adapter:
src/include/openmeta/ocio_adapter.hhost-apply adapter:
src/include/openmeta/exr_adapter.hdirect bridge:
src/include/openmeta/dng_sdk_adapter.hnarrow translator:
src/include/openmeta/libraw_adapter.h
Notes:
ExportNamePolicy::ExifToolAliasandExportNamePolicy::Specare both covered by interop tests and used for split-parity workflows.Flat host-style interop naming keeps numeric unknown names (
Exif_0x....) for parity workflows.
Python binding entry points:
Document.export_names(...)Document.ocio_metadata_tree(...)Document.unsafe_ocio_metadata_tree(...)Document.dump_xmp_sidecar(...)(lossless or portable via format switch)Document.phaseone_raw_geometry()andDocument.phaseone_raw_processing()for normalized Phase One/Leaf RAW source metadata queries.Document.vendor_raw_processing(family)for Sony/Canon/Nikon/Fujifilm/Pentax/Panasonic/Olympus/Kodak/Minolta/Sigma/ Samsung/Ricoh/Apple/DJI/Google/FLIR/Casio/Sanyo/KyoceraRaw/Reconyx/HP/JVC/ GE/Motorola/Nintendo/Microsoft grouped RAW/source-processing field summaries.
C++ adapter entry points:
visit_metadata(...)inopenmeta/interop_export.his the intended base for host-owned metadata mappingsbuild_exr_attribute_batch(...)inopenmeta/exr_adapter.hexports one owned EXR-native attribute batch (part_index,name,type_name,value,is_opaque) fromMetaStorebuild_exr_attribute_part_spans(...)groups that batch into contiguous per-part spansbuild_exr_attribute_part_views(...)exposes zero-copy grouped per-part views over the same batchreplay_exr_attribute_batch(...)replays the grouped batch through explicit host callbacks
Python typed behavior:
Document.export_names(style=ExportNameStyle.FlatHost, ...)exposes the stable v1 flat-host naming contract used by host-side metadata mappings. See FlatHost Mapping Contract.Document.ocio_metadata_tree(...)is safe-by-default and raises on unsafe raw byte payloads; useDocument.unsafe_ocio_metadata_tree(...)for legacy/raw fallback output.safe API:
build_ocio_metadata_tree_safe(..., InteropSafetyError*)unsafe API:
build_ocio_metadata_tree(...)build_ocio_metadata_tree(..., const OcioAdapterRequest&)inopenmeta/ocio_adapter.h(stable flat request API)build_ocio_metadata_tree(..., const OcioAdapterOptions&)(advanced/legacy shape)
C++ XMP sidecar entry points:
dump_xmp_sidecar(..., const XmpSidecarRequest&)inopenmeta/xmp_dump.h(stable flat request API)dump_xmp_sidecar(..., const XmpSidecarOptions&)(advanced/legacy shape)
Optional dependencies¶
OpenMeta’s core scanning and EXIF/TIFF decoding do not require third-party libraries. Some metadata payloads are compressed or structured; these optional dependencies let OpenMeta decode more content:
Expat (
OPENMETA_WITH_EXPAT): parses XMP RDF/XML packets (embedded blocks and.xmpsidecars) using a streaming parser with strict limits.RapidFuzz (
OPENMETA_ENABLE_RAPIDFUZZ): opt-in bounded fuzzy entry search for inspection UI. It is disabled by default; when enabled, CMake requires either arapidfuzz::rapidfuzzpackage target orOPENMETA_RAPIDFUZZ_INCLUDE_DIRpointing at headers containingrapidfuzz/fuzz.hpp.zlib (
OPENMETA_WITH_ZLIB): inflates Deflate-compressed payloads such as PNGiCCP(ICC profiles) and compressed text/XMP chunks (iTXt,zTXt).Brotli (
OPENMETA_WITH_BROTLI): decompresses JPEG XLbrob“compressed metadata” boxes so wrapped metadata payloads can be decoded.
CLI tool¶
metaread prints a human-readable dump of blocks and decoded entries
(EXIF/TIFF-IFD tags, XMP properties, IPTC-IIM datasets, ICC profile fields/tags,
and Photoshop IRB resource blocks). Output is ASCII-only and truncated by
default to reduce terminal injection risk.
metavalidate reports decode/validation issues in text or JSON and emits
machine-readable issue codes (for example xmp/output_truncated and
xmp/invalid_or_malformed_xml_text) suitable for CI gating. Draft C2PA
verification results are diagnostic only: signature success is reported as
signature_verified_only and does not establish asset hard binding. The
trusted-chain option binds certificate trust to the signature key, but does
not make the result an asset-authenticity or trust gate.
Python¶
Python bindings use nanobind. The wheel also ships helper scripts as
openmeta.python.* modules.
python3 -m openmeta.python.metaread file.jpg
python3 -m openmeta.python.metadump --format portable file.jpg
python3 -m openmeta.python.metadump file.jpg output.xmp
python3 -m openmeta.python.metadump --format portable --c2pa-verify --c2pa-verify-backend auto file.jpg
python3 -m openmeta.python.metadump --format portable --c2pa-verify --c2pa-verify-require-trusted-chain file.jpg
python3 -m openmeta.python.metadump --format portable --portable-include-existing-xmp --xmp-sidecar file.jpg
openmeta.python.metatransfer remains a thin command-line wrapper. Its
--xmp-writeback, --xmp-destination-embedded,
--xmp-destination-sidecar, --output, and --force flags map directly
onto the C++ file-helper options and persistence flags. It reports sidecar and
cleanup paths returned by the C++ result instead of deriving a separate
Python-side contract. Its --target-width, --target-height,
--target-orientation, --target-samples-per-pixel,
--target-bits-per-sample, --target-sample-format,
--target-photometric, --target-planar-configuration,
--target-compression, and --target-exif-color-space flags populate the
same target image spec used by the C++ transfer request.
Resource policy defaults¶
For C++ callers, initialize from recommended_resource_policy() and only
override fields you need:
#include "openmeta/resource_policy.h"
openmeta::OpenMetaResourcePolicy policy
= openmeta::recommended_resource_policy();
policy.jumbf_limits.max_box_depth = 24; // optional override
For JUMBF/C2PA preflight traversal checks, call
measure_jumbf_structure(bytes, policy.jumbf_limits) before full decode.
Other preflight estimate APIs use the same bounded-options model:
measure_scan_auto(file_bytes)measure_scan_jpeg(bytes),measure_scan_jpeg_random_access(jpeg_range, scratch, read_limits),measure_scan_png(bytes),measure_scan_png_random_access(png_range, scratch, read_limits),measure_scan_webp(bytes),measure_scan_webp_random_access(webp_range, scratch, read_limits),measure_scan_gif(bytes),measure_scan_tiff(bytes),measure_scan_mrw(bytes),measure_scan_jp2(bytes),measure_scan_jp2_random_access(jp2_range, scratch, read_limits),measure_scan_jxl(bytes),measure_scan_jxl_random_access(jxl_range, scratch, read_limits),measure_scan_bmff(bytes),measure_scan_bmff_random_access(bmff_range, scratch, read_limits)measure_exif_tiff(exif_bytes, exif_options)measure_xmp_packet(xmp_bytes, xmp_options)measure_icc_profile(icc_bytes, icc_options)measure_iptc_iim(iptc_bytes, iptc_options)measure_photoshop_irb(irb_bytes, irb_options)measure_exr_header(exr_bytes, exr_options)measure_jumbf_payload(jumbf_bytes, jumbf_options)
Documentation build¶
Sphinx docs require:
doxygenPython packages listed in
docs/requirements.txt
uv pip install -r docs/requirements.txt
cmake -S . -B build -DOPENMETA_BUILD_SPHINX_DOCS=ON
cmake --build build --target openmeta_docs_sphinx
The 0.5.6 editing/translation batch¶
Exact-key typed editing now covers supported EXIF/IPTC/XMP and private/custom entries with complete candidate validation and explicit wire-hint replacement. Three additional capture, six environment, six encoding/composite and four structured fields and the 0.5.9 text/version batch bring capture-related translation to 75 unique ExifIFD targets across sixteen APIs. See Metadata Editing, Metadata Translation and Capture Synchronization Milestone. Fuzzy Search remains the lowest priority, after Adapters and Utilities.
Structured Capture Update (0.5.8)¶
OECF, SpatialFrequencyResponse, CFAPattern and DeviceSettingDescription add
four native targets in one transaction through
translate_xmp_structured_capture_metadata and its Python counterpart.
The 0.5.8 combined inventory was 65 distinct native tags across fifteen APIs
(62 retained by compatible-file transfer). ABI 3 and host synchronization
remain unchanged; big-endian snapshots for these four tags need a 0.5.8
reader. See Metadata Translation for the encoding, bounds and conflict contract.
EXIF Text Update (0.5.9)¶
The 0.5.9 text/version batch adds UserComment, both version fields and seven EXIF 3 text tags. The new EXIF text API also supports UTF-8 owner/lens fields. The combined capture inventory is 75 distinct ExifIFD tags across sixteen APIs; compatible-file transfer retains 72. Version and Artist/Software companion requirements are explicit. ABI 3, snapshot v1 layout and host synchronization responsibilities remain unchanged. BOM-less big-endian UserComment snapshots require a 0.5.9 reader. See the EXIF text/version translation contract for limits and the recorded OIIO/ExifTool reader limitations.
EXIF 3.1 Development and Correction Update (0.5.11)¶
The six development/correction fields A40D-A412 use one transactional C++ and Python contract. The host selects EXIF version 0300 or 0310 for selected writeback; no version is inferred or changed. Shared validators cover editing, translation, portable XMP and serialization. Present standard values remain in transfer output, including rendered-image output; RAW and MakerNote history is not inferred. See Metadata Translation.