Data Types, Schema, and Provenance

After completing this topic

Before reading data, you will be able to check the data type and unit of each column and record the meaning of a single row as a schema. You will record the accession, download path, version, and terms of use of external data in the provenance, and distinguish public access from free redistribution.

Starting point: does opening a file mean you can analyze it?

The fact that a CSV file opens and the first row is visible does not mean preparation for analysis is complete. You need to check what the column names mean, whether the numbers are actual measured values or codes, and whether a single row is an independent unit or a repeated measurement. A data file is an input that goes into computation, but the file itself does not explain both the research question and the measurement process.

For example, consider a table with sample_id, condition, value, and date. If you do not know what unit value is in, whether date is the measurement date or the processing date, and whether sample_id is a biological unit or simply a file row number, you cannot begin statistical computation.

Schema: the contract for your data

A schema is a contract that records the structure of the data and the meaning of each field.

Data types are not decoration. Using an ID that looks like a number as a continuous variable creates the error of assigning magnitude and order to a name. If a date remains only as a string, you may get the time order or interval calculations wrong. Fix the data type, unit, allowed values, and missing-value representation together with the data.

provenance: where did the value come from?

Provenance is a record that traces the path along which a value was generated, collected, transformed, and stored. If you obtained data from an external repository, record at minimum the accession, the original page, the download date and time, the file hash, the transformation steps, and the citation information. The official guidance from GEO and ENCODE likewise treats accessions and source citations as part of an identifiable record.

This record does not automatically guarantee that the data is "genuine." Instead, it lets other people re-examine which inputs you used and which transformations you applied. If you converted raw count into normalized values, you must record the transformation code together with the input and output files.

Separating rights from access conditions

The fact that something can be downloaded from a public web page does not mean it is in the public domain or freely redistributable. An accession is an identifier for the data, and a license is the condition of use. Do not treat the two fields as the same thing.

When including data in educational content, record the following states separately.

• access: where can it be verified
• license: under what conditions can it be used
• redistribution: can the file itself be redistributed
• attribution: who and which project must be credited
• privacy/ethics: are there additional restrictions on educational reuse

If the rights status has not been confirmed, do not include the original file in the content; provide only the accession and the official access procedure. Copyright's distinction between ideas or facts and their expression helps in writing content independently, but it does not substitute for a legal judgment about any individual use.

Writing the schema and provenance together

A minimal manifest for a dataset can be composed as follows.

dataset_id: example-001
source_type: synthetic | repository
accession: none | verified accession
source_url: canonical page
downloaded_at: ISO 8601
file_sha256: verified hash
schema_version: 1.0.0
transformations: []
license_status: verified | pending | restricted
redistribution_status: allowed | link_only | prohibited | unknown

For synthetic data, record the fact that there is no accession as well. It is not that an original does not exist; rather, the generation rules, seed, generation code, and output schema themselves become the provenance. For external data, do not leave only the source URL — specify which file and which version were used.

Returning to the research question to interpret

Writing the schema and provenance lets you trace what the analysis results mean, because you can tell what unit the values are in, at what point they were measured, and what transformations they went through. However, complete provenance does not automatically prove the quality of the measurement or the validity of the research question.

Conversely, numbers without a source and units have a narrow range of interpretation even if computation is possible. Rejecting data before analysis is not a cumbersome procedure but a way of clarifying which inputs the results depend on.

A checklist to pass before analysis

Before starting the actual analysis, do not read the schema and provenance separately — cross-check them together.

1. Does analysis_unit match the unit of the curriculum and the research question?
2. Do the data type, unit, and measurement time of the outcome agree between the documentation and the table?
3. Do all IDs explain a uniqueness rule or a group structure?
4. Are there no notation discrepancies among categorical values (A, a, condition_A)?
5. Do the missing-value codes and exclusion rules remain in the original manifest?
6. Can you find the pre- and post-transformation files and the execution environment again?

This check is not a procedure for judging whether the data are perfect. It is a procedure for recording, at minimum, which assumptions you confirmed and which you could not confirm. Leave unconfirmed items as pending, and do not use expressions that raise the confidence of the analysis results.

Connecting provenance to sentences in the analysis

In a final report, provenance does not remain only in a separate administrative appendix. It must be able to answer the question, "From what data was this result computed?" For example, linking dataset_id, schema_version, transformations, and file_sha256 to the result manifest lets you distinguish previous results from new ones when a file with the same name has changed.

Conversely, if you record only the date the file was downloaded and omit the hash or the transformation steps, it is hard to confirm whether a different file was obtained from the same URL. Reproducibility is not completed by a single seed; it comes closer when the inputs, environment, transformations, and outputs can all be tracked together.

Common failures and how to check them

• Do not mistake a file name for an accession or a license.
• Do not use IDs that look like numbers as measured values.
• Do not merge NA, an empty string, and 0 into the same missing-value representation.
• Do not overwrite the original and make the pre- and post-transformation states untrackable.
• Do not treat public accessibility as permission to redistribute.

Key takeaways

• A schema is a data contract that describes rows, columns, data types, units, and missing values.
• Provenance is the record that tracks where data came from and the path of transformations it went through.
• Accession, license, redistribution, and attribution are separate fields.
• Synthetic data must also record its generation rules and version.
• When rights or provenance are uncertain, you can leave out the original file and keep only the official access path.

On to the next topic

In the next installment we visualize the center, spread, and distribution of data in more detail. Only after the schema and units are confirmed can quality checks be interpreted.

References

• NCBI GEO: https://www.ncbi.nlm.nih.gov/geo/info/
• ENCODE citing guidance: https://www.encodeproject.org/help/citing-encode/
• U.S. Copyright Office FAQ: https://www.copyright.gov/help/faq/faq-protect.html
• Creative Commons licenses: https://creativecommons.org/share-your-work/cclicenses/

The schema and manifest examples in this installment are educational constructions written independently by BioStatPy.