What is data lineage?
Data lineage is the record of how data moves and changes from its source to where it is ultimately used.
It can show where a piece of information originated, which systems or processes handled it, what transformations were applied, and where the resulting data was consumed.
For intelligence and investigation teams, this matters because the meaning and reliability of information depend partly on its context. An analyst may need to know whether a name came from a police report, a company registry, an intelligence source, or an automated extraction process. They may also need to understand whether the information has been transformed, combined with other data, or passed through additional systems before appearing in an analytical result.
Data lineage provides that context by making the journey of information visible.
Why does data lineage matter?
Investigations increasingly depend on information drawn from multiple systems and sources.
A single analytical result might depend on data from case management systems, financial records, communications data, external databases, documents, and other intelligence holdings. When those sources are combined and transformed, it can become difficult to understand how an individual piece of information reached the point where an analyst is using it.
Data lineage helps answer questions such as:
- Where did this information come from?
- Which systems contributed to it?
- What transformations were applied?
- Which reports, analyses, or downstream systems depend on it?
- What could be affected if the source data changes?
This is useful for troubleshooting and data quality, but it is also important when analysts need to understand and explain the basis of an intelligence product or assess the potential impact of a change to an underlying dataset.
Data lineage and provenance
Data lineage and provenance are closely related, but they describe slightly different aspects of an information journey.
Data lineage focuses on the flow of data through systems and processes. It can show how information moves from source to destination and how it is transformed along the way.
Provenance is broader and focuses on the origins, history, and contributing sources or activities associated with a particular piece of information or resulting artefact. The W3C provenance model, for example, describes the entities, people, and activities involved in producing or influencing data.
In an intelligence environment, the distinction can be useful. Lineage might show that information moved from a source system through an ingestion pipeline into an analytical dataset. Provenance can provide additional context about the source and processes behind a specific piece of information.
How does data lineage work?
A lineage model typically connects three elements:
Source
The source is where data originates or enters an organisation’s data environment. This could be a database, application, external dataset, document repository, or another system.
Transformation
Between source and destination, data may be cleaned, standardised, filtered, enriched, aggregated, matched, or otherwise transformed.
Understanding those transformations is important when assessing how the resulting information was produced.
Destination
The destination is where the resulting data is consumed. This could be a report, dashboard, analytical environment, database, application, or another downstream process.
A lineage view connects these stages so that users can trace the flow of information through the wider data environment. Modern lineage systems can capture this information at different levels of granularity, from broad system-to-system flows to more detailed relationships between individual data assets.
What can data lineage help investigators understand?
Although data lineage is often discussed as a data governance concept, it can also support investigative and analytical workflows.
Understanding the origin of information
An analyst may need to establish where a particular data point came from and whether it originated in an authoritative source, an external dataset, or an earlier analytical process.
Investigating unexpected results
If an analytical result looks incorrect, lineage can help trace the data backwards through the processes that produced it and identify where an issue may have been introduced. This is one of the established uses of lineage in data environments.
Assessing the impact of change
If a source dataset changes, lineage can help identify which downstream datasets, reports, or analytical products depend on it.
This kind of impact analysis is particularly important in complex environments where a change to one system can affect many downstream consumers.
Supporting confidence and transparency
When information contributes to an intelligence assessment, understanding its origins and transformations can help analysts assess how it should be interpreted and explain the basis of their work.
Lineage does not establish that information is correct. It provides visibility into how that information was produced and where it came from.
Data lineage in complex investigations
Consider an investigation involving a person of interest.
Information about that person might appear in several systems, each using different identifiers or representations. Data from those systems could then be normalised, entity-resolved, combined, and made available to analysts through a shared intelligence environment.
This becomes increasingly important as organisations connect more sources and build richer analytical environments. Open standards such as OpenLineage exist specifically to support consistent collection and exchange of lineage metadata across data systems.
Why graph technology is useful for data lineage
Lineage is fundamentally about relationships.
A source is connected to a process. That process produces or changes a dataset. The dataset is consumed by another process, which may produce another result. As the number of systems and dependencies grows, those relationships can become difficult to understand in traditional lists or documents.
Graph technology provides a natural way to model these connections.
A lineage graph can represent:
data sources → processes → datasets → transformations → analytical outputs
This makes it possible to explore lineage as a connected structure rather than as a static document.
Graph-based approaches can also make it easier to combine lineage with other contextual information, such as data ownership, provenance, entity relationships, or operational metadata.
Data lineage and knowledge graphs
Data lineage and knowledge graphs address related but different problems.
A knowledge graph represents entities and the relationships between them to provide a connected model of a domain.
Data lineage represents how information moves and changes through data systems and processes.
They can work together.
For example, a knowledge graph might represent people, organisations, accounts, locations, and events in an investigation, while lineage information can describe which source systems and transformations contributed to those entities and relationships.
Combining the two can give analysts both context about what the information means and visibility into where it came from.
How GraphAware Hume supports provenance and data context
GraphAware Hume is designed to maintain context around information as it is brought together into a connected intelligence picture.
When data from multiple systems is combined, the same real-world person, organisation, or location may appear in several source datasets. GraphAware Hume can connect those records while preserving information about their originating sources, helping analysts understand relationships across datasets without losing the underlying source context.
This source awareness supports more transparent analysis because analysts can distinguish between information from different systems and understand how records contribute to the wider intelligence picture.
GraphAware Hume combines this with graph visualisation and investigative analysis, allowing analysts to explore relationships while retaining visibility of evidence and provenance.
The result is not simply a record of where data came from. It is a connected analytical environment in which source context can remain part of the investigation.
FAQs
What is data lineage?
Data lineage is the record of how data moves and changes from its source through different systems and processes to where it is ultimately consumed.
Why is data lineage important?
Data lineage helps organisations understand where data came from, how it was transformed, what depends on it, and how changes or errors may affect downstream results. It is commonly used for troubleshooting, data quality, governance, compliance, and impact analysis.
What is the difference between data lineage and data provenance?
Data lineage focuses primarily on the flow and transformation of data through systems and processes. Provenance provides broader information about the sources, entities, people, and activities involved in producing or influencing a piece of data.
What is a data lineage diagram?
A data lineage diagram is a visual representation of how data moves between sources, processes, datasets, and destinations. It can show both the flow of information and the transformations applied along the way.
How is data lineage used in investigations?
Data lineage can help investigators understand the origin and history of information, investigate unexpected analytical results, assess the impact of changes to source data, and explain how information reached an analytical output.
Can data lineage be represented as a graph?
Yes. Because lineage consists of entities and relationships between them, it can be represented as a graph. This allows users to explore dependencies and trace connections between sources, processes, datasets, and outputs.
Does data lineage prove that information is accurate?
No. Lineage shows where information came from and how it was processed. It helps users assess context and trace problems, but the existence of lineage does not by itself establish that the underlying information is accurate.