Table of Contents
There are various terms used for the movement of data, covering why you do it, how you do it, and when it should be done. Common data movement patterns include Migration, Offloading, Tiering, Replication, Back Ups and Archiving. In this series of blog posts, each approach will be explored in more detail.
In this post we will be looking at Data Archiving.
What is Data Archiving?
Data Archiving is the process of moving data from a production environment into a separate storage solution, typically optimised for low cost and long-term retention. Once archived, this data is usually removed from the source system to reduce its size and improve performance.
Archived data is usually retained in a structured and accessible format but is no longer part of day-to-day operations. It is usually removed from the source after a period of time which could be as little as minutes and up to whatever duration is appropriate.
If you have data that is bloating a production database and can be moved elsewhere, Data Archiving may be the solution you need.
Why Archive Data?
Data Archiving is typically driven by a combination of performance, cost, and compliance requirements.
- Performance:
Removing large volumes of historical data reduces table sizes, index depth, and overall I/O, improving query performance and maintenance operations (backups, index rebuilds, etc.). - Cost optimisation:
Archived data can be stored on cheaper storage tiers (e.g., Azure Archive, cold storage, or less performant disks). - Regulatory and compliance needs:
Many organisations are legally required to retain data for defined periods (e.g., finance, healthcare, insurance). A strong archiving strategy should integrate with data governance programs. Data retention policies often require records for 2 to 15 years. - Operational simplicity:
Keeping production systems lean makes troubleshooting, development, and reporting more efficient.
In short, archiving allows you to retain data without paying the operational cost of keeping it “live.” Active databases also run more efficiently when they contain only necessary data.
When Should You Archive Data?
The timing of archiving is highly dependent on business and regulatory requirements.
Typical approaches include:
- Time-based policies (most common)
e.g., archive records older than 12 months - Event-based triggers
e.g., archive data once a case/order/account is closed - Volume-based thresholds
e.g., archive when a table exceeds a certain size
Regardless of the rule used, archiving should be implemented as a consistent and automated process (e.g., scheduled jobs, pipelines, or policies) to ensure compliance is maintained over time.
What Should You Archive?
What you archive depends on both business value and technical constraints:
- Entire tables (e.g., audit/history tables)
- Subsets of data (e.g., rows older than X)
- Partitioned data (e.g., sliding window scenarios)
- Specific entities (e.g., inactive customers, completed transactions)
It is important to clearly define:
- Retention rules
- Data ownership
- Access requirements after archiving
Key Considerations
Access & Retrieval Strategy
Although archived data is no longer part of day-to-day operations, it is rarely “forgotten” data. In most cases, it still needs to be accessible – whether for audit purposes, reporting, or occasional business queries. Because of this, it is important to define how archived data will be accessed and what the expectations are around retrieval.
For example, will users be able to query the archive directly through reporting tools, or will access require a restore or rehydration process? The answer to this will often depend on the storage medium used. Data stored in a reporting database or secondary system may be available almost immediately, whereas data in deep archive storage could take hours to retrieve.
Clear expectations should also be set around performance and availability. Archived data does not typically need to meet the same SLAs as production systems, but there should still be a defined and documented approach so that consumers understand what is possible and how long it will take.
Storage Format & Structure
The way archived data is stored has a significant impact on its usability and longevity. A key decision is whether the data should retain the same structure as the source system or be transformed into a more storage-efficient format.
Keeping the original schema can make retrieval and reintegration simpler, particularly if the data needs to be queried in a relational way. However, this often comes at a higher storage cost. Alternatively, data can be exported into formats such as flat files, compressed datasets, or columnar storage formats like Parquet, which are designed for efficient storage and large-scale analysis.
It is also important to consider how the data is organised over time. Partitioning archived data—by date, business unit, or another logical boundary—can make both storage management and retrieval much more efficient, particularly as the archive grows.
Data Integrity & Validation
One of the most critical aspects of archiving is ensuring that no data is lost or corrupted during the process. Since archived data is often moved and then removed from the source system, there needs to be a high level of confidence that the archive is complete and accurate.
This typically involves validating row counts, ensuring that all expected records have been moved, and confirming that no data has been truncated or altered unintentionally. For more sensitive or critical datasets, additional validation techniques such as checksums or hash comparisons may be used.
Equally important is maintaining a detailed audit trail of the archiving process. This should capture what data was moved, when it was moved, and whether the operation completed successfully. Having this level of traceability is essential for both operational confidence and regulatory compliance.
Security & Compliance
Archived data often contains the same sensitive or regulated information as production systems and therefore must be treated with the same level of care from a security perspective. It is a common misconception that once data is archived, it is somehow less important or lower risk—this is rarely the case. However archiving data will limit the amount of data that is available in the production environment if unauthorised access occurs.
Appropriate controls should be put in place to ensure that archived data is encrypted both at rest and in transit. Access should be tightly controlled and limited to those who genuinely require it, with permissions reviewed regularly.
From a compliance standpoint, it is also important to consider how archiving interacts with regulations such as GDPR. For example, if a user exercises their “right to be forgotten,” the organisation must ensure that the data can be located and removed from both production and archived systems. This can be significantly more complex if the archive is not well structured or indexed.
Deletion & Retention Expiry
Archiving is not the final stage in the data lifecycle. In many cases, data is only required to be retained for a specific period, after which it should be securely deleted.
A clear retention policy should define how long different types of data must be kept, and what should happen once that period expires. These policies are often driven by legal, regulatory, or contractual requirements.
It is important that deletion processes are as well-managed as the archiving process itself. This includes ensuring that data is permanently removed, that appropriate records of deletion are maintained, and that any exceptions (such as legal holds) are handled correctly.
Without a defined retention and deletion strategy, archives can quickly become an ever-growing repository of data that increases both cost and risk.
Technical Implementation Patterns
There are a variety of technical approaches to implementing archiving, and the right choice will depend on the platform, data volumes, and business requirements.
Alternatively, ETL-based approaches using tools like Azure Data Factory or SSIS can be used to extract and move data into external storage systems.
File-based archiving to data lakes or blob storage is also increasingly common, particularly for large datasets or where analytical workloads are required. Each approach has its own trade-offs in terms of complexity, performance, and cost, and these should be carefully considered during design.
Impact on Downstream Systems
Archiving does not happen in isolation, and it is important to understand how removing data from a production system might affect downstream processes.
For example, reports or dashboards that previously relied on full historical datasets may no longer return complete results once data is archived. Similarly, applications that assume the presence of historical records may encounter errors if those records are no longer available.
There may also be relational considerations, such as foreign key dependencies, that need to be addressed before data can be safely removed. In some cases, a hybrid approach may be needed, where certain summary or reference data remains in the production system while detailed records are archived.
Careful analysis and testing are essential to ensure that archiving does not introduce unintended side effects.
Summary
Data archiving is not just a storage decision—it is a balance between cost, performance, compliance, and usability. A well-designed archiving strategy ensures systems remain efficient while still meeting business and regulatory obligations.
Book a call
Call us today if you need expert advice on moving data.
<< Back to Knowledge Centre