Digital archive advice for record managers

As-a-Service means that the State does not own, operate, or maintain the hardware and software for the digital archiving solution. This means that QSA and agencies can be focused on the core business of recordkeeping and do not have to be skilled in hardware or software management, monitoring and maintenance.

As-a-Service also means that the hardware and software are ‘evergreen’ with the Supplier obligated to keep the solution components as current as possible.

QSA is tackling the challenges of digital preservation, ensuring continued access to digital information in the future for Queensland Government, public authorities, and the broader community.

Digital records are at risk of loss through technology change and obsolescence. Digital Preservation encompasses a comprehensive set of well-managed activities intended to ensure the ongoing accessibility of digital materials for as long as required. Digital records are susceptible to various technology-related threats, including hardware and software obsolescence and bit rot. While reliable IT solutions such as digital storage and backups are pivotal, they represent just one facet of the broader picture. Digital Preservation also encompasses a wide range of organisational and resource-related considerations. These encompass risk assessment, sustainability planning, skills development, and change management. In addition to preserving the original digital files, it is equally essential to capture information about the file's context and any relevant documentation. This approach enhances the ability of future users to open and comprehend the files.

Our digital transfer model will include four basic elements:

• Measuring and testing the feasibility of a potential transfer
• Working closely with government agencies to plan and prepare digital records and metadata for transfer
• Making the digital records accessible and searchable online in ArchivesSearch (for open records)
• Making the digital records accessible and searchable online in ArchivesGateway to users with appropriate permissions

Once records are transferred safely into the digital archive the records are monitored over time to ensure ongoing integrity and accessibility.

Transfer of digital records will largely be via upload in ArchivesGateway. This mechanism enables some key data integrity checks like checksums and anti-virus to happen at the time of upload. In case of errors and failures, remediation can occur immediately.

It is also important to distinguish between born-digital records and the digital carriers they are sometimes stored on. The digital transfer process requires agencies to upload their digital records at the individual file level. Any content saved to digital carriers including (but not limited to) CDs, floppy disks, USBs or hard drives will need to be extracted from the carrier first and then uploaded via the agency portal (ArchivesGateway).

QSA is willing to work with agencies to help analyse and extract any digital holdings stored on digital carriers.

Where size of transfer or bandwidth limitations are present, alternative transfer methods can be agreed upon. These can include:

• Physical media where files are sent on an approved physical media device (e.g. hard drives)
• Snowball where files are copied onto an encrypted AWS Snowball device by agencies. This option is only available for extremely large transfers.

Many of the process steps involved in both physical and digital transfers share similarities, however, it's important to note that the sequence of these steps for digital transfers may not always align with those for physical transfers.

A comprehensive analysis is necessary in advance to evaluate the feasibility of the digital transfer and to create a well-thought-out plan.

Flexibility is key when preparing digital transfers.  Expect to encounter numerous instances where analysis needs to be conducted and issues related to files, metadata, or tools must be discussed and resolved throughout the entire process.

For a digital transfer project to succeed, adequate staff resourcing must be available. These skills should encompass expertise in information and records management and technical knowledge. The staff allocated to the transfer should be well-prepared for ongoing discussions, correspondence, and meetings particularly at the outset of the process, to address any issues and questions. These meetings play a pivotal role in ensuring the success of the transfer.

While we don't impose specific file format requirements for transfers, we do provide recommendations for suitable formats for long-term preservation.

We are prepared to accept any file formats that an agency provides. Born digital transfers are ingested and stored in Queensland’s digital archive, where we utilise Archivematica, a long-term preservation system developed by Artefactual, for managing and preserving born-digital digital records.

Archivematica can accept most file formats. One notable exception, however, is zipped files. QSA has intentionally adopted a policy of extracting individual files from zip files prior to ingest for the following reasons:

1. Each file is individually listed as an item in the archival management system. If the files remained zipped, they cannot be correctly matched to their listing.
2. Preserving files individually is considered a more stable digital preservation approach. Whilst zip files are an efficient means of reducing storage overhead, we cannot always attest to how the files have been compressed and whether any bit loss has occurred (especially for older zip files that were generated by bespoke tools).
3. Discoverability and access to files within a zipped archive are compromised. The lack of individual listings and detailed descriptions, coupled with concealed metadata, hampers their searchability.

QSA adopts a prudent approach to digital preservation, with a primary focus on preserving the original files and normalising them when necessary. This is achieved by ensuring that the files are free from viruses, conducting fixity checks to ensure file integrity over time, and having a comprehensive understanding of the file formats present in our collections. This proactive approach ensures the long-term viability and accessibility of digital records.

Queensland’s digital archive is hosted within Amazon Web Services (AWS) in Australia, with no data stored outside of Australia.

The environment is managed and monitored by QSA’s As a Service partner using AWS monitoring and security control services. The As a Service partner is ISO270001 certified and is audited on this requirement every year.

All customer data stored in AWS is encrypted at rest. All data stored in AWS is copied across 3 data centres that are geographically separate.  The digital archive also maintains a copy of your data at CITEC. This data is also encrypted at rest.

ArchivesGateway (the Agency portal) requires multi factor authentication to log in with audit trails enabled. ArchivesGateway users are all role based and associated with their relevant agency only.

Digital records transferred to QSA will be visible via ArchivesGateway, but access to them will follow the principle of least privilege, ensuring that users will only have access to what they absolutely need in order to perform their responsibilities, and no more.

Digital records visible in ArchivesGateway will be classified as non-sensitive, sensitive, or highly sensitive.  The classification will be applied by the agency in consultation with QSA. Access to the records will be governed by the access classification.

Open digital records will be available via ArchivesSearch.

According to QSA Digital Preservation Policy, only born digital records created and managed digitally for business purposes by public authorities will be ingested into the digital archive.

Digitised copies of permanent records will only be ingested if the original physical record no longer exists. Where a physical record has been digitised because it is at serious risk of significant deterioration or loss, or for any other reason (for example, when it is stored on unstable formats such as magnetic media or in letterpress books), the physical record is still considered to be the permanent record and should be transferred to QSA.

It's important to note that this distinction between born digital records and digitised copies does not impact how the content will be treated in terms of preservation and access. Both types of digital records are subject to similar preservation and access challenges.