Supporting the City of Surrey to move 20 million documents from Laserfiche to Microsoft 365
Image credit: surrey.ca
About the City of Surrey
The City of Surrey is a municipality in British Columbia, Canada. It is the largest city by land area and second most populated in Metro Vancouver.
The City employs over 4,689 staff as of 2023.
Technologies
SharePoint Online
Power Platform
Azure Function
Azure Cloud Services
ShareGate for migration
Foxit PDF Maestro Server for combining and converting files and OCR
The business challenge
The City of Surrey is one of the fastest growing municipalities in BC.
Over the years, digital records for Engineering & Planning (Projects), Planning and Development (Buildings), and Finance (Homeowner Grants) grew to about 6 terabytes of information and millions of documents.
These files were stored in a Laserfiche repository, and only accessible by designated city staff. The goal of this project was to move these files into Microsoft 365 for a modern user experience.
In addition to the growth in records volume, these files were not easily searchable, except by limited metadata. A key goal of moving these files into Microsoft 365 is to create a robust search experience that supports searching within the documents themselves to aid in finding the right content.
Solution highlights
Migration planning
The electronic records the City wanted to migrate were stored as individual TIFF files in a Laserfiche repository.
The City engaged Gravity Union to help migrate the following groups of documents:
Homeowner grant information
Planning and engineering building records including inspections, drawings, permits, applications, and legal documents
Engineering projects records including correspondence and design files
Gravity Union first completed an initial assessment to evaluate the content, its use, and the volumes that need to be migrated. Gravity Union collaborated with the City on a vision for the migration and aligned on detailed requirements and high-level architecture for the new solution.
High-level architecture and process flow for the new solution
Design planning
Information architecture and metadata mapping was built into early prototypes to decide how to apply metadata to document sets in SharePoint.
The solution architecture required careful planning to accommodate the large existing volume of content along with continuing growth in the City.
Prototypes helped stakeholders visualize the solution and agree on how activities such as content holds and the process for uploading of new files would work.
A complex migration project
To make the files searchable (including the contents of the file), the migration process included clean up and translation steps. These steps convert the individual TIFF files in Laserfiche into combined and searchable PDFs.
Moving large volumes of data from on-premises to the cloud has complexities. As part of the migration, Gravity Union moved files to Azure storage to run functions that process the files and used the application, Foxit Maestro Server, to convert the TIFF files to PDFs with OCR. The effort required included steps to resolve issues with data integrity and quality so that the files are setup for use in Microsoft 365.
Example of processing steps run from code
Processed files in a SharePoint document library
Search as the front door
Gravity Union used a modern PnP (Patterns and Practices) search solution to enable a better search experience for the millions of documents.
The search landing page is the front-end experience for users. It is based on configuring the modern PnP search webparts to search by key metadata such as project number, portfolio ID as well as building address for building-related content.
The search results show the relevant matches to a query and a preview of the PDF document for quick verification, printing, downloading, or emailing.
Search result with preview of PDF document
Search filters
“The new solution significantly enhances property information access, boosting staff efficiency in application reviews and expediting building permit issuance within our community. The intuitive modern user interface facilitated a seamless transition for our staff, requiring minimal training.”
— Manager of Business Transformation, Planning and Development, City of Surrey
Content holds using a Power Automate solution
The City did not have access to full eDiscovery in M365 yet, so a simple process to identify content for holds was implemented for the short-term.
Gravity Union enabled a simple content holds process with Power Automate for building and project documents. A field is checked in the document library when a set of files is subject to a hold, and this initiates a set of processes run by Power Automate including updating the security settings.
When a user views documents under a content hold, they see a message informing them of the hold and directions of who to contact.
Behind the scenes, security settings are updated so that only the file registry team has access to edit the file and metadata. They can also take the necessary steps to process the hold. A SharePoint list is updated automatically to display the files subject to a hold.
SharePoint List displaying files that are subject to a legal hold
Benefits and Results
Process efficiency and fewer errors — Reduced the number of manual steps in processing documents using Power Automate and Azure Functions
Productivity improvement with searchable documents — By moving from TIFF files to PDFs and enabling OCR, documents are searchable, and it is easier for City staff to find information they are looking for
Reliability in content — Content and metadata is cleaned up resulting in a more authoritative source of content
“It has been an amazing experience working with Gravity Union. They pushed us to think differently about how we work and what could be possible. Their responsiveness, leadership, technical expertise, and unwavering commitment were instrumental in successfully executing this intricate migration.”
— Program/Project Manager, City of Surrey
Norfolk County
We worked with an Ontario municipality to create a multi-year Records Management strategy and roadmap that considers investments in governance, scanning and digitization.
College of Surgeons and Physicians of Manitoba (CPSM)
See how we helped the College move towards paperless collaboration on SharePoint Online and Teams.
City of West Kelowna
The City of West Kelowna moves to Microsoft 365 as part of a broader paperless initiative.