Demystifying the New Migration API for SharePoint
Feature

Demystifying the New Migration API for SharePoint

7 minute read
Steven Pogrebivsky avatar
SAVED

One of the most anticipated presentations of this year’s Microsoft Ignite conference was the “Migration to SharePoint Online Best Practices and New API Investments” session. As surprising as it may seem, this is the first time Microsoft has released an API that is specifically aimed at on-boarding content into SharePoint Online and OneDrive for Business. Because my company began involvement with the new migration API in October 2014, I found the presentation especially interesting. 

I have to mention how unique this experience was for our development and senior technical team. As soon as we signed on, Simon Bourdages and a group of Microsoft developers were working hand-in-hand with us. While the MetaVis team worked on the packaging, delivery and a simplified U/X, Microsoft developers focused on the core API and infrastructure required to support the functionality. The collaboration continued non-stop, and within two months we had a functioning prototype. For anyone used to the old Microsoft, that is lightspeed.

As I sat through the Q&A at the end of the session, it was apparent that there were still real gaps in understanding the new API. My aim here is not to provide an in-depth technical overview of the PowerShell commands (you can find that online), but to answer the basic questions such as: why, how, is it really fast, and should I use it?

Why Is This API Important?

From an Office 365 customer standpoint, the most prominent reason for using the API will be speed. During his presentation, Bourdages alluded to a five time increase over traditional methods. Our testing had a wider range of results, which I will touch on a little later, but the bottom line is that under all circumstances the new migration API performed better than legacy methods.

There is another important, but less obvious, reason for Microsoft to build and promote the new API. The current approach used by all independent software vendors (ISVs) is to leverage CSOM and other legacy APIs. If implemented improperly, that method runs the risk of overwhelming SharePoint servers, making them unresponsive to standard users.

With SharePoint Online, such an event can easily bring down the service for many tenants, not just the one doing the migration (and, according to reports, this exact situation has already occurred). To counter this, Microsoft plans to begin throttling CSOM calls. The trickle-down effect is that migrations that use legacy approaches will become even slower than they are today.

How Does It Work?  

The first thing to understand about the new API is that it leverages Azure Storage. If you don’t already have access to one of these environments, you can register and purchase it using this link. When ordering storage, keep in mind that its location is also important (more on this later).

During his Ignite presentation, Bourdages mentioned that the API is primarily designed to read content from a file system or SharePoint, therefore the content must reside on and be accessible from one of these on-premises locations. Once identified, an API command will build “packages” from the content and XML files that define attributes, metadata and job parameters. Another API command will then upload these packages and XML files to your Azure Storage location. 

Once the content and XML files have been uploaded, the package is added to an Azure Migration Queue. At this point there is nothing left to do but wait. Behind the scenes, Microsoft tenancy bots scan the queues. When an available package is identified, the processing begins moving content, metadata and item-level permissions into a final destination in either SharePoint Online or OneDrive for Business. Upon completion of a package, detailed log files are pushed back into Azure for review by the user. 

2015-21-May-Pogrebivsky-Post.jpg

Is It Really Fast?

As mentioned above, the new API is definitely faster. In our tests we saw performance increases ranging from two to 10 times over standard techniques (no throttling was observed). We found that performance was a function of many variables, but several had a significant impact. 

Learning Opportunities

Size and quantity of files being migrated. It’s not surprising that migrating many smaller files takes longer than fewer larger files. Part of this is due to the duration of building packages with many smaller files. There is little that can be done about this, since file sizes are what they are.

Capacity and latency of Internet connection. Another less than surprising factor is that slower Internet connections will slow down the migration. As mentioned during its presentation, Microsoft does offer an option of saving all your packages onto a hard drive and shipping it to Microsoft’s data center.

Location of Azure Storage. Something less obvious is that you have the option of selecting the location of your Azure Storage. Choose one that is in the same location as your tenant. This will reduce the processing time of packages into SharePoint Online. You can identify the location of your tenant by emailing Microsoft’s support team.

Concurrent migrations into a single content database. This is one variable that you have very little control over. Currently the API is limited to processing a single package into any one content database at a time. In SharePoint Online and OneDrive for Business, content databases are auto-assigned for each site collection and OneDrive. It is quite possible that your entire tenancy resides on a single database. If this is the case, your packages will be processed one at a time.

(Update: Microsoft has extended support for up to eight concurrent packages per content database, depending on load.)

What Are Its Limitations?

It is probably easier to review the Migration API’s capabilities, rather than the limitations. The API will migrate content, metadata and item-level permissions. Microsoft has left the door open to adding other objects in the future, but currently this is all that is supported. 

During the Q&A session, some questions were raised about “exotic” objects such as web parts, workflows and InfoPath forms. Although I certainly sympathize, a more appropriate question would cover more mundane but critical objects such as sites, lists, content types and permissions. None of these are handled by the new API. They must exist prior to the migration via manual creation or by using third-party tools that are designed for this purpose.

Another common question is about performing incremental or delta migrations. This is also a basic feature of professional third-party migration tools, which allows users to continue using their on-premises environments until the end of the migration. An incremental migration then picks up the changes and reproduces them in the target. This functionality is also not part of the current API. 

Who Should Use It?

The Azure Migration API is available to everyone via PowerShell. However, during the presentation Bourdages indicated that the API is really intended for ISVs and IT administrators. I would go further and suggest that unless you have a very narrow scope, a third-party tool from a proven ISV will always be more economical and faster than doing migrations yourself. 

Despite any hype that you may hear, SharePoint migrations are not simple or easy. Good migration tools encompass years of experience with many SharePoint environments to avoid common and not-so-common difficulties. In addition, ISVs that have worked with the new API will leverage it for improved performance, but also build out the other objects required for a high-fidelity migration.

Regardless of how you decide to proceed, for those of you embarking on SharePoint Online or OneDrive for Business migrations, this API is good news. Your migrations will certainly happen faster and more efficiently. Joe Newell, who presented at Ignite along with Simon Bourdages, said something that a true migration professional can appreciate: “The most important part of a migration is finishing it!” This API will certainly help.

Creative Commons Creative Commons Attribution-Share Alike 2.0 Generic License Title image by  Rafael Edwards 

About the Author

Steven Pogrebivsky

Steve Pogrebivsky is an expert in information and content management systems with over 20 years of experience. He was a co-founder and CEO of MetaVis Technologies, which built tools for Office 365, SharePoint and other cloud-based content management systems. Connect with Steven Pogrebivsky: