Related sites:
Newsletter: Perspectives on Power Platform
Company: Niiranen Advisory Oy
If you mostly work with configuring the Microsoft Dynamics CRM solutions via the customization UI with its graphical point & click tools instead of writing custom code in Visual Studio, you may not keep such a close eye on the changes introduced in the CRM SDK. After all, the term SDK stands for “software development kit” and you may not consider yourself as the target audience if you don’t actually develop any new software that would interface with Dynamics CRM.
The What’s New page in the CRM SDK documentation is a place that would be very useful for also non-developers to have a look at whenever updates to the product are released. Apart from the end user guidance material published on CRM Customer Center, the SDK is pretty much the official place where changes to the platform functionality are introduced. With the latest CRM 2013 Service Pack 1 release (a.k.a Leo) now available, there’s a long list of new and improved features to review. One new addition that didn’t receive a mention on that page, though, is the Configuration Migration Tool, which is a very handy tool if you need to transfer metadata entities across different CRM organizations.
What Are “Metadata Entities”?
In theory the CRM solution files should contain all of the configuration data about your CRM organization. This is what is often referred to as metadata, which could be described as “data about data” in short. As an example, the list of possible values for the Relationship Type attribute on your account record would be considered metadata, whereas the actual account records referring to one of these values would be “real” data stored in the business entities. To illustrate the difference, a blank CRM organization containing only your customizations would not contain any records for the business entities (accounts, contacts, opportunities), but it would contain all of the records in the metadata entities that would be necessary for executing the customized business processes, since otherwise the configuration of that environment would be incomplete.
In practice there are often valid reasons for why it makes more sense to store this type of metadata in a custom entity of its own, instead of using option sets. This provides a whole different level of flexibility in terms of what type of metadata and how much information you can store. Maintaining the list of values can be done without the need to update the customizations. Filtering available records to the end user can be done via standard security roles and BU’s. Building dependent lists on the form can be achieved via the filtered lookups feature. You can leverage the new Quick View Forms to show data from related parental records. And so on.
One problem with using these type of metadata entities is that the data cannot be packaged into a CRM solution file. While you can easily export any changes to the standard customization components and move them to a different CRM organization, getting the metadata entity records from one org to another requires a bit more effort. As a result, keeping the different development, test and production environments in sync requires specific attention to be paid on these metadata during the release process.
Introducing the Configuration Migration Tool
The CRM 2013 Service Pack 1 release includes the new Unified Service Desk (USD) application that relies heavily on storing its configuration data into these metadata entities. In order to support the systematic deployment of USD solutions configured for the customer’s environment, Microsoft has introduced new tools in the version 6.1 (a.k.a. CRM 2013 SP1) SDK. These are the Configuration Migration Tool and the Package Deployer tool. While the latter one is surely useful in deploying larger ISV solutions, I believe it’s the Configuration Migration Tool that can benefit a far wider audience. Also, since you don’t need to run Visual Studio to work with the tool, I’m personally more comfortable discussing its features in my blog as a non-developer who generally steers clear from anything that isn’t close to point & click.
In short, what the Configuration Migration Tool (from here on referred to as “CMT”) allows you to do is 1) choose the entities that you want to migrate data from, 2) export all the records from those entities as a .zip file and 3) import it into a different CRM organization. There’s a great illustration available in the SDK article describing CMT, presented below:
“Awesome, now we have a built-in tool for CRM data migrations!” Well, actually, no. While CMT can move a set of records from several entities in one package and maintain their relationships, it has not been designed for the purpose of business data migration. For example, you cannot filter out any of the records to be transferred from the source organization, nor are there any guarantees for the tool’s performance when moving very large sets of data. Here’s what the Dynamics CRM product team has to say about the tool:
Check out this CMT feature overview video on YouTube for more information on the above points.
Moving Configuration Data with CMT
Let’s see how the tool works in practice. To test the features, I created two custom entities for storing my configuration data: “Category” and “Subcategory”. A category can have many subcategories (1:N relationship), but in order to make the experiment more interesting, I also added a lookup on the category form that allows selecting the default subcategory to be used. So, we have both a 1:N and N:1 relationship between our entities, as shown on the Category form below. For the purpose of testing I added category A with subcategories 1*, 2, 3 and 4, then category B with subcategories 1 and 2*, with the asterisk determining the chosen default subcategory (notice the duplicate names in the subcategory data, we’ll get to that later on).
After launching CMT, choosing to define the schema (step 1) and entering the connection information to log into my own development CRM 2013 server via IFD (hint: enter the server URL with the org name as “org.domain.com” and the user name as “[email protected], no other connection option worked for me), we’re presented with a screen for choosing the components to be included in the schema. We can pick from entities included in a specific solution. After selecting the entity, we can either choose which fields we want to transfer data from or just be lazy and add the whole entity into our schema definition. Save and Export will give us an XML file with the contents from the included entities, fields and relationships.
The next step is the data export. Here we’ll choose the schema XML we just created and a target file for the data export. Hitting the Export button will start the process, report the progress for each entity in the CMT app window and finally produce a .zip file containing the data for all the components defined in the schema file.
The last step is to connect to a different organization that already has the same schema. So, if you’ve not yet transferred your solution files into the target organization, now’s the time to do that, since CMT will not modify the organization schema, but rather just assume that it is identical to the source organization as far as the selected entities and fields go.
After you’re sure that the source and target schema are the same, start the import process from the .zip file. The CMT window will again show the details of the process and inform you about the number of records imported.
Import process completed! Now we have two CRM orgs with identical data in our chosen configuration entities and fields.
Reviewing the Results
In the CMT UI it looked like all of the data was imported with no errors. When inspecting the results from the target organization, though, something wasn’t quite right. Even though the CMT log showed that 6 Subcategory records had been imported, all I had in my target system was 4 records:
On the Category record form, it seemed like the reference to the default subcategory had been populated (as a result of the “second pass updates” shown in the import process log window above), but the references were actually incorrect. The subcategory value for Category A was actually the Subcategory 1 record that was a child record of Category B in the source system. Hmm, what’s going on here?
CMT provides access to the log file of the latest import job conveniently from the top toolbar, so I opened up the logs and went to review the entries. I found two rows about the subcategory entity, saying “Switching Insert to Update for Subcategory, jkn_name = 1, ID: 590a3faa-5dee-e311-9414-00155d569f46”. So, it appears that because subcategory records with primary field values 1 and 2 had already been inserted into the target system, CMT did not create the last two records but instead updated the contents of the earlier records, thus creating the mismatches discovered.
Had I taken the unconventional RTFM route and actually studied the SDK documentation before the import, I would have already known that this is actually the default behavior of CMT. Unless you specify a uniqueness condition during the schema creation stage, CMT uses the primary field name of the entity to compare against the existing data on the target system. To change this, you should click Tools – Configure Import Settings and then select the fields that determine a unique record in your source data. OK, that’s definitely a good feature to be aware of!
Since there was now a mismatch between the configuration data in different environments, this was a good opportunity to try out the capabilities of CMT in maintaining changes in such data. To make the sample data a bit easier to match, I decided to update the subcategory names (primary fields) with the parent category prefix, such as A1, B2 and so on. This way there would be no duplicate values to be found, even without touching the uniqueness condition.
I then started a new export with the existing schema and created a new .zip file. Then I ran the import process to the target organization again. The question in my mind during this operation was “I wonder what the end result will look like after this”. It turns out that the export-import process was able to correctly update the existing records, create the missing subcategories and link them to one another with no errors. Configuration successfully migrated!
One nice feature of the Configuration Migration Tool is that not only will it carry over the visible record fields with identical content, but it can also preserve the record GUID’s across CRM organizations. This means that any solution components with references to specific GUID values (such as workflow processes) should also transfer from one environment to another without requiring manual reconfiguration tasks. That’s how CMT know how to update subcategory “1” to “A1”, for example, without creating a brand new record. The only configuration changes that would need manual intervention would be the deletion of configuration records from the target system, as this falls outside of the scope of functionality in CMT.
If you haven’t previously explored the contents of the Dynamics CRM SDK but want to try out the Configuration Migration Tool, then just download the SDK, extract it and navigate to the Tools folder to find the CMT application files.
Hi Jukka,
Heard about this but haven’t had the opportunity to use it yet.
Did you happen to notice how it handled Created On, Created By, Modified On and Modified By? I’m assuming it can’t do any better than other import tools (it would be nice if we could preserve all four…) but I wondered if it preserved the Created On? (The CRM instance adaptor seems to do this by default).
Also I wonder if it can preserve GUIDs in configuration data that you can’t delete/re-create (for example the Team associated with the default Business Unit or CRM Online Users) – the instance adaptor doesn’t seem to be able to handle these.
For CRM Online I’m currently trying to use the new sandbox minimal copy features to do this. They still don’t seem to be working properly so wondering if this might be an acceptable alternative…
Simon, the Created On and Modified On dates were not preserved by Configuration Migration Tool. I’d imagine this also falls within the “non goals” of this tool, as it isn’t designed for business data migration. The Instance Adapter probably overrides the Created On date via the supported API methods and stores the physical creation date to Record Created On field?
I ran a test between 2 on-premises organizations with a schema file consisting of business units, teams and users. The results were quite interesting, as the CMT import progress showed all green (apart from one yellow alerts for the INTEGRATION and SYSTEM users that were skipped), but no new records whatsoever were created in the target organization at any point. The records were simply queued for second pass update, but eventually none of them found their way into CRM.
Jukka, I have tried to connect (Import) to CRM Online trial today and it will not connect with no error. I have tried both O365 and Online options but the same result. Have you managed a successful connection Online?
Craig, I tested this with a new CRM Online trial org (EMEA) and the connection seems to work just fine for me.
Thanks Jukka. PS. I think your blogs are top notch, keep up the great work 🙂
[…] Configuration Migration Tool: Working Smarter with Metadata Entities […]
[…] Configuration Migration Tool: Working Smarter with Metadata Entities […]
[…] Configuration Migration Tool: Working Smarter with Metadata Entities […]
This tool is terrific. Thanks for tackling the deep dive! Extremely helpful. Readers may note that the speed of export and import is significantly slower given that the GUIDs are preserved and so the SQL performance is not as high as when allowing the system to create the GUIDs.
Great Work!
Joe
I had assumed (wrongly I think) that the new sandbox copy features in CRM Online were somehow based on this tool behind the scenes. That doesn’t seem to be the case – using the sandbox copy features we appear to be able to preserve Created On, Created By, Modified On and Modified By as well as (most of) the GUIDs.
There are a few anomalies however:
Minimal Copy:
• GUID and Name of Default Business Unit is not preserved
• Version Number attribute is not preserved on entities I’ve looked at (I’m not sure if this is correct or not)
• Default Duplicate Detection rules are not copied correctly (owner etc.)
• Default System Jobs are not copied or created at all
• Not clear which entities are copied and which are not.
Full Copy:
• GUID and Name of Default Business Unit is not preserved
• Version Number attribute is not preserved on entities I’ve looked at (I’m not sure if this is correct or not)
• Default Duplicate Detection rules are not copied correctly (owner etc.)
• Some (but not all?) workflows that were “waiting” now appear “cancelled” in the System Jobs grid but “waiting” if you open them (possibly something to do with still being in Admin mode for the moment?)
I’ve not had time to investigate further – hoping some of them will be fixed/better documented in due course.
Thanks for reporting your findings, Simon. I haven’t yet had a chance to try the CRM Online sandbox copy process, but as you mentioned, it is indeed a separate mechanism from the Configuration Migration Tool. The latter works for all deployment models, but sandbox management is specific to Microsoft hosted cloud organizations. Whereas in an on-premises environment you could copy your production CRM database to create a development sandbox, in CRM Online you would have to use the new sandbox management tools provided in the Spring ’14 release.
For those who are interested in the CRM Online sandbox related features, here’s a few articles from MSDN: Manage CRM Online sandbox instances and Copy an instance. For a quick overview of what the sandboxes mean in practice, you can watch this video on YouTube or read this blog post from Inogic.
One thing you need to be aware of is that all the sandbox environments count towards your CRM Online subscription’s storage quota. Never perform a Full Copy of your production instance if you’re close to 40% utilization rate of your current storage. This is because a Full Copy would push you beyond the 80% threshold, after which certain functionality in CRM Online may not be available to you. For more information on the implications of reaching the storage limit, read this article from the CRM Team blog.
[…] To get started on how to use the tool, I’ve posted a simple post on how to use the tool here or more detailed one from Jukka here. […]
When I try to migrate Email Attachments
DataMigrationUtility.Import Information 8 ** BEGIN IMPORT PROCESS **
DataMigrationUtility.Import Information 8 ** Log Start: 12/04/2014 00:59:00
DataMigrationUtility.Import Information 8 Parsing import data file
DataMigrationUtility.Import Information 8 Parse Complete
DataMigrationUtility.Import Information 8 Beginning Import To CRM.. Importing 1 entities – 1 records
DataMigrationUtility.Import Information 8 Running Schema Validation
DataMigrationUtility.Import Information 8 Validating 1 Entities
DataMigrationUtility.Import Information 8 Schema Validation Completed
DataMigrationUtility.Import Information 8 Schema Validation Complete
DataMigrationUtility.Import Information 8 Processing Entity Data
DataMigrationUtility.Import Information 8 Processing Entity: Email Attachment
DataMigrationUtility.Import Information 8 Processing Entity Email Attachment, 1 Records in this entity
DataMigrationUtility.Import Information 8 Processing Entity Email Attachment, 1 Records in this entity
DataMigrationUtility.Import Information 8 Beginning Import of entity
DataMigrationUtility.Import Information 8 Pre-Processing Entity Email Attachment
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field attachmentid () – (action:Update) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field attachmentid () – (action:Update) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field objectid () – (action:Update) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field objectid () – (action:Update) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field owninguser () – (action:Skip) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field owningbusinessunit () – (action:Skip) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field activityid () – (action:Update) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Warning 4 Adding activitymimeattachment to the reprocess List as field activityid () – (action:Update) failed validation for reason code: LookupRecordNotAvailable
DataMigrationUtility.Import Verbose 16 GetFieldsForThisRow duration = 00:00:00.0009866
DataMigrationUtility.Import Verbose 16 Restricted Record Type Check= False
DataMigrationUtility.Import Information 8 Processing Email Attachment (threaded), Imported 0 of 1
DataMigrationUtility.Import Information 8 Connection 1 has imported 0 of 0 : 0 – THREAD COMPLETE
DataMigrationUtility.Import Information 8 Connection 1 has imported 0 of 0 : 0 – THREAD COMPLETE
DataMigrationUtility.Import Information 8 Connection 1 has imported 0 of 0 : 0 – THREAD COMPLETE
DataMigrationUtility.Import Information 8 Connection 1 has imported 0 of 0 : 0 – THREAD COMPLETE
DataMigrationUtility.Import Verbose 16 Result of Check and Retrieve Record Id = 00000000-0000-0000-0000-000000000000
DataMigrationUtility.Import Error 2 Insert to activitymimeattachment FAILED with Error : Email
DataMigrationUtility.Import Information 8 Connection 1 has imported 1 of 1 : 1 – THREAD COMPLETE
DataMigrationUtility.Import Information 8 Processing Email Attachment (threaded), Imported 1 of 1
DataMigrationUtility.Import Information 8 Entity Email Attachment, Completed Importing 0 of 1
DataMigrationUtility.Import Information 8 Stage Complete
DataMigrationUtility.Import Information 8 Import Process completed
DataMigrationUtility.Import Information 8 Import Process completed – Import duration: 00:00:05.9836505 – RPS:.17
DataMigrationUtility.Import Information 8 *********************************************************
DataMigrationUtility.Import Information 8 ** EXIT IMPORT PROCESS COMPLETE **
DataMigrationUtility.Import Information 8 *********************************************************
DataMigrationUtility Information 8 The import to CRM is complete. Duration of import: 00:00:05.9854031
Tano, I’m not sure if the email attachment is a supported data type for the Configuration Migration Tool. As the name suggests, it’s primarily been designed for moving configuration data between environments, not actual business data like activities. I recommend you to have a look at tools designed for actual data migration tasks, such as Kingswaysoft SSIS Integration Toolkit for Microsoft Dynamics CRM or Scribe.
Hello Jukka,
So, what you mean about CMT and matching is that we should configure the mapping before the schema creation? Because of that, the matching will be done using the primary field instead of GUID? That’s right?
Thanks for clarifying that.
Regards,
Oscar, that’s correct. You’ll need to define the matching criteria before generating the schema file.
I would imagine a not uncommon use-case would be transferring product catalogs between instances but the the dependency of product on target uom and uomschedule seems to preclude this unless there is any trick available? Thanks.
Thanks jukka
Another great blog Jukka. If anyone is wondering I just confirmed through testing that the tool also works for CRM 2011! 🙂
However I was attempting to migrate the setup data for CRM Goals (Goal Metrics, Rollup Fields & Rollup Queries) and ran into an issue. It seems that the CMT doesn’t allow you to select Rollup Fields as part of the schema to export. Any Ideas on how to hack it to do so?
Thanks!
Hi ,
I have a requirement to sync two organizations . Will this tool be useful in that case?
I tried exporting and importing couple of entities and i never get the complete record imported . Keep getting warning on second pass updates and it never happens. Can you please help me out here . Thanks 🙂
Hi, it seems that hirachies can not be exported and imported this way. We want to copy our whole bunch of product families, bundles and products from our test environment to the live environment, but the import only adds the items that are on top in the hiarchies. i.e. bundles which are childs of product families are not imported. How to fix that?
DoesNoteMatter, from what I’ve understood, Configuration Migration Tool should support this scenario of moving product catalog data across deployments. Are you sure you have selected all the required entities listed in this TechNet article: “Manage product catalog configuration“?
For Teams you need to replace the Business Unit information in the xml with the target information of the root/respective business unit. Also remove the tag about Queue in the xml. After importing the queue is created automatically. Or separately the queues can be migrated as well
Hi Jukka Niiranen,
what if, on the target instance we don’t have some entities but the source instance from where we have generated zip file using CMT have those entities, will the process stuck in between or will CMT allow to import the entities data which are available for target instance by leaving those entities which are not available on target instance.
Thanks
Popeye, I haven’t tested such a scenario. Since you are able to select the entities you want to export, I’d find it a bit strange to attempt a migration of data that cannot possibly be imported into the target instance. Even if part of the data would get properly migrated, it still sounds like an unnecessary risk to take. At least I would rather not have any error messages shown at the end of an import process, as it’s usually quite important to have the certainty that the data you attempted to migrate was pushed to the target instance fully without any unwanted alterations.
Great article. Do you know if there’s a command line option? We’re working with USD and need to suck out the config data for source control and builds, in a format that the PackageDeployer will understand.
John, I haven’t come across such an option in CMT myself. Have you looked into Adxstudio ALM Toolkit to see if it would meet this requirement? It’s been made available for free after Microsoft acquired Adxstudio so it could almost be considered as part of the CRM SDK now.
Haven’t looked at the Adx Toolkit before. Don’t think it will satisfy this though as we need the data to come out in a format that will be understood by the Package Deployer. We’ve got around it at the moment by having some Powershell that, amongst other things, pops the Configuration Migration Tool in front of the developer on a Clean of the USD Visual Studio project. It works for us at the moment but if we could command line it then there’s less clicks for the developers to go through (or forget to do!).
Will certainly look deeper into the Adx Toolkit though, There’s some stuff there that would be useful for other things that we’re either doing already or need to do.
The Adxstudio ALM Toolkit is not supported anymore after the Microsoft acquisition, it’s abandonware. Do not make any new or further invests in it.
Is there a best practice guide for selecting entities to export? The lazy option would be to select all the entities and export all of them, but this is obviously not ideal. How can I determine which entities contain metadata that either I or a team member have modified? Forgive my question if it’s naive–I’m an administrator-by-accident of a new (and tiny) CRM instance.
Jim, I don’t think there’s a way to select “only entities with customizations” in the standard UI. To identify where customizations have been made, it might be good to start with an export like Metadata Document Generator in XrmToolBox to get a list of each entity’s components and then view the resulting Excel file for rows with customized fields. If you need to go even deeper in the metadata analysis, give Snapshot a try.
Great Article Jukka.
Useful Information
Great Article Jukka, and valuable information.