Data migration from Salesforce to SAP Hybris Cloud for Customer

This week marked the beginning of the next subproject in our SAP Hybris C4C introduction: migrating the data. Steffen already told you a bit about that in his recent blog article. Today, I am going to dive deeper into how exactly we did it.

Approach & data export from Salesforce.com

Salesforce.com (SFDC), our previous CRM system, was directly connected via an SAP Data Services Adapter. SAP Hybris Cloud for Customer has an oData interface for which SAP Data Services also provides an adapter. Our initial plan to load our SAP Hybris Cloud For Customer installation with data coming from salesforce.com was to use exactly this interface. The reason was that oData services make it easy to consistently query, update, add and delete data via http. And based on SAP Data Services this is incredibly simple and fast.

Unfortunately, however, it turned out that the oData approach for directly migrating the data to C4C wasn't meant for us: After reading out the Salesforce data using the Salesforce Adapter, the data could be processed and mapped to the oData Services provided by C4C, but we were unable to directly write it into C4C. We run a few analyzes to understand why that happened. The results showed that the problem occured to incompatible oData protocol versions between SAP Data Services and the C4C interface and that additional analyzes would have been required both by us and SAP Support to obtain a suitable software patch for the oData Adapter. This scenario would have cost us a lot of time, which was not included in our project schedule. Much to our regret, this forced us to scrap this very elegant approach.

It was time for plan B: We decided to fall back on the Excel-based migration templates that comes with SAP Hybris C4C. This meant exporting the data using the Salesforce Adapter, temporarily storing it in staging tables using Data Services, and then analyzing, cleaning, and preparing it before making it available as CSV files via flat file export.

Data cleansing and transformation using SAP Data Services

We needed homogenous and error-free data to guarantee the validity and credibility of the information we base our business decisions on. Data cleansing as a means to ensure unique and consistent data records in our target SAP Hybris C4C system was one of the major milestones in this project.

And once again, we achieved this with SAP Data Services:

• Duplicates were automatically identified and consolidated.

• We relied on external validation services for our address cleansing and had the addresses validated and, if necessary, automatically completed.

• Our business data in particular was enhanced with more detailed information such as geodata.

This allowed us to really take the quality of our data to the next level and, in turn, minimize the need for manual touch-up before importing the data into C4C.

In the next step, we mapped and transformed the data with SAP Data Services. Data transformation is inevitable because source and target system almost never use the same data model. That’s why you start by mapping the data and use the findings to define transformation rules in order to make the right adjustments before importing the data. For more information on how mapping works you can refer to Steffen's recent blog article. During the import, the transformation rules are automatically enforced by the software. For a more detailed explanation: We created our Excel-based C4C migration templates, which describe the data structure for the C4C import, as flat file formats in Data Services, allowing the tool to directly map the data delivered by the Salesforce Adapter to the respective flat file structure. We used what is called lookup tables to map the code.

Data import into SAP Hybris C4C

The data migration was completed by importing the data into the target system, in our case SAP Hybris Cloud for Customer (C4C). Based on a multi-level approach we started by loading basic objects such as customers, accounts, campaigns, etc. In a second step, we then loaded any linked object such as contacts, leads, and activities.

For each object we created in C4C, we also stored its Salesforce ID to allow us to correlate the newly assigned C4C ID with the original Salesforce ID. For this purpose, we created the corresponding tables in Data Services to allow for a lookup from the Salesforce ID to the newly assigned C4C ID. We used Data Services to populate these tables either via CSV export from C4C or directly via OData from C4C.

In the last step, we migrated any archived objects such leads or activities. This allowed us to retain the complete activity history (phone, email, visits/activities) including any links to accounts and contacts. In total, this included 33,869 email activities, 37,121 phone activities, and about 48,000 other activities.

This, however, required us to take a bit of a "detour": First, we exported the data from Salesforce into a CSV file. Then we had to tediously prepare it in Data Services to ensure that C4C would be able to correctly interpret, for example, the line breaks in a description after the data import. Without these preparations, it would have been impossible to import the data because said line breaks would have completely shifted the entire CSV structure and the data would have been matched to the wrong fields.

So here is the status quo today: Our high-quality data is now in C4C. Next up, we will start testing the new system with a small group of pilot users before we can hopefully go live soon.