Data Sets

Updated 2 days ago by Copado Solutions

Introduction

Data sets is a feature included in Copado Data Deploy that enables you to take a snapshot of the data stored in a certain environment and save it for use in the future or deploy it to other environments in your pipeline.

While you can easily move data across different environments with data templates, you may sometimes need to deploy a static set of data, something that is not subject to changes, as most of the data in your orgs can be. This is where data sets come into play. A data set can capture a specific piece of data at a certain point in time and save it in a CSV file that you can later deploy to other environments.

Data sets are based on a data template, therefore if you want to capture, for instance, CPQ data from your production org, you need to have a CPQ template that uses your production org credential and then select it in the data set, but we will get to that later.

Let’s take a look at some of the use cases for data sets.

Use Cases

  • Import data from external databases into a data set and deploy that to a Salesforce environment.
  • Generate a training environment with a specific set of data to test a feature.
  • Create a data snapshot from a Salesforce environment to use in other environments or tools (e.g., for analysis or reporting purposes).
  • Create different versions of your data at specific points in time so that you can revert the changes and go back to an older version if required.
  • Schedule regular data backups using a scheduled job with a data set deployment.

Considerations

  • Data sets are only supported in Lightning mode.
  • The Match Owners and Match Record Types options are not currently supported.
  • The option to migrate attachments is not available at the moment.

How to Create a Data Set

To build a data set, follow these steps:

  1. Navigate to the Data Sets tab and click on New.
  2. Give your data set a meaningful name and provide a description if desired.
  3. Select the data template that contains the data that will be used to create the data set in the Data Template lookup field. For more information about data templates and how to build a data template, check out the Data Templates article
  4. Click on Save.
  5. Once you have configured your data set, click on Generate Data Set to actually generate the data:

Generate data set button
Please make sure the org you have selected to generate the data set is authenticated, otherwise, you will receive an error.

Copado will generate two files and add them to the Files section of your data set:

  • A Lightning CSV file following the naming convention Filename: {Data Template Name}-{DataTemplateRecordId}.records.csv
  • A template file following the naming convention {Data Template Name}-{DataTemplateRecordId}.template. This template file is necessary for the correct functioning of the data set and cannot be deleted.

There will be a CSV file for each object. For example, if you have a template with accounts and contacts, the data set will include two CSV files, one for accounts and one for contacts. For more information about CSV files and their formatting, review the article How to Upload a CSV File from an External System to a Data Set.

You can find more information about file generation in the Last File Generation section of your data set:

Last File Generation section

A deployment job will be created for the file generation. Once it is completed, the FIle Execution Status field will be set to Completed Successfully. If there is a problem generating your data set, a Completed with Errors status will be displayed. You can click on the Execution Job link to navigate to the deployment job and troubleshoot any potential issues.

Data set files support file versioning. Therefore, if you need to generate new files for an existing data set, Copado will generate a new version of each file rather than create brand new files. Alternatively, you can upload a new version of your CSV file by clicking on Upload New Version in the Files section:

Upload New Version button

For more information about CSV files and their formatting, review the article How to Upload a CSV File from an External System to a Data Set.

To delete previous versions of the files, simply click on Remove File Versions:

Remove File Versions

If you have access to data sets, you can upload new file versions. However, to delete previous file versions, you need to be the owner of the file or have the Modify All Data permission.

Please note that you cannot delete a specific previous version of a file due to a Salesforce limitation. If you use the Remove File Versions button, Copado will remove all previous versions and just keep the latest one. Similarly, if you want to go back to a specific previous version of a file, you need to download the file for this version and upload it as the latest version.

There are some additional limitations regarding files you should be aware of. For more information about this, check out Salesforce’s article Salesforce CRM Content Limits.

Now that you have your data set, it is time to deploy it to other environments. For more information about how to do this, refer to the article Deploying a Data Set.


How did we do?