Upgrading Grouparoo from v0.6 to v0.7

Grouparoo v0.7 is our biggest release yet! Learn how to upgrade so you can import and export multiple Models.

As always, begin by updating the packages within your Grouparoo project by following the generic upgrade guide to bump your dependencies to v0.7.x.

An example of a Grouparoo deployment being upgraded can be found here.

Models and Records

The v0.7 release is all about preparing Grouparoo to be able to import multiple Models. A Model defines the shape of your Records. Each Model has its own Sources, Records, and Properties. Together, this allows you to do things like create different types of Records such as Leads, Customers, Users, and Visitors that contain different Properties. In the future, Grouparoo will be adding the ability to handle other types of Models than Profiles - including Accounts, Events, and Custom Objects.

The first and biggest change is that what used to be called Profiles is now called Records. You'll see this change throughout the Grouparoo UI, and you will need to change all instances of profile, profileId, and confirmProfiles in your config files to record, recordId, and confirmRecords respectively.

Next, you will need to generate your first Model. If you were using Grouparoo v0.6 or earlier, you essentially had a single Model with the following attributes:

{
  "class": "Model",
  "id": "users",
  "name": "Users",
  "type": "profile"
}

Adding the above to ./config/models/users.json (be sure to mkdir ./config/models as needed) will add a new Users model to your Grouparoo project. You can also generate a new model with the the grouparoo cli, grouparoo generate model Users.

Note the id, users as you will need to add a modelId to a few objects to let them know which model they should be using:

• Sources
• Destinations
• Groups
• Sample Records (previously called Sample Profiles)

For example, a Source with a new modelId might look like:

{
  "class": "Source",
  "id": "demo_users",
  "name": "Product Users",
  "type": "postgres-import-table",
  "appId": "demo_db",
  "modelId": "users", // <-- Note the new modelId
  "options": {
    "table": "users"
  },
  "mapping": {
    "id": "user_id"
  }
}

Sample Records in your code config have also changed class from Profile to Record:

[
  {
    "id": "pro_c83846e5-aebe-4981-aa0d-3b1d5b71157d",
    "class": "Record", // <-- Note the new class name
    "modelId": "users", // <-- Note the new modelId
    "properties": {
      "user_id": [10]
    }
  }
]

Migration Notes

Code Config:

To migrate an existing Grouparoo Deployment from v0.6 to v0.7, and you have been using code config, you will need to check in your new code config model before the migration will succeed. This is required as Grouparoo needs to know the id of the Model to apply to your existing records. Otherwise, an error like error: Make a single Model config file will prevent Grouparoo from running.

Grouparoo Enterprise:

If you have been using Grouparoo Enterprise, a new Model will be created for you and applied to your existing Records. You can then rename or adjust this model in the Grouparoo Enterprise UI.

Plugin Names Changed and Clarified

To prepare Grouparoo for exporting multiple types of models, the names of our Plugin Connections have changed. It is very likely that you will need to update the "type" property of your Destinations and Sources. Plugin connections now follow the pattern {type}-{direction}-{model}. For Destinations, the name of the Model is the name the Destination uses (usually something like "contact", "profile", or "user").

Examples of changes include:

# old name             -> new name

# Sources (incomplete list of examples)
bigquery-table-import  -> bigquery-import-table
bigquery-query-import  -> bigquery-import-query
snowflake-table-import -> snowflake-import-table
snowflake-query-import -> snowflake-import-query

# Destinations (incomplete list of examples)
braze-export           -> braze-export-users
hubspot-export         -> hubspot-export-contacts
mixpanel-export        -> mixpanel-export-profiles

The easiest way to see the new names of the Plugin Connections provided by the Grouparoo plugins you have installed is to either use the grouparoo configure command or view the list in the CLI via grouparoo generate --list

Destinations can Export all Records of a Certain Model

Now that Grouparoo can support multiple Models, we can use that distinction to choose which Records to export. Prior to v0.7, Grouparoo Destinations required a Group to know which Records to Export. Now, you can choose to export all members of a Model.

To support this, Destinations now have the property collection required in their configuration. destination.collection can be group, model, or none and is required. Updating a Destination looks like:

{
  "class": "Destination",
  "id": "newsletter",
  "name": "Newsletter",
  "type": "mailchimp-export",
  "appId": "mailchimp",
  "modelId": "users", // --> Note the new `modelId`
  "collection": "group", // --> Note the new `collection`
  "groupId": "all_emails",
  "syncMode": "additive",
  "options": {
    "listId": "MAILCHIMP_LIST_ID"
  },
  "mapping": {
    "FNAME": "first_name",
    "LNAME": "last_name",
    "email_address": "email",
    "language": "language"
  },
  "destinationGroupMemberships": {
    "High Value Spanish Speakers": "high_value_spanish_speakers"
  }
}

Grouparoo File System Removed

Version 0.7 removes the Grouparoo File System which was previously used to upload and download files by the Grouparoo application, usually in S3. This has been removed to simplify the application, as it was an underutilized feature. Because of this, it is no longer possible to export members of your Groups to CSV.

Remove the following plugins if you have them installed, as they no longer are maintained and not released for versions v0.7+ of Grouparoo:

• @grouparoo/files-local
• @grouparoo/files-s3

CSV Plugin Simplification

With the removal of the Grouparoo file system, there is now only 1 type of CSV import plugin, named csv-import-table which accepts a URL of a remotely-available CSV file to import. The CSV source that read a CSV file uploaded to Grouparoo has been removed.

Salesforce Plugin Key Names

To match Grouparoo's new terminology, the @grouparoo/salesforce plugin has been updated, and config changes are required in any config files with a Salesforce Destination:

# Old Key Name             -> New Key Name

profileObject              -> recordObject
profileMatchField          -> recordMatchField
membershipProfileField     -> membershipRecordField
profileReferenceField      -> recordReferenceField
profileReferenceObject     -> recordReferenceObject
profileReferenceMatchField -> recordReferenceMatchField

Invalid Properties

Grouparoo now notes if a Record Property that you are importing is invalid for the type of the property. This might mean that Grouparoo encountered a duplicate value for a unique property, or that an email-type property doesn't contain an "@". In previous versions of Grouparoo, this would have caused an exception. Now, Grouparoo will set the Record Property's value to null and list a reason. Note that a property becoming null may change the Groups that the Record is a member of.

You can choose to list records with invalid properties as well.

Destination Deletion

Prior to Grouparoo v0.7, when a Destination was deleted either via by removing its code config file or in the enterprise UI, the Grouparoo application would attempt one final export for the Records that were previously sent to the Destination. This behavior has been removed, and now deleting a Destination will simply delete it from Grouparoo and not effect the previously exported Records in the Destination.

The old behavior may have lead to unintended removal of Records from your Destinations in certain SyncModes. If you do want to remove the previously exported Records from a destination, choose to sync a collection of "none" in combination with SyncMode "sync" after previously exporting Records.

Record Snapshot Tests

To match Grouparoo's new terminology, the jest testing helper previously titled getProfile() has been renamed getRecord(). You will need to update your test code to use the new terminology and re-record your test snapshots.

Bootstrapped Property

The option for creating a "Bootstrapped Property" as part of defining a Source has been removed. This option's functionality was deprecated in a previous release.