Configuration Files
Last Updated: 2022-02-09Grouparoo projects contain configuration files for Apps, Models, Sources, Properties, Groups, Destinations, and any other objects generated in the course of configuration. These config files can be created and updated with UI config.
Config files are generated by the UI as JSON, but can also be converted to JavaScript to manipulate data with more flexibility or load configuration from external sources.
JSON files are parsed with JSON5
, which enables you to use comments and multi-line strings.
For examples of config files, you can browse app-example-config on Github.
Config Files Location
Config files are saved at the root of your Grouparoo project directory, in a directory called config
. A typical Grouparoo project structure looks like this:
my-grouparoo-project/
├── package.json
├── package-lock.json
├── .env
└── config/
└── ... (config files go here)
You can optionally choose to use another directory for your configuration by setting the GROUPAROO_CONFIG_DIR
environment variable, for example GROUPAROO_CONFIG_DIR=/path/to/config
.
Common Config Attributes
Every type of object has its own definition but all of them require these attributes:
id
: (required) A unique identifier for this object.- Lowercase letters, numbers, and underscores only
- No spaces
- Fewer than 38 characters
- Must be unique among other objects of the same type
class
: (required) The type of object that is being defined such asApp
,Model
,Source
,Property
,Group
,Destination
name
: (required) The display name of this object
Apps
The App configures the connections available to create Sources and Destinations.
id
: (required) Unique identifier for this app config.name
: (required) A display name for the app.class
: (required) This must be set to"App"
.type
: (required) The type of the app defined by a plugin. While the type can match the name of the plugin, plugins may offer multiple apps and the names may not match.refresh
: App refresh query. See below for details.options
: The App configuration options. These vary depending on the type of app.
Example:
{
"class": "App",
"id": "data_warehouse",
"name": "data_warehouse",
"type": "postgres",
// App Refresh Query
"refresh": {
"query": "..."
"recurringFrequency": 6000
}
// Option vary depending on the type of app
// Below is an example of the options for the Postgres app
"options": {
"host": "localhost",
"port": 5432,
"database": "...",
"schema": "public",
"user": "...",
"password": "...",
"ssl": false,
}
}
App Refresh Query Configuration
App Refresh Query enables Grouparoo to check your data warehouse for new data, and run all Schedules for that App if it finds any. The following Plugins support App Refresh Query:
- BigQuery
- ClickHouse
- MySQL
- Postgres
- Redshift
- Snowflake
- SQLite
For the best results, configure your query to return a single row of data. For example, you might set it up as:
"refresh": {
"query": "SELECT MAX(updated_at) FROM meta_table"
"recurringFrequency": 6000
}
This query would check every minute for a new updated_at
value in a table called meta_table
. The table you check with your App Refresh Query may or may not be one that you include as a Source.
If there are Schedules on this App's Sources that you do not want to be run, you can add "refreshEnabled": false
to your Schedule's config.
Models
The Model config file is currently very basic:
class
: (required) Should be set to"Model"
.id
: (required) The unique value that identifies this Model.name
: (required) A display name for the Model.type
: (required) Can be set to"profile"
,"account"
, or"custom"
.
Example:
{
"class": "Model",
"id": "users",
"name": "Users",
"type": "profile"
}
Sources
Each Source has a common set of attributes. Certain types of Sources may have additional attributes, but all Sources will have the following common attributes:
class
: (required) Should be set to"Source"
.id
: (required) The unique value that identifies this Source.name
: (required) A display name for the Source.type
: (required) The type of Source will vary based on the Generator you're using. You likely don't want to change this value.appId
: (required) Theid
of the App (connection) that the Source uses.
Example:
{
"class": "Source",
"id": "demo_users",
"modelId": "users",
"name": "Product Users",
"type": "postgres-import-table",
"appId": "demo_db",
// Options depend on the type of app
"options": {
"table": "users"
},
// Mapping settings
"mapping": {
"id": "user_id"
}
}
Schedules
A Schedule tells Grouparoo how frequently to check the Source for updated data and import it into the Application Database. This is the most common way to keep data fresh in Grouparoo.
Here are the attributes for a Schedule:
id
: (required) Like the Source, a Schedule must have a uniqueid
to identify it.name
: (required) A display name. It matches theid
by default.class
: (required) This must be set to"Schedule"
.sourceId
: (required) This must match theid
of the Source. It will be set automatically.recurring
: (required, default:true
) Should this Schedule run regularly?incremental
: (default:true
) Should this schedule only update records that have changed since the last run?refreshEnabled
: (default:false
) Should this schedule also trigger when the App Refresh query finds new data?recurringFrequency
: (required) The length of the Schedule's interval. This is measured inms
. For example, 15 minutes would be1000 * 60 * 15
or900000
.options
: These are the custom options that the specific Source type will bring.filters
: These can be used to exclude certain rows of data from your Schedule. Depending on your Source's database, only some of these filters may be available.
Filters Ops
"exists"
- exists with any value (is not null)"notExists"
- does not exist with any value (is null)"eq"
- equals"ne"
- does not equal"gt"
- greater than"gte"
- greater than or equal to"lt"
- less than"lte"
- less than or equal to"substring"
- contains"notSubstring"
- does not contain"in"
- in a given set
For example, to create a schedule that will run every 5 minutes checking the "updated_at"
column for rows where status = "active"
and last_purchase"
exists (is not null), you could do the following:
{
"class": "Schedule",
"id": "users_schedule",
"name": "Users Schedule",
"sourceId": "users",
"recurring": true,
"recurringFrequency": 5 * 60 * 1000,
"confirmRecords": false,
"options": {
"column": "updated_at"
},
"filters": [
{
"key": "status",
"op": "eq",
"match": "active"
},
{
"key": "last_purchase",
"op": "exists"
}
]
}
If you've configured an App Refresh Query when you configured your App, you can also opt to leave a particular Schedule out of its updates by adding "refreshEnabled": false
to the Schedule's config.
Properties
Each Property has a common set of attributes, while other attributes are specific to the type of Property. Here are the common attributes:
class
: (required) Should be set to"Property"
.id
: (required) The unique value that defines this Property.isArray
: (required) Is the Property an array?name
: (required) A display name for the Property. Will be set to theid
by default.sourceId
: (required) Theid
of the Source that the Property is created from.type
: (required) The data type of the Property.unique
: (required) Iftrue
, Records will be forced to have unique records for this Property.
Property type
depends on the Source App's available properties and may include some or all of these types:
boolean
date
email
float
integer
phoneNumber
string
url
Example:
{
"class": "Property",
"id": "contact_email",
"type": "email",
"key": "contactEmail",
"sourceId": "account_users",
"unique": false,
"isArray": false,
// Options can vary depending on the type of Source
"options": {
"aggregationMethod": "least recent value",
"column": "email",
"sortColumn": "created_at"
},
"filters": []
}
Aggregation Method and Filters
Your Property config file will also contain aggregationMethod
and Filter
options. Options for these may vary depending on your source type.
Here are some common examples:
aggregationMethods
"exact"
"sum"
"average"
"count"
"min"
"max"
"most recent value"
"least recent value"
filters
"exists"
- exists with any value (is not null)"notExists"
- does not exist with any value (is null)"eq"
- equals"ne"
- does not equal"gt"
- greater than"gte"
- greater than or equal to"lt"
- less than"lte"
- less than or equal to"substring"
- contains"notSubstring"
- does not contain"in"
- in a given set
These can be combined to create more complex properties such as:
{
"id": "active_paid_subscriptions",
"name": "Active Paid Subscriptions",
"class": "property",
"sourceId": "users_table",
"type": "integer",
"unique": false,
"identifying": false,
"isArray": false,
"options": {
"column": "subscription_id",
"aggregationMethod": "count",
"sort": null,
},
"filters": [
{
"key": "status",
"op": "eq",
"match": "active",
},
{
"key": "last_purchase",
"op": "exists"
}
{
"key": "subscription_type",
"op": "ne",
"match": "trial",
}
]
},
Which would count every subscription per Record, but leave out any trial subscriptions or Records with no value for last_purchase
. Or...
{
"id": "annual_purhcases_2022",
"name": "Annual Purchases 2022",
"class": "property",
"sourceId": "purhcases_table",
"type": "float",
"unique": false,
"identifying": false,
"isArray": false,
"options": {
"column": "purhcase_amount",
"aggregationMethod": "sum",
"sort": null
},
"filters": [
{
"key": "created_at",
"op": "gt",
"match": "2021-12-31T11:59:59.999+00.00"
},
{
"key": "created_at",
"op": "lt",
"match": "2023-01-01T00:00:00.000+00:00"
}
]
}
...which would return the sum of a Record's purchases in 2022.
Groups
Group config files have a common format.
id
: (required) The unique value that identifies this Source.class
: (required) Should be set to"Group"
.modelId
: (required) The model that the group belongs to.name
: (required) A display name for the Group.rules
: (required) An array of rules to filter Records. See more details below.
Example:
{
"id": "high_value_accounts",
"class": "Group",
"modelId": "accounts",
"name": "High Value Accounts",
"rules": [
{
"propertyId": "account_value",
"op": "gt",
"match": "800"
}
]
}
Configuring a Group is all about adding Rules to the Group. A Rule is some logical operator to Filter Properties. A Record must match every Rule to be included in the Group.
Available operations depends on both the type of Property and the underlying Grouparoo database (SQLite vs. Postgres). See below for a list of rules. (If in doubt, choose Postgres.)
Note you can use SQL wildcard characters (%, _, [ ], ^, -) in your Group rule match values as appropriate and necessary.
The following operators are available on boolean
properties:
exists
{"propertyId": "subscribed","op": "exists"}
notExists
{"propertyId": "subscribed","op": "notExists"}
eq
{"propertyId": "subscribed","op": "eq","match": "true"}
ne
{"propertyId": "subscribed","op": "ne","match": "true"}
YYYY-MM-DDThh:mm:ss.mmm+TZD ("2020-09-01T08:15:00+00:00") A date passed as YYYY-MM-DD will pass validation, but may result in less accurate Group results.
date
properties:exists
{"propertyId": "last_purchase_date","op": "exists"}
notExists
{"propertyId": "last_purchase_date","op": "notExists"}
eq
{"propertyId": "last_purchase_date","op": "eq","match": "2021-05-25T04:45:00.000+00:00"}
ne
{"propertyId": "last_purchase_date","op": "ne","match": "2021-05-25T04:45:00.000+00:00"}
gt
{"propertyId": "last_purchase_date","op": "lt","match": "2020-12-25T04:45:00.00+00:00"}
lt
{"propertyId": "last_purchase_date","op": "lt","match": "2021-05-25T09:30:00.000+00:00"}
gte
{"propertyId": "last_purchase_date","op": "gte","match": "2019-06-14T18:16:00.000+00:00"}
lte
{"propertyId": "last_purchase_date","op": "lte","match": "2020-12-25T04:45:00.000+00:00"}
relative_gt
{"propertyId": "last_purchase_date","op": "relative_gt","relativeMatchNumber": "60","relativeMatchUnit": "days"}
relative_lt
{"propertyId": "subscription_ends","op": "relative_lt","relativeMatchNumber": "30","relativeMatchUnit": "days"}
The following operators are available on email
properties:
exists
{"propertyId": "email","op": "exists"}
notExists
{"propertyId": "email","op": "notExists"}
eq
{"propertyId": "email","op": "eq","match": "hello@grouparoo.com"}
ne
{"propertyId": "email","op": "ne","match": "hello@grouparoo.com"}
like
{"propertyId": "email","op": "like","match": "%smith%"}
notLike
{"propertyId": "email","op": "notLike","match": "%smith%"}
startsWith
{"propertyId": "email","op": "startsWith","match": "test"}
endsWith
{"propertyId": "email","op": "endsWith","match": "grouparoo.com"}
substring
{"propertyId": "email","op": "substring","match": "grouparoo"}
iLike
{"propertyId": "email","op": "like","match": "%smith%"}
notILike
{"propertyId": "email","op": "notILike","match": "%smith%"}
The following operators are available on float
properties:
exists
{"propertyId": "ltv","op": "exists"}
notExists
{"propertyId": "ltv","op": "notExists"}
eq
{"propertyId": "ltv","op": "eq","match": "113932.97"}
ne
{"propertyId": "ltv","op": "ne","match": "113932.97"}
gt
{"propertyId": "ltv","op": "gt","match": "113932.97"}
lt
{"propertyId": "ltv","op": "lt","match": "113932.97"}
gte
{"propertyId": "ltv","op": "gte","match": "113932.97"}
lte
{"propertyId": "ltv","op": "lte","match": "113932.97"}
The following operators are available on integer
properties:
exists
{"propertyId": "visits","op": "notExists"}
notExists
{"propertyId": "visits","op": "notExists"}
eq
{"propertyId": "visits","op": "eq","match": "6"}
ne
{"propertyId": "visits","op": "ne","match": "6"}
gt
{"propertyId": "visits","op": "gt","match": "6"}
lt
{"propertyId": "visits","op": "ne","match": "6"}
gte
{"propertyId": "visits","op": "gte","match": "6"}
lte
{"propertyId": "visits","op": "lte","match": "6"}
The following operators are available on phoneNumber
properties:
exists
{"propertyId": "phone","op": "exists"}
notExists
{"propertyId": "phone","op": "notExists"}
eq
{"propertyId": "phone","op": "eq","match": "+1-555-555-5555"}
ne
{"propertyId": "phone","op": "ne","match": "+1-555-555-5555"}
like
{"propertyId": "phone","op": "like","match": "%780%"}
notLike
{"propertyId": "phone","op": "notLike","match": "%780%"}
startsWith
{"propertyId": "phone","op": "startsWith","match": "+1"}
endsWith
{"propertyId": "phone","op": "startsWith","match": "80"}
substring
{"propertyId": "phone","op": "startsWith","match": "80"}
iLike
{"propertyId": "phone","op": "notLike","match": "%780%"}
notILike
{"propertyId": "phone","op": "notLike","match": "%780%"}
The following operators are available on string
properties:
exists
{"propertyId": "lastName","op": "exists"}
notExists
{"propertyId": "lastName","op": "notExists"}
eq
{"propertyId": "lastName","op": "eq","match": "Ramirez"}
ne
{"propertyId": "lastName","op": "ne","match": "Ramirez"}
like
{"propertyId": "lastName","op": "like","match": "__m%"}
notLike
{"propertyId": "lastName","op": "notLike","match": "__m%"}
startsWith
{"propertyId": "lastName","op": "startsWith","match": "Mc"}
endsWith
{"propertyId": "lastName","op": "endsWith","match": "son"}
substring
{"propertyId": "lastName","op": "substring","match": "al"}
iLike
{"propertyId": "lastName","op": "iLike","match": "%al%"}
notILike
{"propertyId": "lastName","op": "iLike","match": "%al%"}
The following operators are available on url
properties:
exists
{"propertyId": "company_url","op": "exists"}
notExists
{"propertyId": "company_url","op": "notExists"}
eq
{"propertyId": "company_url","op": "eq","match": "https://www.grouparoo.com"}
ne
{"propertyId": "company_url","op": "ne","match": "https://www.grouparoo.com"}
like
{"propertyId": "company_url","op": "like","match": "%grouparoo.com"}
notLike
{"propertyId": "company_url","op": "like","match": "%grouparoo.com"}
startsWith
{"propertyId": "company_url","op": "startsWith","match": "https"}
endsWith
{"propertyId": "company_url","op": "startsWith","match": ".com"}
substring
{"propertyId": "company_url","op": "substring","match": "grouparoo"}
iLike
{"propertyId": "company_url","op": "substring","match": "%Grouparoo.com"}
notILike
{"propertyId": "company_url","op": "substring","match": "%Grouparoo.com"}
The following operators are available on boolean
properties:
exists
{"propertyId": "subscribed","op": "exists"}
notExists
{"propertyId": "subscribed","op": "notExists"}
eq
{"propertyId": "subscribed","op": "eq","match": "true"}
ne
{"propertyId": "subscribed","op": "ne","match": "true"}
YYYY-MM-DDThh:mm:ss.mmm+TZD ("2020-09-01T08:15:00+00:00") A date passed as YYYY-MM-DD will pass validation, but may result in less accurate Group results.
date
properties:exists
{"propertyId": "last_purchase_date","op": "exists"}
notExists
{"propertyId": "last_purchase_date","op": "notExists"}
eq
{"propertyId": "last_purchase_date","op": "eq","match": "2021-05-25T04:45:00.000+00:00"}
ne
{"propertyId": "last_purchase_date","op": "ne","match": "2021-05-25T04:45:00.000+00:00"}
gt
{"propertyId": "last_purchase_date","op": "lt","match": "2020-12-25T04:45:00.00+00:00"}
lt
{"propertyId": "last_purchase_date","op": "lt","match": "2021-05-25T09:30:00.000+00:00"}
gte
{"propertyId": "last_purchase_date","op": "gte","match": "2019-06-14T18:16:00.000+00:00"}
lte
{"propertyId": "last_purchase_date","op": "lte","match": "2020-12-25T04:45:00.000+00:00"}
relative_gt
{"propertyId": "last_purchase_date","op": "relative_gt","relativeMatchNumber": "60","relativeMatchUnit": "days"}
relative_lt
{"propertyId": "subscription_ends","op": "relative_lt","relativeMatchNumber": "30","relativeMatchUnit": "days"}
The following operators are available on email
properties:
exists
{"propertyId": "email","op": "exists"}
notExists
{"propertyId": "email","op": "notExists"}
eq
{"propertyId": "email","op": "eq","match": "hello@grouparoo.com"}
ne
{"propertyId": "email","op": "ne","match": "hello@grouparoo.com"}
like
{"propertyId": "email","op": "like","match": "%smith%"}
notLike
{"propertyId": "email","op": "notLike","match": "%smith%"}
startsWith
{"propertyId": "email","op": "startsWith","match": "test"}
endsWith
{"propertyId": "email","op": "endsWith","match": "grouparoo.com"}
substring
{"propertyId": "email","op": "substring","match": "grouparoo"}
The following operators are available on float
properties:
exists
{"propertyId": "ltv","op": "exists"}
notExists
{"propertyId": "ltv","op": "notExists"}
eq
{"propertyId": "ltv","op": "eq","match": "113932.97"}
ne
{"propertyId": "ltv","op": "ne","match": "113932.97"}
gt
{"propertyId": "ltv","op": "gt","match": "113932.97"}
lt
{"propertyId": "ltv","op": "lt","match": "113932.97"}
gte
{"propertyId": "ltv","op": "gte","match": "113932.97"}
lte
{"propertyId": "ltv","op": "lte","match": "113932.97"}
The following operators are available on integer
properties:
exists
{"propertyId": "visits","op": "notExists"}
notExists
{"propertyId": "visits","op": "notExists"}
eq
{"propertyId": "visits","op": "eq","match": "6"}
ne
{"propertyId": "visits","op": "ne","match": "6"}
gt
{"propertyId": "visits","op": "gt","match": "6"}
lt
{"propertyId": "visits","op": "ne","match": "6"}
gte
{"propertyId": "visits","op": "gte","match": "6"}
lte
{"propertyId": "visits","op": "lte","match": "6"}
The following operators are available on phoneNumber
properties:
exists
{"propertyId": "phone","op": "exists"}
notExists
{"propertyId": "phone","op": "notExists"}
eq
{"propertyId": "phone","op": "eq","match": "+1-555-555-5555"}
ne
{"propertyId": "phone","op": "ne","match": "+1-555-555-5555"}
like
{"propertyId": "phone","op": "like","match": "%780%"}
notLike
{"propertyId": "phone","op": "notLike","match": "%780%"}
startsWith
{"propertyId": "phone","op": "startsWith","match": "+1"}
endsWith
{"propertyId": "phone","op": "startsWith","match": "80"}
substring
{"propertyId": "phone","op": "startsWith","match": "80"}
The following operators are available on string
properties:
exists
{"propertyId": "lastName","op": "exists"}
notExists
{"propertyId": "lastName","op": "notExists"}
eq
{"propertyId": "lastName","op": "eq","match": "Ramirez"}
ne
{"propertyId": "lastName","op": "ne","match": "Ramirez"}
like
{"propertyId": "lastName","op": "like","match": "__m%"}
notLike
{"propertyId": "lastName","op": "notLike","match": "__m%"}
startsWith
{"propertyId": "lastName","op": "startsWith","match": "Mc"}
endsWith
{"propertyId": "lastName","op": "endsWith","match": "son"}
substring
{"propertyId": "lastName","op": "substring","match": "al"}
The following operators are available on url
properties:
exists
{"propertyId": "company_url","op": "exists"}
notExists
{"propertyId": "company_url","op": "notExists"}
eq
{"propertyId": "company_url","op": "eq","match": "https://www.grouparoo.com"}
ne
{"propertyId": "company_url","op": "ne","match": "https://www.grouparoo.com"}
like
{"propertyId": "company_url","op": "like","match": "%grouparoo.com"}
notLike
{"propertyId": "company_url","op": "like","match": "%grouparoo.com"}
startsWith
{"propertyId": "company_url","op": "startsWith","match": "https"}
endsWith
{"propertyId": "company_url","op": "startsWith","match": ".com"}
substring
{"propertyId": "company_url","op": "substring","match": "grouparoo"}
Destinations
Each Destination has a common set of attributess. Certain types of Destinations may have additional attributes, but all Destinations will have the common attributes. These common attributes, which may look very similar to other config object shapes in Grouparoo, are listed here:
id
: (required) The unique value that defines this Destination.name
: (required) A display name for the Destination. Will be set to theid
by default.class
: (required) Should be set to"Destination"
.type
: (required) The type of Destination will vary based on the app you're using. You likely don't want to change this value.appId
: (required) Theid
of the App (connection) that the Destination uses.collection
: (required) Which records should be exported from this Destination? Values can be"group"
,"model"
, or"none"
.groupId
: Theid
of the Group whose members should be exported to the Destination. (This is where Groups in Grouparoo really come in handy.) You need to choosecollection: "group"
to choose agroupId
.syncMode
: (required): The SyncMode you wish to use for this Destination. Values can be"Sync"
,"Additive"
, and"Enrich"
.mapping
: See details below.destinationGroupMemberships
: See details below.
Example:
[
{
"id": "newsletter",
"name": "newsletter",
"class": "destination",
"type": "mailchimp-export",
"appId": "...",
"collection": "...",
"groupId": "...",
"syncMode": "...",
"options": {
"listId": "...",
},
"mapping": {
"email_address": "email",
"FNAME": "firstName",
"LNAME": "lastName",
},
"destinationGroupMemberships": {
"High Value Customers!": "highValueCustomers",
},
}
]
Destination Mappings
Mappings are a series of key-value pairs that tell Grouparoo which Properties to export to the Destination for each Record. The structure may vary for each type of Destination, but generally, the keys are the fields in the Destination, while the values are the id
values for the Properties in Grouparoo.
Here's an example of a Mailchimp mapping configuration:
{
"mapping": {
"email": "email",
"FNAME": "firstName",
"LNAME": "lastName"
}
}
Notice the odd structure of the keys. That's specific to Mailchimp.
Destination Group Membership
Grouparoo also provides the ability to attach Group membership to the exported data. How this looks within the Destination is specific to the Destination. But the shape of the config file largely remains the same. In most Destinations, Destination Group Memberships become either tags (Mailchimp) or lists (HubSpot), or are appended as additional properties to the Record (Customer.io).
destinationGroupMemberships
are a set of key-value pairs that tell Grouparoo which memberships to publish to the Destination. For example, you may have sent all your users to a Mailchimp Destination, but you wanted to tag high-value customers.
In the example below, the key represents the tag in Mailchimp, while the value is the id
of the Group in Grouparoo:
{
"destinationGroupMemberships": {
"High Value Customers!": "highValueCustomers"
}
}
Validating & Applying Your Config
You can validate your config at any time using the validate
command:
$ grouparoo validate
And you can apply that config (save it to your Grouparoo application's database) using the apply
command:
$ grouparoo apply
Note that apply
will run validate
, but it's recommended to run validate
on its own first, just to be safe.
Having Problems?
If you are having trouble, visit the list of common issues or open a Github issue to get support.