Transformers

Introduction

Transformers are re-usable JavaScript functions that manipulate data in UserClouds. They allow you to minimize the data that you pass or store for each use case. This is key for complying with the data minimization principles in regulations like GDPR.

Transformers can be arbitrarily smart about how they preserve and obscure the information and structure of the data they encode.

Transformers can be arbitrarily smart about how they preserve and obscure the information and structure of the data they encode.

Transformers allow you to retain select structure and information from the raw data for different use cases, like sorting alphabetically, common domain analysis or simply flowing through your systems without triggering validation errors. For example, if you want to conduct analysis assessing the differences in behavior between children and adults, you may use a data transformer to pass a string indicating child or adult, instead of pulling raw Date of Birth from the store.

Transformers are used in two places:

  • Each accessor (read API) is associated with a transformer per column, dictating how the data in that column is obscured or minimized when the accessor is called
  • When a token is created, a data transformer is specified to dictate how the token should be generated from the raw data (and therefore what information, if any, should be retained in the token)

Examples of Transformers

Transformers range from the simplest UUID function that replaces any given data with a unique identifier, to custom-written Javascript that runs in a carefully-controlled sandbox.

One simple transformer receives raw data, and creates a UUID token.

One simple transformer receives raw data, and creates a UUID token.

Let’s look at four possible transformers, to see how transformers work.

FunctionExample DataExample Output
Replace with UUID[email protected]6ac33e45-1117-457f-b610-b49afd0bd551
Retain email structure, first three letters & common domain names[email protected][email protected]
Show year of birth2/24/19551955
Show country from phone+44 7899 123456United Kingdom

Types of Transformers

There are four types of transformers:

  • PassThrough Transformers pass through the raw data, unchanged
  • Data Transformers mask, categorize, inject noise or group raw data. They are used to retain the minimum characteristics of a piece of raw data for a particular task, without being resolvable back into the raw data.
  • Tokenize By Value Transformers create a resolvable token with an associated access policy. If the value of the raw data later changes, the token will resolve to the value of the data at the point of transformation.
  • Tokenize By Reference Transformers create a resolvable token with an associated access policy. If the value of the raw data later changes, the token will resolve to the latest value of the data.

Structure of a Transformer

A transformer consists of:

  • Name
  • Description
  • id - a unique transformer identifier, which can be used as a reference when creating accessors and tokens
  • Input Type
  • Output Type
  • Transform Type - as described above
  • Transform Function - a transform function with the signature func(data Object, parameters Object) (Token | error)
  • Transform Parameters - a static JSON object (not containing un-encoded PII) that is available at runtime, allowing you to parameterize and reuse functions like "obfuscate all but the first X letters of these emails"
The PreserveCommonValue parameter allows you to optionally preserve common email domains (like gmail.com). Rare domains, like userclouds.com, will be obscured.

The PreserveCommonValue parameter allows you to optionally preserve common email domains (like gmail.com). Rare domains, like userclouds.com, will be obscured.

Managing Transformers

UserClouds has several off-the-shelf transformers for common use cases, like masking emails, national ID numbers and credit card details. However you can also create custom functions, in three ways:

  1. Call the CreateTransformer() API
  2. Create functions in the UserClouds UI

For more info on creating transformers, see our How To Guide on Creating a Transformer.


What’s Next