by James Lawson, Consulting Editor
Transferring files back and forth to an external data cleanser can be a time-consuming and risky process. What are the alternatives?
Alex Dickerson | Enterprise Solutions Sales Lead, Celerity
Mark Webb | Ops Director, Alchemetrics
Cyril Law | Solutions Director, CallCredit Marketing Solutions
Martin Doyle | CEO, DQ Global
Scott Taylor | Leader of Market Development & Innovation, Dun & Bradstreet
Does your data cleansing still involve sending spreadsheets and CSV files hither and yonder? Most marketers prefer to leave data cleansing to the experts but exporting and re-importing customer files remains a bugbear.
However service providers – and users themselves – are increasingly linking remotely to customer systems to run the whole process. With modern web service protocols and cloud platforms, this makes for frictionless, invisible cleansing, integrating and updating.
“Extracting and reloading is time consuming and an unnecessary risk,” says Martin Doyle, CEO at DQ Global. “The riskiest point in the process is where you are handing data over.”
A long-time proponent of cleansing in situ, Doyle points to the numerous pitfalls involved in manual export and import. “It’s too easy to send data in the wrong format or order, columns get mixed up – it happens all the time,” he says.
Instead, configuring a remote update process means the client and service provider can work together to get it right first time. “There’s greater control for the client, high speed and flexibility,” says Doyle. “They can customise and configure it exactly as they please.”
He picks DQ’s Perfect & Merge application as an example. This plug-in for MS Dynamics CRM offers numerous data quality functions including deduplication and field merging.
Delivered as a web service, users can fire up Perfect & Merge from within their own Dynamics applications whether it’s run on premise, in the cloud or a hybrid. DQ’s appRules offers even more extensive functionality, able to connect to over 80 different databases. Whether it’s a big migration project or regular updating, users can pull data from multiple systems to build and update an SCV with no need to import or export.
So you might have your CRM database in the cloud but want to pull down data for local analysts to look at. Another example would be migrating customers from salesforce to another cloud application.
“We are able to connect to both systems in the cloud,” says Doyle. “Our customer can see both applications in real time while checking, cleansing and deduplicating the data.”
Whether the source system is HubSpot, Zoho or MS CRM, users must first configure those applications’ own APIs to open the door for real-time data links. Then they can set up the schedule and put the rules in place to automate the cleanse/update process.
“To plug in is straightforward, really just point and shoot,” says Doyle. “You just input your credentials. The application knows the structure and the location of the data, then you use the workflow tools to set up the processes you need.”
When the user has built their processes for updating, verifying, scoring and so forth, they publish that workflow and have it execute as desired. That might simply mean running a batch cleanse at midnight every weekday or triggering a load routine when new data arrives in a shared input folder.
“We can set it up for the customer and teach them to do it themselves,” says Doyle. “It’s then easy for them to plug into our various third party partners who offer web-service-based address and email verification, enhancement and so on.”
Callcredit Marketing Services are also taking a web service-based approach to remote cleansing and enhancement with its Define API. There are two versions, one for its marketing clients and another for the financial services institutions that contribute their transactional data to the company’s SHARE database that holds financial records and payment performance.
Industry-standard web services plug in client systems to Define’s data and services, with clients employing the SDK (Software Developers Kit) to build and configure the connection. That takes account of the source system’s data model and makes sure the correct fields and records are being cleansed and updated.
“It gives easy instant access to suppression files, cleansing and consented data files,” says Cyril Law, Solutions Director at CallCredit Marketing Solutions. “Callcredit gets involved initially and you have to invest a little time to configure the set-up.”
“We run a couple of tests and once they are happy with that, we migrate them to the live service,” explains Law. “From then on, it’s completely client controlled and very much lights out once configured and scheduled.”
Available 24/7 via the MS Azure cloud hosting platform, Define works via a highly secure encrypted and authenticated connection. The payment model is pay as you go based on the volume of data enhanced or verified.
“You can use streaming or batch transfer and, rather than cleansing the whole file, you only cleanse the updated records,” relates Law. “The industry in general is moving to this kind of solution. Clients often have hybrid in-house and cloud systems, but it makes no difference to Define. As more clients move to cloud platforms like Azure or AWS, the process will become quicker and quicker.”
Define runs alongside more traditional scheduled cleansing options like secure FTP. “We pick the file up automatically and then return the cleansed data back the same way,” says Law. “Once you have it set up, you just keep running it automatically.”
Callcredit has also been upgrading its improvemydata website, with an improved interface to make it more user friendly. “We’ve increased the number of suppression reference files available,” says Law. “Users can also upload their own custom suppression files and make them available to their partners via our site.”
MSPs are traditionally experts at linking to remote systems and Celerity is no exception. Here too API connections are becoming standard to allow faster, more frequent and potentially bidirectional updating.
“The old Extract, Transform, Load processes are quite laborious,” says Alex Dickerson, Enterprise Solutions Sales Lead at Celerity. “As a result, we have started creating solutions in an application-based architecture.”
“These solutions integrate with all applications, from call centres, tills and email channels to back-end data sources such as the stock management system and CRM,” continues Dickerson. “The CRM could be cloud, and the transactional system could be on premise – it doesn’t matter as long as they are integrated.”
Traditionally this kind of MSP work has been about one-way data flows: pulling records from source systems to load and update a hosted SCV. This is still very much business as usual at Alchemetrics but there are increasingly exceptions where data integration across multiple systems is required.
One example comprises an SCV, a CRM database and an internal database held on the client side. Here the data in the source systems needs to be cleansed, merged and matched and then ultimately fed back to those three source systems to keep them synchronised. Near-real time API connections again link all three systems.
“Essentially there are 3 touch points that need to be kept up to date,” says Mark Webb, Operations Director at Alchemetrics. “The client has a picture of their data and can update their own internal database, whilst allowing their customers to feed data into the SCV. Internal users can update the CRM system while taking calls from customers or add new leads while out on the road.”
According to Webb, a CRM system is more commonly part of a build, whereas it just used to be a SCV. He says that using a cloud-based system like Alchemetrics’ Informa platform allows clients to interact more with the loading, the structure and the business rules of their data.
“Through our DQS tool, a client can define a Question Set which defines the layout of the data, the table the data needs to be loaded to and how the data should be cleansed and reformatted on loading,” explains Webb. “This layout can then be used either through the ETL tool or via API connections, allowing the client to do batch uploads or near real time uploads to the SCV.”
Either the batch loading or real-time streaming via an API can be automated for lights-out updating. Again, the critical stage comes during set-up, for examples developing the tools needed to connect to legacy databases.
“It is also becoming increasingly important to understand the client’s requirements in full from day one, ensuring that what we are doing is as expected,” says Webb. “As turnaround times are getting tighter, any rework can have an effect on the rest of the project, especially with more use of CRM systems as an essential part of the build and API connections to various touchpoints.”
In the domain of business data, Dun & Bradstreet has long used online links for updating and enhancement, dating right back to its original Market Spectrum product launched 17 years ago. Today, its integrated solutions use the very latest API technology, working to 300ms update speeds to enable real-time updates and new record loads.
“Our APIs deliver real-time integration and we also offer pre-mastered commercial content for relationship entities – customers, vendors, partners and prospects – as well as a pre-installed Master Data platform that works with systems such as SFDC, Dynamics, Eloqua or Marketo,” says Scott Taylor, Leader of Market Development & Innovation for Master Data at Dun & Bradstreet. “The beauty is that this removes the majority of the IT-based pain points on the client side and simplifies the process.”
By linking to D&B’s reference view of the business world, marketers can find linkages between seemingly unconnected businesses and prevent duplicate creation during new data loading.
“Clients gain immediate access to the pre-mastered commercial content for that entity, allowing them to search before creating a new and possible duplicate record,” says Scott. “More importantly, this knowledge helps you sell into other entities within the same hierarchy – either above, across or below.”
Identifying duplicates, outdated contacts or accounts that have gone out of business is just the start of the journey. Matching instantly to D&B’s own master data makes sure all new records are correct on entry and can be used to support other real-time services like web form validation. The company’s WebFormID can pinpoint a company from as little as an email address and then auto fill business forms with the associated reference data.
“Not only does this prevent duplicate records being created and enable appropriate customer/prospect engagement, but it also dramatically shortens the time taken to complete web forms and reduces form abandonment,” says Scott, noting that a similar service can support data entry by call centre agents.
“The operator can immediately identify the enquirer based on an email or phone number to determine if they are an existing account,” he continues. “They can then use an appropriate script and customer journey accordingly.”
Real time links between the client’s customer records and the reference universe also let D&B send new business information to the relevant account manager. Monitoring 1 billion web pages and 20,000 news sources per day, the company can pipe news alerts and business signals as soon as it becomes available. This rich data also lets D&B deliver insight to sales teams on who to talk to, what to talk about and when to have the conversation.
So rather than pushing and pulling extracts hither and yonder, why not let your service provider tap straight into your systems via an API connection? Viable even with legacy systems, it’s faster, cheaper and opens up a wide range of other data-on-demand services.
“For us, whether it’s in-house or online, it’s just a connection,” says Doyle. “All you need is the right log-in and you’re off.”