With its new deduplication tool, Perfect & Merge, DQ Global has created a software solution that uses sophisticated matching techniques in MS CRM without the need to import an external list, with functions that are streets ahead of the competition.
Handling duplicates isn’t something that packaged CRM systems tend to do very well and Microsoft Dynamics CRM is no exception. Its own dedupe tool only offers exact matching options and a maximum of two duplicates can be compared at any one time.
Enter DQ Global’s dedicated Perfect and Merge package. Built solely to work with MS CRM, this deduplication tool uses sophisticated fuzzy matching techniques to identify duplicate candidates. From there, a comprehensive manual review process plus automated merge options help maximise match accuracy and ensure the right field elements from each dupe are transferred to the final master record.
Work starts by running Perfect & Merge from within MS CRM itself via a convenient button at the top of the home page. Each matching run is logged as a separate session, with the various groups of matches found held separately within each session.
Users start with a table showing all current sessions, listing group numbers, total records, number of dupes found and so forth. Using a simple, five-step wizard, those with administrator privileges can access extensive set-up options for each session, from the records and fields selected to the fine detail of the matching. Standard users can only choose to start a new session or revisit an existing one, with saved templates available to base new sessions on.
Deduplication work often starts by importing an external list but there’s no need for that here. This application works directly on the live data within MS CRM’s internal database, which is far more convenient and reduces the workload for admin staff. With Perfect & Merge constantly refreshing its in-memory view from the database at each processing step, you know you are processing the very latest data. It also means that users can continue to work with the database as normal without inconsistencies appearing.
With a session underway, the user simply chooses a group of dupes to review. That brings up the main review window which displays the fields of all the candidates side by side for comparison with the current Primary (best or master) record on the left end. Where there are many candidates, each record’s display panel can be minimised to free up screen space.
Another fundamental part of DQ’s approach is to link each group of duplicates to an owner. By default, that’s the first person to inspect the group. If they decide not to make a decision, they can select another user to pass that group to or return it to the pool.
That means multiple people can simultaneously be involved in reviewing the results of a dedupe session via their individual ownership of each match group. Linking duplicate decisions to users is an excellent feature, speeding up the whole review process and supporting the concept of data stewardship.
The review controls are commendably limited and straightforward: a Defer button passes on group ownership while clicking a arrow beside each field transfers the data it holds to populate the same field in the current master record. With record notes, you can choose to either replace the existing master data altogether or to append/preface it with the new information.
A Promote button replaces the current master with that record while unticking a box drops a candidate from the group altogether. View Source lets the user inspect and edit the live data held in MS CRM, including any attached items or notes that may help with the match decision.
Each reviewer can also add comments in the master record’s notes about their match decision, but some way of recording and sharing meta data on match groups between users within Perfect & Merge itself would enhance collaborative working: “Not sure about the third dupe in this group, that’s your department, what do you think?”
The fine detail of matching and review options is where this package really shows its quality. For example, to better organise a large group, you can split its candidate records into smaller sub-groups. The obvious reason would be to manage dupes at different companies within a large shared office building that all have the same address and postcode.
Once you have dealt with them, you can also choose to suppress formerly rejected dupes so that they don’t continue to appear as candidates. It’s a simple but smart way to minimise reworking.
The matching itself is key-based, with short keys being generated for each field and then combined into a long key for each record. Comparing these keys generates a confidence score used to decide whether two records are possible matches or not.
There are extensive options for customising this process within the Session set-up wizard, starting with choosing to dupe at account, contact or lead level. There are a host of algorithms available to generate the matching keys for each field, users can vary the confidence threshold at which a record is classed as a dupe (tightness or looseness) and they can make multiple dupe runs with different settings on the same data.
It’s possible to vary the priority (weighting) given to each field for matching: should the email addresses match exactly for two record to qualify as a dupes? You can also choose to omit fields from the matching process altogether. Libraries of synonyms, antonyms, common business abbreviations and so on support a wide range of data transformations (temporary for matching purposes only).
Full automation comes courtesy of the Auto Merge function, giving the option to merge records in a session without review based on custom rules configured in set-up. A typical example would be to discard any dupes where certain specified fields (company name, address, postcode) match exactly with the master record. Auto Merge deactivates the duplicate and transfers any linked contact and other data to the master record.
With records updated and merge fields decided, the user clicks Accept. All that then remains is for the background Smart Merge function to tell MS CRM’s own merge tool to make the changes in the database. Throttling and scheduling controls ensure users don’t notice any performance drop during this final processing.
Perfect & Merge sets out to do a particular job and absolutely succeeds in its goal. Its multi-user manual review approach combined with Auto Merge options works very well, especially where complex business data is concerned. However, if the application will be used with higher volumes of consumer data in particular, more automated batch control of both deduping and record merging is desirable.
The vendors say more automation is coming, along with web service access to standard reference files for suppression and PAF verification. Being able to pull in standalone lists to work on or to dupe against the records already in the database would be another logical upgrade, as would the ability to dupe records one-by-one during data entry.
But with all these and many other enhancements already offered in the vendor’s other applications such as the fully-featured DQ Studio, criticising Perfect & Merge for omitting them rather misses the point. Its tightly-defined set of functions are streets ahead of inferior plug-in utilities as well as MS CRM’s own built-in tools, offering accurate, highly-configurable matching performance built on decades of customer data processing experience.
Capable of working in multiple languages, its well-thought-out, logical workflow and clear GUI make what can be a very fiddlesome, complex task straightforward. For any MS CRM user plagued by duplicates, Perfect & Merge is a very desirable upgrade indeed.