Flow: How To De-duplicate Collection In 3 Ways

Step 1 – Identify Duplicates

First we copy the value from Col A into Duplicated using “Equals” operator.
=> Duplicated = Col A = [item1, item2]

Then use “Remove Uncommon” between Duplicated and Col B. This will remove the items from Duplicated that don’t show up in Col B.
=> Duplicated = [item1, item2] – [item2] = [item1]

(Note: We didn’t use Col A to remove uncommon because then all the unique items in Col A will disappear.)

Step 2 – Remove Duplicates

Now we can use “Remove All” to remove the duplicate from Col A. Since we still need the duplicates to show up once, we only need to do this with one of the collections, not both.
=> Col A = [item 1, item2] – [item1] = [item2]

Step 3 – Consolidate Collections

Finally, we can consolidate the collections after the de-dupe.
=> Consolidated: Col A: [item2] + Col B: [item1, item3] = [item1, item2, item3]

(Note: Instead of creating a new consolidated collection, you can also directly add Col B to Col A)

Considerations for this method

You can write the same operations in many different ways so there is no need to follow my steps strictly. However, this method is quite tricky for record collections. First, the records need to have the same field values. Second, you have to include the same fields in the collection. (Ex. if you store Id, Name, Email of a record in Collection A, but store only Id, Name in Collection B, they will not be considered as the same record).

What if we want to compare only a specific field? Or what if we are comparing records with and without Id? For this case, I will recommend using custom Apex action.

There are many great custom Apex actions on UnofficialSF.com, for example this package for collections.

What if we want to compare only a specific field? Or what if we are comparing records with and without Id? For this case, you can either use a loop to clean up the unneeded fields for each record, or use a Custom Apex Action. I would recommend using

Considerations for this method

This is a nice custom action and I look forward to the day that it becomes a standard element. However, currently we can only match one field. Also, what if we want to count the duplicates and update a quantity field? If so, you can consider the old fashion way – use the Loop!

Since it’s quite a complex method, let’s explain using a use case.

Lulu Mobile use the standard Opportunity and Opportunity Product objects. They have several sources that generate Opportunity Products and sometimes there will be duplicates. Whenever duplicates happen, they want to keep only one line but update the quantity.

We will need Loop, Collection Sort, and Placeholder variables for this method. If we sort the collection based on the fields we want to match, the items with the same values should be arranged together. In this case, we only need to compare the current item with the previous item – if they are the same, we add the quantity together. If not, we continue to the next item. Since the loop only provide the “Current Item” variable, we need to create a placeholder variable that holds the “Previous Item”. Then we can use them to make the comparison.

Partial Flow Chart – Can be incorporated into your own solution