Flow: How To De-duplicate Collection In 3 Ways

De-duplicate collection in Flow is a very interesting topic. There are many different ways to achieve this and it all depends on your creativity! Today I am sharing three methods that help you remove the duplicated items in your collections.

1 – Use the Special Collection Operators in Assignment

I found this method on Accidental Coder and I think it’s brilliant! Currently, there is no “Remove Common” operator in Assignment so we can’t quickly remove the duplicates. However, by using the “Remove Uncommon” operator, we can first find the duplicated records, and then remove them from our final collection.

There are FOUR collections needed for this method:

  • Collection A: [item1 ,item2]
  • Collection B: [item1, item3]
  • Duplicated collection that holds duplicates: [ ]
  • Consolidated collection that holds all records after de-dupe: [ ]

Step 1 – Identify Duplicates

First we copy the value from Col A into Duplicated using “Equals” operator.
=> Duplicated = Col A = [item1, item2]

Then use “Remove Uncommon” between Duplicated and Col B. This will remove the items from Duplicated that don’t show up in Col B.
=> Duplicated = [item1, item2] – [item2] = [item1]

(Note: We didn’t use Col A to remove uncommon because then all the unique items in Col A will disappear.)

Step 2 – Remove Duplicates

Now we can use “Remove All” to remove the duplicate from Col A. Since we still need the duplicates to show up once, we only need to do this with one of the collections, not both.
=> Col A = [item 1, item2] – [item1] = [item2]

Step 3 – Consolidate Collections

Finally, we can consolidate the collections after the de-dupe.
=> Consolidated: Col A: [item2] + Col B: [item1, item3] = [item1, item2, item3]

(Note: Instead of creating a new consolidated collection, you can also directly add Col B to Col A)

Considerations for this method

You can write the same operations in many different ways so there is no need to follow my steps strictly. However, this method is quite tricky for record collections. First, the records need to have the same field values. Second, you have to include the same fields in the collection. (Ex. if you store Id, Name, Email of a record in Collection A, but store only Id, Name in Collection B, they will not be considered as the same record).

What if we want to compare only a specific field? Or what if we are comparing records with and without Id? For this case, I will recommend using custom Apex action.

2 – Use Custom Apex Action

There are many great custom Apex actions on UnofficialSF.com, for example this package for collections.

What if we want to compare only a specific field? Or what if we are comparing records with and without Id? For this case, you can either use a loop to clean up the unneeded fields for each record, or use a Custom Apex Action. I would recommend using

After installing the package, look for “Dedupe” in the Action element. You can find the documentation here. The fieldToDedupeOn is the field you want to match and to identify duplicates. You have to fill in the API name for that field. The inputRecordCollection has to be the consolidated collection – so you have to add the target collections together first.

Considerations for this method

This is a nice custom action and I look forward to the day that it becomes a standard element. However, currently we can only match one field. Also, what if we want to count the duplicates and update a quantity field? If so, you can consider the old fashion way – use the Loop!

3 – Use Loop and Placeholder Variables

Since it’s quite a complex method, let’s explain using a use case.


Lulu Mobile use the standard Opportunity and Opportunity Product objects. They have several sources that generate Opportunity Products and sometimes there will be duplicates. Whenever duplicates happen, they want to keep only one line but update the quantity.

We will need Loop, Collection Sort, and Placeholder variables for this method. If we sort the collection based on the fields we want to match, the items with the same values should be arranged together. In this case, we only need to compare the current item with the previous item – if they are the same, we add the quantity together. If not, we continue to the next item. Since the loop only provide the “Current Item” variable, we need to create a placeholder variable that holds the “Previous Item”. Then we can use them to make the comparison.

Flow Chart (deactivate last login date)

Partial Flow Chart – Can be incorporated into your own solution

Steps (deactivate last login date)

Is This Helpful? Check Out Flow Use Cases Or Write Us One!

Subscribe
Notify of
guest

2 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Nic

Thanks for the post, great summary and I want to try out the apex option now!

Chiel

Tried the Loop solution. Worked like a charm. Thank for this great post.