Batch actions

In this article, you'll learn how to create more efficient actions by splitting huge collections and processing them in individual actions.

Betty Blocks avatar
Written by Betty Blocks
Updated over a week ago

Warning!

This is a legacy document. The features described below are only usable within the classic-generation environment.

Currently, Betty Blocks offers faster and more advanced options that are available in next-gen. Before you start working on some new features in your application, consider doing it using the next-gen version. Good luck!


Logic and workflows are a big part of the application-building process. This is done through Actions in Betty Blocks. Avoid developing extensive actions in your application; instead, establish many small actions that form a chain of actions. Short actions are easier to maintain, especially as your application grows.

Another advantage of short actions is that you can use them in large quantities while still maintaining decent application performance. This is in contrast to massive actions that analyze massive volumes of data. Let's talk about how to use batches to create some efficient actions.

Batches

For example, suppose you're working with a collection of 10,000 records. You could perform what you want by looping through all 10000 entries, but this is inefficient and potentially result in a timeout because it processes in serial. Instead, it's preferable to split the collection into multiple, smaller collections of 100 records and send them to your action for processing in parallel.

Below is a step-by-step plan on which we'll zoom in afterward:

1. Create an action called Part 1 containing:

  • Collection variable called limited_collection with Limit set to 1

  • Number expression variable called total_amount containing the following expression:

count_full_collection(var:limited_collection)
  • Number expression variable called number_of_batches containing the following expression:

ceil(var:total_amount / 100)
  • Set number_of_batches in the Times option.

  • Set batch_number as value for Index.

3. Create another action called Part 2 which is called upon in each loop iteration containing:

  • Input variable called limit

  • Input variable called offset

  • Collection variable called collection_subset and set input variable limit to the limit option and input variable offset to the offset option.

4. Create an action step in the Loop flow of Part 1, calling upon Part 2.

  • Create a number variable called limit containing the value 100.

  • Create a number variable expression called offset containing the following expression:

(var:batch_number - 1) * int(var:limit)
  • Pass the variables limit and offset onto the corresponding input variables of action Part 2.

5. Process a smaller, more manageable collection in Part 2 for each iteration of your Loop step, based on the values passed along from action Part 1.

Some of these steps may seem a bit vague, but we'll discuss each step accompanied by images to clarify.

1. Create an action called Part 1

The action is where it all starts. This action would originally contain a collection variable too big to process efficiently. Instead, limit the variable to 1 and call it limited_collection.

limited_collection

With this collection, you'll create the collection you would normally use, but just with 1 record. Now we'll use this variable in another variable called total_amount. This time of the type number expression.

total_amount

count_full_collection(var:limited_collection)

You'll get the number of records in the collection without having to render the entire collection this way. This is important in order to maintain high levels of performance. We can calculate the number of batches to process now that we know how many records we have.

In this tutorial, we're working with a batch size of 100. Feel free to change it for your application, but stick to the same amount for the whole process. Create a number expression variable called number_of_batches to calculate the number of batches.

number_of_batches

ceil(var:total_amount / 100)

2. Create a loop step

After creating the action, add a Loop step to the action.

Loop step

In the Loop step, we're not looping through a collection, but instead, we're looping the number of batches. Do this by selecting the number_of_batches in the Times option. Set the Index option to batch_number. This will make the current iteration available through a number variable called batch_number.

3. Create an action called Part 2

Create another action called Part 2 which is called upon in each loop iteration.
Create 2 input variables: limit and offset.

These input variables will receive values from action Part 1, which we're going to use to define a new collection variable. 

collection_subset

The Limit option on a collection variable makes sure no more records than the value the option holds are included in the collection. The Offset option on a collection variable makes sure the collection starts collecting records from the position equal to the value the option holds.
For now, we'll leave this action for what it is. First, we need to connect this new action to the action we made in the beginning: Part 1.

4. Create an action step in the Loop flow of Part 1

Now we're adding an action step within the Loop flow of action Part 1.
In this action step we need to create 2 variables to pass on to the input variables from action Part 2: limit and offset.

limit

This variable has the same value as our batch size: 100

offset

(var:batch_number- 1) * int(var:limit)

With the offset variable, we can calculate which subset of records we are going to collect in the subaction of each iteration.

  • 1st iteration: record 1 - 100

  • 2nd iteration: record 101 - 200

  • 3rd iteration: record 201 - 300

  • etc...

These variables can now be passed on to the input variables of Part 2.

5. Process a smaller, more manageable collection in Part 2

This is where we'll put the steps that we were going to execute in the 'inefficient' action in the first place. In this subaction, we determined how many records we're processing, how many batches it takes to process them, and built a structure to pass on the values required to collect the desired records.

For this example, we're going to loop through a collection of 10000 records, update a property in each record and send an email. 

  • In our original design, we would loop through 10000 records, processing each record after the one before; serial.

  • In the improved design, we initiate 100 subactions, each processing 100 records. The background queue will pick up multiple actions at the same time due to parallel processing. This results in the action being done in a much shorter time!


Did this answer your question?