Automate checking a ticket system's availability with Node.js and Pipedream

This post is about how I recently built a workflow to notify me when tickets became available on a particular site.

I was recently on holiday and wanted to book seats on a tour bus for the coming weekend. Unfortunately, being peak season, the tickets were sold out on their booking page. I asked the service company if there was a wait list we could join but there wasn’t. They advised I should keep checking back to the website as it will automatically update the availability when someone cancels or moves their booking.

The thought of spending time manually checking and refreshing the booking page repetitively did not appeal - but it did seem like the perfect job for a computer.

The idea

I’d not built a web scraper before, but it seemed like something a front end dev could figure out in an evening.

The bus bookings were made with an online booking system called Trybooking. From googling, I knew they had an API that needed authentication with a API key, so my first thought was that perhaps the key was sent in a query string or a cookie I could find in inspector dev tools.

When I scanned the network tab I found a fetch request with a name matching the number in the Trybooking url. The request url included the terms "calendar session times" which sounded like a winner!

Lucky for me, the url was unauthenticated.

The data

Opening the url returned a simple array of objects, each with an event date and an isAvailable boolean.

[
    {"eventDate":"2022-07-16T00:00:00","isAvailable":false}, 
    {"eventDate":"2022-07-17T00:00:00","isAvailable":false},
    {"eventDate":"2022-07-23T00:00:00","isAvailable":false}, 
    {"eventDate":"2022-07-24T00:00:00","isAvailable":true}, 
    {"eventDate":"2022-07-30T00:00:00","isAvailable":true}, 
    {"eventDate":"2022-07-31T00:00:00","isAvailable":true}
]

I could see the dates I wanted were the first two objects; 16 July and 17 July. I would just need to check if either of those date's isAvailable value was true. This should be pretty easy!

I quickly wrote some pseudo code to plan out what I needed to do.

// every X minutes (chron job?)
// get the page contents
// check for event date and value
// if available is true
// notify me by email or sms

The execution

I didn't have any spare servers to run a script, so looked for some kind of service that could host and run it for free. I came across Pipedream which fit the bill perfectly.

It allows 333 "invocations" per day on their free plan. I only needed this web scraper for 3-4 days, so this would determine how often I could run my script.

333 invocations divided by 24 hours a day, divide 60 minutes per hour by that, gives you roughly an invocation every 5 minutes.

I signed up for Pipedream's free tier and started a new workflow.

Set up intervals

It first asks what type of trigger you need. I selected "Schedule" to set up a regular chron job.

I then chose a "Custom interval" of every 5 minutes. I needed to generate and select a sample event to move on to the next step.

Write the node script

Next I added a node step to the workflow.

I wrote a basic script that:

• uses the Axios library to request the web page
• adds the response data to a JavaScript object variable
• checks if there was availability on Saturday or Sunday
• prints a message to the console based on the availability (or not)

import axios from "axios";

export default defineComponent({
    async run({ steps, $ }) {

        async function fetchHTML(url) {
            const { data } = await axios.get(url)
            return data;
        }

        // fetch the booking site availability as array
        const data = await fetchHTML("https://www.trybooking.com/events/calendar-session-times/911314");

        if (data[0].isAvailable || data[1].isAvailable) {
            console.log("Availability!");
        } else {
            console.log("No availability boo");
        }
    }
});

Pipedream has a "test workflow" function to check what I had so far. Running it worked.

Set up notification alerts

I knew the ability to email yourself results was built into Pipedream, but since I was on holidays I wasn’t planning to be checking my emails often.

Twillio has API’s for sending messages so I signed up for a free trial. Unfortunately, the rules on sending sms in Australia are pretty strict and it takes around 3 business days to be verified and approved to do so. More time than I really had to get this up and running, so I stuck with the built in email method. It would be cool to add sms notifications down the track.

There are two ways to email yourself in Pipedream. One as a separate step in the workflow and one inside the node step. I wanted to keep as much as possible within the code as possible, so went with the second option.

if ( data[0].isAvailable || data[1].isAvailable) {
    $.send.email({
    subject: `Availability!`,
    text: `They have availability! Yay!`
    })

} else {
    $.send.email({
    subject: `No availability :(`,
    text: `They have no availability yet - booooo`
    })
}

I deployed my workflow and triggered a test run. Within 30 seconds I had an email to the email address I signed up to Pipedream with. It showed the correct flow that there was no availability.

Now, while I want the workflow to run every 5 minutes - I don't want an email every 5 minutes.

This is where Pipedreams Data Stores comes in.

Persistng state

Data Stores allow you to pass information from one instance of a workflow to the next instance of the workflow. So I can set a workflow state of whether there is availability, and set an if/else statement to only send an email if the availability actually changes.

First I add a props state to create and access a data store for this workflow.

props: {
    data: { type: "data_store" }
  },

Next I retrieve the last state from the data store (though there isn't one yet, we'll be setting that soon). I also set a variable for when the availability is on this instance of the workflow.

const previousAvailability = await this.data.get('previousAvailability');
const currentAvailability = data[0].isAvailable || data[1].isAvailable; 

Then compare the availability status in the data store with the availability status from this instance.

If they're the same, we can bail out and do nothing. I don't need an email notification to tell me nothing has changed.

If they're different and availability has changed, then send me an email to tell me what the current state is.

if (currentAvailability !== previousAvailability) {
    if ( currentAvailability ) {
        $.send.email({
            subject: `Availability!`,
            text: `They have availability on ${isSaturdayAvailable ? 'Saturday' : 'Sunday'} - yay!`
        })
    } else {
        $.send.email({
            subject: `No Availability :(`,
            text: `They have no availability.`
        })
    }
}

The last step was to set or update the data store availability prop to the value returned from this instance of the workflow.

    await this.data.set('previousAvailability', currentAvailability);

Deploy

With this last step done and dusted, I deployed the app live.

This is what the final node script looked like:

import axios from "axios";

export default defineComponent({
    // Define that the "db" variable in our component is a data store
    props: {
        data: { type: "data_store" }
    },

    async run({ steps, $ }) {

        async function fetchHTML(url) {
        const { data } = await axios.get(url)
        return data;
        }

        // fetch the booking site availability as array
        const data = await fetchHTML("https://www.trybooking.com/events/calendar-session-times/911314");

        const isSaturdayAvailable = data[0].isAvailable;
        const isSundayAvailable = data[1].isAvailable;

        // retrieve the last availability status from the data store
        const previousAvailability = await this.data.get('previousAvailability');

        // store the availability status from this instance of the workflow
        const currentAvailability = isSaturdayAvailable || isSundayAvailable


        // compare the previous availability with the current availability
        // send email only if the availability has CHANGED
        if (currentAvailability !== previousAvailability) {
            if ( currentAvailability ) {
                $.send.email({
                    subject: `Availability!`,
                    text: `They have availability on 
                    ${isSaturdayAvailable ? 'Saturday' : 'Sunday'}`
                })
            } else {
                $.send.email({
                    subject: `No Availability :(`,
                    text: `They have no availability.`
                })
            }
        }

        // Update the data store to the current availability status
        await this.data.set('previousAvailability', currentAvailability);
    }
});

And that's it!

I was able to get notified by email when tickets became available for our chosen dates (or if they very quickly sold out again before I had time to get to buy them).

Overall, I was pleasantly surprised how fast and easy it was to set up, and can definitely see myself creating similar workflows in the future.

Jess is a senior software engineer and web accessibility nerd. When she’s not writing, speaking or tweeting about tech, you’ll find her putting together lego or going on walks with her doggo.