How to Create a Python Script to Automatically Monitor Prices Online

A guide to setting up your own web scraper, creating a database of historical prices, and an email notification system— and setting it all up to run automatically on an ongoing basis.

Photo by Engina Kyurt on Unsplash

The modern world is full of pricing algorithms that change the prices of things all the time. By building a tool to monitor those prices, you can stay one step ahead, and feel confident about making informed decisions about big purchases. Plus, the pieces we’ll create here can be adapted for use in all kinds of other projects too. Let’s get started.

In my case, I want to keep an eye out on the prices for podcast ads from Overcast (a podcast app that sells banner ads for other podcasts to help grow their audience). Here’s what the top of that page looks like:

Overcast’s ad page

Importantly, they update their pricing every day at midnight GMT based on how many ads are purchased vs unsold. I don’t know if I’ll ever buy an ad for my podcast talking to sci-fi authors and reviewing books (The Hugonauts), but if I do, I definitely want to buy low!

So, let’s get to the good stuff — the code. We’ll first look at all the code you’ll need to build the program, and then how to automate it all to run on an ongoing basis at the end.

Import our libraries:

If you don’t have any of those libraries, make sure you install them before moving forward.

Scraping the website:

In general, the code block below does three things:

1. Scrapes the HTML from our target website using requests
2. Uses Beautiful Soup to pull out the column headers and rows of data from the table
3. Combine the headers and data rows into a python dictionary (and add in a timestamp for when we scraped the data)

The code above is going to be the least-generalizable part of this program because every website's HTML is different(unless you’re looking to scrape overcast data too). However, all the tools and methods used will work on any website — and with not too many alterations if the website you’re interested in has a table on it as well.

Saving our data down to a local SQL database:

I use a Microsoft SQL Local DB that I set up on my machine, but you could use any other DB, SQLite, or even keep appending to a local csv and use that as a database of sorts (see the CSV method in use on a previous post here).

The engine parameters for sqlalchemy will need to be updated to match your SQL server settings. Google your particular SQL server type plus ‘sqlalchemy engine’ to find good examples for your particular server.

Summarize our data and organize it into the body of an email to let ourselves see the current pricing vs historic prices

Having all that pricing data is nice — but it doesn’t really help us make better decisions if we don’t use it. So we’ll also automatically send ourselves an email each day showing us three things for the three potential categories (Arts, Fiction, and Science) where we might want to buy an ad:

1. Today’s price, along with the price for the most recent 9 days
2. The all-time low price
3. The average price

Here’s the code to make it happen. Since we’re doing the same thing for three different categories, we’ll define functions first and then use them for each of the three categories to assemble our email. We’re using email.MIME library to construct our email body.

Here’s what the body of our email looks like when it arrives in the inbox (showing only three days worth of data because at this writing the script has only been running for three days):

The body of the automated email we’ve constructed

Use SMTP to connect to our email server and send our message

Note that Google disabled allowing SMTP to send emails through Gmail as of 5/30/2022 . To remedy that, I set up a Hotmail account to be the ‘from’ email, and it’s been perfect for these purposes.

If you are using a Hotmail or outlook email address, the code below will work for you right out of the box after adding your email username and password. If you want to use a different email provider, look up the SMTP settings for your email provider to find the url and port you’ll need for the first smtplib line of code below.

Note the input(‘Enter any key to quit’) line at the end of the except branch. That will keep our program window open and waiting for a response from us if the email fails to send once you set up your automation so you can see any error messages that were generated.

We’ve got working code! Let’s package it up so our computer can run it on its own.

I’m on Windows, so the steps below are Windows-specific. Mac users, your flow is going to be slightly different — after you’ve got your .py file, you’ll use Crontab to set up your task scheduling. Here are our three steps on Windows:

1. Save your code in a .py file
2. Create a .bat file which directs your computer to open your python environment and execute the .py script
3. Use your computer’s task scheduler to run the .bat file on a schedule of your choice

Okay…

Step 1 — making the .py file.

To make the .py, simply take your working code (for instance, all the code blocks above) and paste it into a text editor file. Then save the file as yourfilename.py.

Make note of the location where you saved your file. You can test your .py file from the terminal by navigating to the directory where it is saved and running the code below.

If it works and executes your script, so far so good!

python your_file_name.py

Step 2 — the .bat file

You’ll see articles all over the web telling you to skip this step and direct Task Scheduler to the .py file and python.exe. Do not do that!

For the vast majority of Python users, your code will not run because your code is not written using a base Python install — it was written on top of Conda (or Anaconda), or you’ve pip’d in a bunch of new libraries, or you’re running a virtual environment to handle your libraries — none of which your computer will handle appropriately without instruction. And that’s what our .bat file is for.

If Windows Task Scheduler isn’t running your .py file, here’s what you need to do. Open your text editor to a clean new file, and drop in the following code:

@echo off
call C:\Users\brent\miniconda3\condabin\activate.bat
cd “D:\Brents_Files\Documents\Python\Monitoring_Overcast_ad_pricing”
python overcast_price_monitor.py

The call line tells your computer what environment to use to open the file. In my case, I use Conda but not a virtual environment (yet) — so I point to the .bat file that activates Conda.

The cd line should point to the directory where you saved your .py file, and the final line that calls python also needs to be updated with your chosen file name.

Save this text file with a .bat ending, and you’re almost done! To test your file, double click the .bat file, and it should run all the way through!

If it fails, you’ll need to debug before going any further. Note, if double-clicking it opens it in a text editor, that doesn’t mean your file doesn’t work — it just means you’ve told your computer to open .bat files in a text editor instead of running them. You can execute your .bat file from your terminal to test it instead.

Set up task automation to run our script on a recurring basis

I’m on Windows, so I’ll be using Task Scheduler. For Mac users, you’ll be using Crontab instead.

Click the start button and search for ‘Task Scheduler’:

How to open Windows Task Scheduler

Now, in the pane on the right hand side, hit ‘Create Task…’

Windows Task Scheduler Create Task

Name your task, write a description for yourself— and most importantly chech the ‘Run with highest privileges’ box! When you’re done with those steps, go to the Triggers tab.

Windows task scheduler screenshot and instructions

On the triggers tab, hit ‘New’ and set your parameters for when and how often you want your task to run.

No screenshot of the triggers page because it’s fairly self-explanatory. Next, hit the ‘Actions’ tab.

Windows Task Scheduler Actions tab instructions

On the ‘Actions’ tab, hit ‘New’. In the popup that appears, either enter the full path (with the file name) to the .bat file you created in the ‘Program/script’ field, or hit browse and navigate to the .bat file and hit ok.

Then, in the ‘Start in (optional) field, enter the path to the folder in which the .bat file resides. I know that seems duplicative, but Windows seems to need both.

That’s it! Never pay too much for online purchases again.

Happy coding y’all.