I recently got assigned a daily task of looking up government tenders for a certain company, filtering through various opportunities to find those that match its portfolio and then pushing the results to the procurement team to capitalize on them.

The task being well-paying, I decided to execute my "11thMinAut" rule (read as 11th Minute Automation), which says if one iteration of a repetitive task takes 10 minutes of my time, I have to automate it just to save the 11th minute.

Uhhh, not this type of automation | Credits: giphy.com

So in this post, I will be taking you through the setting up of the script that does my "dirty" work.

By the end of this post, we will have covered:

  1. Code deployment on Heroku.
  2. Execution of a Cron-job using an external cron service.
  3. The use of Africa's Talking API to send an SMS from our server.
  4. DOM-traversal using Cheerio.
  5. The use of the request module to fetch data from a remote server.
  6. The use of Postman and Google Chrome Developer Tools to unravel how a website is populating its data.

Prerequisites:

With the above prerequisites satisfied, we can proceed. I will adopt a step-by-step approach, should you get stuck in any of the steps below, shoot your question in the comment box below this article.

Setting up our project:

The first thing that we will do is log into our Github account and create a new repository. I will name mine govtenders. You can give yours a name of your choice.

Next, we will open a terminal in our local machine and paste the commands below (remember to substitute the name DaggieBlanqx with your Github username, and the term govtenders with the name you choose for your repository):

mkdir govtenders
cd govtenders
git init
npm init --yes
echo node_modules/ >> .gitignore
echo package-lock.json >> .gitignore
echo *.lock >> .gitignore
git remote add origin https://github.com/DaggieBlanqx/govtenders.git
git push -u origin master

What the above command does is:

  • Create a directory called govtenders.
  • Then enter into that folder.
  • Then initialize it as a local git repository.
  • Then initialize the folder as an npm project. The --yes flag tells the generator to use the defaults instead of prompting us with questions.
  • Through the gitignore file, we exclude the node_modules folder, package-lock.json file, and every other type of a .lock file from being uploaded to Github.
  • The next command links our local repository with the repository we created on Github.

Next, we will install the modules we need for this project. In our terminal, we will paste the commands below:

npm i --save africastalking cheerio dotenv express moment nodemon request

This command installs the latest versions of these modules.

Let us quickly talk about each of these modules:

  • The request module will help us do server-side fetch.
  • The cheerio module will be useful in traverse HTML elements.
  • The africastalking module will provide us with AfricasTalking APIs, which include bulk SMS sending.
  • The dotenv module will help us to have our environment variables.
  • The moment module will help us in parsing dates.
  • The nodemon module will help in auto-restarting our local server after making local file changes.
  • The express module will help us to set up an HTTP route that an external cron will ping.

Next, we open our folder inside our favourite text-editor.I prefer Sublime-Text.
We will be writing code, adding and committing these changes to Github, it is tedious to type in these commands every time we need to upload code, and therefore, we will add the script below in our package.json file:


    "start": "nodemon index.js",
    "quickpush": "cls && git status && git add * && git commit * -m \"Updated Code\" && git push"
    

Let us test our script by running the below command in our terminal. We expect our code to be uploaded into our remote repository (Github) :

npm run quickpush

The last part of the setup will be on Heroku. We will log in to our Heroku account, create a new app, connect the code from our Github repository to the app. We do not want to deploy our app manually after every code change, and therefore, we will automate the deployment part.

Take note of the app URL because we will need it in the last step to set up a cron job.

Note: The name of our Heroku app must be universally unique. It is what Heroku uses to create a subdomain for your app, and therefore naming it govtenders just like mine will not work since I have already taken that name - find a unique name for this part.

Deploying our Github code to Heroku Cloud
Phew! We are done with the set up | Credits : giphy.com

The good stuff !

Now that we are done with the set-up we can proceed to getting data.

We want a website from which we can source tenders. I prefer https://tenders.go.ke as it is owned by the government with most (if not all) of their open-to-the-general-public tenders listed here.

We will visit the website (https://tenders.go.ke/website/tenders/index) and then inspect the DOM using Chrome-F12-Developer tools.

Notice what happens in the network tab when we request more than 50 entries on the table; JSON data loads via XHR.

We will right-click on the XHR request and copy the curl-for-bash code.

Viewing XHR on tenders.go.ke

Next, we will open Postman and paste the curl code so that we can get a well-formatted snippet of the NodeJS request code.

Take note of the request parameters' start and length. These parameters will determine how much data we get. We will copy the resulting Node-JS code snippet.

Next, we will open a new file in our text editor and name it index.js.

This file will have the code that will spin our server, fetch data from the government portal, sort through the tenders, and notify us via SMS for each tender that matches our interests.

I have wrapped the fetch request inside a promise-based function named getTenders()

Let us test if our code can fetch all the tenders by running the function below, and you can set the number of tenders you want to fetch by passing in the function arguments, for now, I will go with 0 and 10 :

getTenders(0,10)
.then((d)=>{
	console.log(d);
})
.catch((e)=>console.log('could not run'));

If you have this data on the terminal, you are on track!

Next, we will spin up a server with express.

Quick question: Why are we running the getTenders() function inside a request-response loop?

Answer: Because we need it to be triggered by a ping from an external cron service. It is an easy way for us to get things done. However, in a production app, we would do this a bit differently in using a queuing service like RabbitMQ. For this tutorial, we will keep it simple.

Let us check our progress by running the below command:

npm run start

You can open your browser on http://localhost:8080 or http://localhost:3000 or whatever port you have set. You should see the index page.

Next, the data we have fetched is of a data-type string, and we need to convert it to an object so that we can extract its properties. We will also call in the moment module so that we can work with dates in filtering of the tenders that have been posted in less than 24 hours and those that are closing in less than 48 hours.

The next part involves ReGex - if you do not know how ReGex works do not worry, soon I will do a tutorial on ReGex, but for now, I will explain what the ReGex section of my code does.

var ourInterests = /Supply|Internet|installation|School|Hospital/gi;

For example, say our company deals with the supply of equipment to schools and hospitals and also does installation of internet & related software.

The regex above will capture every word that has the terms; supply, internet, installation, school, or hospital.

The /g flag means global, and it tells ReGex to execute the search for a match globally, that is, not to stop after the first match.

The /i flag means insensitive, and it tells ReGex to match a text regardless of whether it is an uppercase or lowercase.

Running the RegEx method ourInterests.test('some unfiltered string data in here') returns a Boolean; either true if it finds a match or false if it does not.

In the video below, we utilize the power of RegEx to filter tenders of interest. We will do this inside the /findtenders route.

Next, we will use Cheerio to extract the link of a tender from HTML. Cheerio works like jQuery, where once you have loaded your HTML through the load method, you can manipulate HTML like you would using jQuery.

Side note; Cheerio delivers a tonne of value in web scraping, to learn more about Cheerio check out its documentation.

And now the sweet part : sending of SMSes , don't I love this!

Africa's Talking has provided a simple-to-use and user-intuitive SMS API that you can use to send bulk SMSes.

In the video that follows, you need to have an API key and username from Africa's Talking. You can generate one by following this quick guide https://help.africastalking.com/en/articles/2249244-what-is-my-username-and-api-key.

Paste your username and API key in the .env file using the variables AT_username and AT_apiKey, respectively.

IMPORTANT : These two are what we consider sensitive information - look here , you do not want a malicious person using your airtime to send SMSes, so in our .gitignore file, we will add a line to tell git not to push our .env file into the Github repository. Just below the line node_modules/, paste the line below inside the .gitignore file.

.env

We also need to save these credentials in a secure place in the Heroku dashboard called config-vars. If you are not sure how to do this, check out this tutorial https://devcenter.heroku.com/articles/config-vars .

Let us dive into the code ,

Next, we upload our code to Github using the below command on the terminal.

npm run quickpush

If you turned on auto-deploy on Heroku, your app should deploy without your intervention. If you did not, you need to click the deploy button on the Heroku dashboard.

The next and final step is setting up your external cron.

For this, I will use a website called Cron-Job.org https://cron-job.org/

Remember the URL assigned for your Heroku app? We need it now, mine was https://govtenders.herokuapp.com/.

Create your cron-job.org account by signing up on https://cron-job.org/en/signup/

Then log in via https://cron-job.org/en/members/

Then click on the Cron jobs tab: https://cron-job.org/en/members/jobs/

I want my script to send me all new tenders that match my interests every day at 6:35 PM (I could set it any time ; 6:35 fits my schedule) and also remind me when a closing date of a tender is within 48 hours.

Check out the below video showing the last step on how to set up a cron that pings my site and triggers my script every day at 6:35 PM (18:35):

Drum rolls please :

If you did everything right, you are going to get an SMS message.

Screenshot of a succesful SMS

If you missed a step, you can compare your code with mine below.

.env file

AT_username = ab96d932717fbbbaaaabbbcccddeeefff29020987365624skoisoi
AT_apiKey = govtendersismyusername

.gitignore file

node_modules/ 
package-lock.json 
*.lock 

package.json

{
  "name": "govtenders",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "nodemon index.js",
    "quickpush": "cls && git status && git add * && git commit * -m \"Updated Code\" && git push"
  },
  "keywords": [
    
  ],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "africastalking": "^0.4.7",
    "cheerio": "^1.0.0-rc.3",
    "dotenv": "^8.2.0",
    "express": "^4.17.1",
    "moment": "^2.27.0",
    "nodemon": "^2.0.4",
    "request": "^2.88.2"
  }
}

View : index.js https://gist.github.com/DaggieBlanqx/b5e622c981db923754c137d99a89f621

I have made the entire project available on Github . you can clone and fiddle with it.

Conclusion & final thoughts :

Final thoughts | Credits: giphy.com

Our tutorial talks about simple & minute automation, but when you look at serial automation in a broader sense - at what it can do to jobs out there, it becomes a double-edged sword.

On one hand it cuts costs by saving time and money for businesses while on the other , it minimizes the need for hired human labor - and people who depend on these jobs end up losing their sources of livelihood.

Sensitive as it could be, I want to hear your thoughts on this.

Do not forget to follow me on Twitter @daggieblanqx .

A lot more is coming but for now , bye | Credits: giphy.com
You've successfully subscribed to Decoded For Devs
Welcome back! You've successfully signed in.
Great! You've successfully signed up.
Your link has expired
Success! Your account is fully activated, you now have access to all content.