Building and Automating SEO Reports with Python and Twilio

Getting high quality reports in a timely fashion is very important to help make the right decisions. Especially in SEO if you are running a business through a website. The web moves fast and a internal decision at Google may have huge repercussions on your or your clients rankings.

In this article we will combine python, MongoDB and the Twilio service to automate the production and sending of SEO reports via WhatsApp.

Obviously the code in this article could be adapted to produce any kind of report. I build my reports using SEO data using babbar.tech and yourtext.guru API. You could of course switch to any other API to build your reports.

We will build a very small API to add or remove websites from the reports. We will interact with it using WhatsApp which require to have both a twilio account and a professional WhatsApp account. You can follow this procedure to set up both. Also the reporting will be done via WhatsApp so it is very important to have those tools set up.

This API will be available through WhatsApp making it a natural interface for your customers to add or remove information to their reports. Clients can simply use the messaging service not only to receive the reports but also to add or remove websites to/from their report. They will get the reporting they want without using a complicated interface and having to remember another login/password for another service.

The report will contains classic SEO information on the website included in it. We will get the main metrics from Babbar to have a general understanding of the SEO of the website. We will also include a few pages from the sites that are on the second page of Google search results and get the optimization scores from Yourtext.guru for both the the webpage and the first page of Google search results. This will help the customer see the effort required to get a higher ranking on Google for the specific keyword. With all this information for each website, your customers will be able to see if there is an immediate danger on their website and if there are some actions they can take to improve the rankings of webpages that are almost on the first page of Google search results. We do not present all the keywords the website is ranked on as it will too much information for a WhatsApp report. Here we focus on specific keywords that may offer great opportunities with smaller SEO efforts.

Here are the steps we are going to follow through this article

Build the SEO report
1. Get general information on the websites SEO performance
2. Get specific page rankings
3. Get the content optimization scores of these pages and their competition
Setup and API using flask
Interact with the API using WhatsApp

Table of Contents

Building a SEO report

Our first task will be to build an SEO report for a specific host. We will use both babbar.tech and yourtext.guru API to get general metrics about the host as well as specific quality metrics on some urls from the host that are ranked on the second page of Google search results. The report will also include informations about the first page of the Google search results to mesure the effort required to access the first page of the search results.

Accessing the babbar.tech overview data for a host is done using the /host/overview/main route.

Python

def get_babbar_data(host: str) -> dict:
"""We will get the host data from babbar.tech"""

BABBAR_OVERVIEW = f"https://www.babbar.tech/api/host/overview/main?api_token={os.environ.get('BABBAR_KEY')}"
BABBAR_KEYWORDS = f"https://www.babbar.tech/api/host/keywords?api_token={os.environ.get('BABBAR_KEY')}"

keys = ["health", "hostValue", "hostTrust", "semanticValue", "babbarAuthorityScore"]

res = requests.post(BABBAR_OVERVIEW, json={"host": host})

if res.status_code == requests.codes.OK:
	data = res.json()
	host_data = {key: data[key] for key in keys}
else:
	return None

This function takes a host as an argument and tries to fetch the general information available at babbar.tech regarding this host.

This is the information we will include in the report. The Host Value is a popularity metric, the Host Trust is self explanatory and the Semantic Value is also a popularity value takes into account the semantic similarity of the pages to distribute popularity. The Http Health metric is the percentage of Ok codes received when fetching the page and the BAS is an aggregative metric that takes into account, the popularity, the trust and the spamminess of the host. We then ask babbar.tech for 3 pages from the website that are ranked between positions 11 and 20 in Google with the associated keyword. We will store the url, the ranking and the associated keyword.

Python

res = requests.post(BABBAR_KEYWORDS,
                    json={"host": host,
                    "lang": "fr",
                    "country": "FR",
                    "n"： 3，
                    "min": 11,
                    "max": 20})
if res.status_code == requests.codes.OK:
    data = res.json()
    host_data["keywords"] = [{key: e[key] for key in ["rank", "keywords", "url"]} for e in data["entries"]]
return host_data

I only take 3 pages as this is a toy example and also this report will be accessible through WhatsApp messages and therefore should stay succinct to be readable. Now that we have the information regarding the rankings the next step is to get the semantic information from Yourtext.guru. The first step is to order the redaction guides. A redaction guide could be seen as the vocabulary that should be present on a page if we want it to rank on a specific keyword.

Python

def get_ytg_scores (keywords: dict) -> dict:
"""We will get the optimisation scores from ytg for the urls positionned on the keywords we got from babbar.tech"""

    URL_GUIDE = "https://yourtext.guru/api/guide"
    URL_SCORE = "https://yourtext.guru/api/check"
    URL-SERPS = "https://yourtext.guru/api/serp"

    guide_ids = []

    for kw in keywords:
        res = requests.post(URL_GUIDE,
                            headers={"KEY": os.environ.get('YTG_KEY')},
                            json={"query": kw["keywords"],
                            "type": "premium",
                            "lang": "fr_fr"})	
        if res.status_code == requests.codes.OK:
        	guide_ids.append((kw["url"], res.json()["guide_id"]))

Once we have this redaction guide we can check the content of the webpage against the specific vocabulary to get its optimization score. We will also get the keyword first page of Google search results to get the optimization scores of the competition.

We extract the content of the page using trafilatura to get only the “main” content. The yourtext.guru API will send us back a lot of precise information but we are only interested in the optimization score of the content and its danger score (that could be seen as the overoptimization for search engines).

Python

for (url, g_id) in ready:
    dl = trafilatura.fetch_url(url)
    content = trafilatura.extract(dl)
    res = requests.post(URL_SCORE + f"/{g_id}",
				        headers={"KEY": os.environ.get('YTG_KEY')},
                        json={"content": content})
    if res.status_code == requests.codes.OK:
        data = res.json()
        results[url] = {"soseo": data["score"],
        "dseo": data["danger"],
        "avg3": [0, 0].
        "avg10": [0, 0],
        "pos10": [0, 1]}

We also get the optimization and danger scores of the first page of Google search results for the specified keyword. We use this information to include in our report to the optimization and danger of the 10th position, the average of the first page of the search results and the average of the first three positions. These scores are an indication of the effort required on text optimization to make it to the first page.

We now have all our information for our report and can write it before sending it to the user.

Python

res = requests.get(URL_SERPS + f"/{g_id}", headers={"KEY": os.environ.get('YTG_KEY')})
if res.status_code == requests.codes.OK:
    data = res.json()
    for s in data["serps"]:
        if s["position"] < 4:
            results[url]["avg3"][0] += s["scores"]["soseo_main_content"] / 3
            results[url]["avg3"][1] += s["scores"]["dseo_main_content"] / 3
            results[url]["avg10"][0] += s["scores"]["sose_main_content"] / 10
            results[url]["avg10"][1] += s["scores"]["dseo_main_content"] / 10
        if ["position"] == 10:
            results[url]["pos10"] = (s["scores"]["soseo main content"], s["scores"]["dseo main content"]

Writing the report is a simple matter of formatting a string with all the information we have.

Python

report = f"""Here is the report for the host {host}:
– Health: {report_data["health"]}
– Host Value: {report_data["hostValue"]}
– Host Trust: {report_data["hostTrust"]}
– Semantic Value: {report_data["semanticValue"]}
– Babbar Authority Score: {report_data[ "babbarAuthorityScore"]}
"""

for kw in report_data["keywords"]:
    report += f"""On keyword: {kw["keywords"]}, your URL {kw["url"]} is ranked {kw["rank"]}."""
    if 'ytg' not in kw: 
		continue
    report +=f"""
Its YTG scores are:
– SOSEO: {kw["ytg"] ["soseo" ]}
– DSEO: {kw["ytg"] ["dseo"]}.
The scores for the 10th position are:
– SOSEO: {kw["ytg"] ["pos10"] [0]}
– DSEO: {kw["ytg"] ["pos10"] [1]}.
The average scores for the 10 first positions are:
– SOSEO: {round(kw["ytg"] ["avg10"][0], 2)}
– DSEO: {round(kw["ytg"] ["avg10"] [1], 2)}.
The average scores for the 3 first positions are:
– SOSEO: {round(kw["ytg"]["avg3"] [0], 2)}
– DSEO: {round(kw["ytg"] ["avg3"][1], 2)}.
"""

We are now ready to write the API so we can interact with it through WhatsApp. We will use the Flask framework to build the API as it is very simple and does the job perfectly.

WhatsApp as an interface

We build a very simple API that only exposes one route. This route will receive messages sent by our users via WhatsApp. There are several things to unpack here. Since this route will be used by the WhatsApp service Twilio it needs to be accessible through either the POST of PUT method. I chose the POST method as it is better known that the PUT equivalent.

Twilio will send three fields formatted as form data that are very important to interact with it

From: This field represents the number of the end that sent us the message it is of the form “whatsapp:XXXXXXXX” since we have set up a WhatsApp sender on Twilio.
To: This field represents the number of the WhatsApp sender from Twilio. It is the number the customer sent its message to. The customer expects to receive an answer from this number and it should be used to send replies.
Body: This is the body of the message. It contains the command sent by the customer.

Python

@app.route("/weeklyreport", methods=["POST"])
def weekly_report():
    fr = request.form["From"]
    to = request.form["To"]
    body = request.form["Body"]
    if body.startswith("ADD "):
        add_host_to_report(body.split(' ')[1], fr)
        send_message(fr, to, f"{body.split(' ')[1]} has been added to the weekly report.")
    elif body startswith("DEL "):
        delete_host_from_report(body.split(' ')[1], fr)
        send_message(fr, to, f"{body.split(' ')[1]} has been removed from the weekly report.")
    else:
        send_message(fr, to, "Sorry, I didn't understand your message.")
    return "<Response></Response>"

The API will only accept two kinds of messages:

ADD host, that will add the host for the next reports

DEL host, that will remove the host for the next reports

If any other message is received the API will simply answer that is doesn’t understand the command.

The API is therefore super simple. The function send_message is depicted below. It simply creates a twilio.rest.Client using an account SID and an authorization token. Since the messages we are sending with this function are quite short we do not need to worry about splitting them before sending them.

Python

def send_message(to, from_, body):
    print(f"Sending message to {to} from {from_}: {body}")
    client = Client(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN)
    client.messages.create(to=to, from_=from_, body=body)

The API route ends by sending an empty response to Twilio. It is easier to use the python client than to try to format the messages ourselves.

Managing hosts for the reports

We will use MongoDB to store the hosts that should be included in the reports. We will store the hosts together with the number of the user so we’ll know where to send the reports. We will use the pymongo package to requests the MongoDB server.

We are using to python functions to interact with the pymongo server add_host_to_report and delete_host_from_report. Both functions access the collection “host” in the database “reports” in the MongoDB server. The first function adds the pair (host, user) to the collection while the second one deletes it from it.

Python

def add_host_to_report(host: str, user: str):
    """We add the host to MongoDB so it is included in future reports."""
    client = MongoClient()
    db = client.reports
    collection = db.hosts
    collection.insert_one({"host": host, "user": user})

def delete_host_from_report(host: str, user: str):
    """We delete the host from MongoDB so it is not included in future reports."""
    client = MongoClient()
    db = client.reports
    collection = db.hosts
    collection.delete_one({"host": host, "user": user})

Sending the reports

The reports are sent in an automated manner not because the customer ask for them. So we use a Python script that will simply scan the whole MongoDB collection and build a report for every host it finds before sending it to the user.

The reports are then sent through WhatsApp. Since these reports might be too long they are broken into 1000 characters messages that are sent with 1 second time intervals not to clog the Twilio service. Here since we only have one WhatsApp sender active on Twilio we have an hardcoded value for the from_ argument of the function. If you were to have multiple ones (eg. one for each country) you could easily add that information in MongoDB.

Python

twilio = Client(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN)
client = MongoClient()
db = client.reports
collection = db.hosts
hosts = collection.find()
for host in hosts:
    report = build_report(host["host"])
    messages = textwrap.wrap(report, 1000)
    for m in messages:
        twilio.messages.create(to=host["user"], from_="whatsapp:+XXX', body=m) 
    	time.sleep (1)

The message are then received on the user’s WhatsApp

In order to program the construction and the sending of the reports one just need to add the execution of the script to their crontab.

0 9 * * 1 /usr/bin/python3 /path/to/your/script/app.py

This line will trigger the report sending every monday at 9am.

What next ?

Now that we have a simple structure for automating reports it is possible to duplicate it to send other kind of reports to customers through WhatsApp. The reports are also pretty simple and don’t have many option for personalization other than adding and removing websites it is possible to extend the command the clients can send via message to get more personalization. Connecting a LLM behind the API could also help the customers use natural language instead of commands to interact with the API and will help with their user experience. Using natural language in a messaging service feels more natural to the end user but is outside the scope of this article.

Building and Automating SEO Reports with Python and Twilio

Building a SEO report

WhatsApp as an interface

Managing hosts for the reports

Sending the reports

What next ?

You may like to read:

Help & Info