path: root/posts.org

#+hugo_base_dir: ./
#+hugo_section: ./posts

#+hugo_weight: auto
#+hugo_auto_set_lastmod: t

#+author: Roger Gonzalez

* Programming :programming:
:PROPERTIES:
:ID:       622d1d7a-cef9-4eb6-838c-552086182fec
:END:
All posts in here will have the category set to /programming/.
** Using MinIO to upload to a local S3 bucket in Django :python::django::minio::docker::dockercompose:
:PROPERTIES:
:ID:       b693b4e8-0550-4238-8a64-30866a47768a
:EXPORT_FILE_NAME: using-minio-to-upload-to-a-local-s3-bucket-in-django
:EXPORT_DATE: 2021-01-10
:END:

So MinIO its an object storage that uses the same API as S3, which means that we
can  use the same S3 compatible libraries in Python, like [[https://pypi.org/project/boto3/][Boto3]] and [[https://pypi.org/project/django-storages/][django-storages]].

*** The setup
:PROPERTIES:
:ID:       f2ed5f13-5a27-4da8-a8d2-72500d652ba1
:END:

Here's the docker-compose configuration for my django app:

#+begin_src yaml
version: "3"

services:
  app:
    build:
      context: .
    volumes:
      - ./app:/app
    ports:
      - 8000:8000
    depends_on:
      - minio
    command: >
      sh -c "python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8000"

  minio:
    image: minio/minio
    ports:
      - 9000:9000
    environment:
      - MINIO_ACCESS_KEY=access-key
      - MINIO_SECRET_KEY=secret-key
    command: server /export

  createbuckets:
    image: minio/mc
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      apk add nc &&
      while ! nc -z minio 9000; do echo 'Wait minio to startup...' && sleep 0.1; done; sleep 5 &&
      /usr/bin/mc config host add myminio http://minio:9000 access-key secret-key;
      /usr/bin/mc mb myminio/my-local-bucket;
      /usr/bin/mc policy download myminio/my-local-bucket;
      exit 0;
      "
#+end_src

- ~app~ is my Django app. Nothing new here.
- ~minio~ is the MinIO instance.
- ~createbuckets~ is a quick instance that creates a new bucket on startup, that
  way we don't need to create the bucket manually.

On my app, in ~settings.py~:

#+begin_src python
# S3 configuration

DEFAULT_FILE_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"

AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID", "access-key")
AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY", "secret-key")
AWS_STORAGE_BUCKET_NAME = os.environ.get("AWS_STORAGE_BUCKET_NAME", "my-local-bucket")

if DEBUG:
    AWS_S3_ENDPOINT_URL = "http://minio:9000"
#+end_src

If we were in a production environment, the ~AWS_ACCESS_KEY_ID~,
~AWS_SECRET_ACCESS_KEY~ and ~AWS_STORAGE_BUCKET_NAME~ would be read from the
environmental variables, but since we haven't set those up and we have
~DEBUG=True~, we are going to use the default ones, which point directly to
MinIO.


And that's it! That's everything you need to have your local S3 development environment. 

*** Testing
:PROPERTIES:
:ID:       61f1cffa-59de-405e-853b-57547a96165b
:END:

First, let's create our model. This is a simple mock model for testing purposes:

#+begin_src python
from django.db import models


class Person(models.Model):
    """This is a demo person model"""

    first_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)
    date_of_birth = models.DateField()
    picture = models.ImageField()

    def __str__(self):
        return f"{self.first_name} {self.last_name} {str(self.date_of_birth)}"
#+end_src

Then, in the Django admin we can interact with our new model:

[[/2021-01-10-135111.png]]

[[/2021-01-10-135130.png]]


If we go to the URL and change the domain to ~localhost~, we should be able to
see the picture we uploaded.

[[/2021-01-10-140016.png]]

*** Bonus: The MinIO browser
:PROPERTIES:
:ID:       40acc926-5083-4682-b9be-63cac1c253cb
:END:
MinIO has a local objects browser. If you want to check it out you just need to
go to http://localhost:9000. With my docker-compose configuration, the
credentials are:

#+begin_src bash
username: access-key
password: secret-key
#+end_src

[[/2021-01-10-140236.png]]

On the browser, you can see your uploads, delete them, add new ones, etc.

[[/2021-01-10-140337.png]]

*** Conclusion
:PROPERTIES:
:ID:       f5384bdc-9100-46eb-81f8-d6c8a8f52ba8
:END:

Now you can have a simple configuration for your local and production
environments to work seamlessly, using local resources instead of remote
resources that might generate costs for the development.

If you want to check out the project code, you can check in my Gitlab here:
https://gitlab.com/rogs/minio-example

See you in the next one!

** How to create a celery task that fills out fields using Django :python::celery::django::docker::dockercompose:
:PROPERTIES:
:EXPORT_FILE_NAME: how-to-create-a-celery-task-that-fills-out-fields-using-django
:EXPORT_DATE: 2020-11-29T15:48:48-03:00
:ID:       f8ed204b-1f57-4c92-8c4f-128658327aed
:END:

Hi everyone!

It's been way too long, I know. In this oportunity, I wanted to talk about
asynchronicity in Django, but first, lets set up the stage: 

Imagine you are working in a library and you have to develop an app that allows
users to register new books using a barcode scanner. The system has to read the
ISBN code and use an external resource to fill in the information (title, pages,
authors, etc.). You don't need the complete book information to continue, so the
external resource can't hold the request.

*How can you process the external request asynchronously?* 🤔 

For that, we need Celery.

*** What is Celery?
:PROPERTIES:
:ID:       d3f8a2ad-09c2-4ae7-8b84-50210af7a2dc
:END:
[[https://docs.celeryproject.org/en/stable/][Celery]] is a "distributed task queue". Fron their website:

> Celery is a simple, flexible, and reliable distributed system to process vast
amounts of messages, while providing operations with the tools required to
maintain such a system. 

So Celery can get messages from external processes via a broker (like [[https://redis.io/][Redis]]),
and process them.

The best thing is: Django can connect to Celery very easily, and Celery can
access Django models without any problem. Sweet!

*** Lets code!
:PROPERTIES:
:ID:       0e6a021e-ab2a-48d0-92a0-39fd4f7c3409
:END:
Let's assume our project structure is the following:
#+begin_src 
- app/
  - manage.py
  - app/
    - __init__.py
    - settings.py
    - urls.py
#+end_src

**** Celery
:PROPERTIES:
:ID:       77b6e575-bc24-4ad3-b504-74bdef9145d3
:END:
First, we need to set up Celery in Django. Thankfully, [[https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html#using-celery-with-django][Celery has an excellent
documentation]], but the entire process can be summarized to this:

In ~app/app/celery.py~:
#+begin_src python
import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")

app = Celery("app")

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object("django.conf:settings", namespace="CELERY")

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    """A debug celery task"""
    print(f"Request: {self.request!r}")
#+end_src

What's going on here?
- First, we set the ~DJANGO_SETTINGS_MODULE~ environment variable
- Then, we instantiate our Celery app using the ~app~ variable.
- Then, we tell Celery to look for celery configurations in the Django settings
  with the ~CELERY~ prefix. We will see this later in the post.
- Finally, we start Celery's ~autodiscover_tasks~. Celery is now going to look for
  ~tasks.py~ files in the Django apps.

In ~/app/app/__init__.py~:
#+begin_src python
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ("celery_app",)
#+end_src

Finally in ~/app/app/settings.py~:
#+begin_src python
...
# Celery
CELERY_BROKER_URL = env.str("CELERY_BROKER_URL")
CELERY_TIMEZONE = env.str("CELERY_TIMEZONE", "America/Montevideo")
CELERY_RESULT_BACKEND = "django-db"
CELERY_CACHE_BACKEND = "django-cache"
...
#+end_src

Here, we can see that the ~CELERY~ prefix is used for all Celery configurations,
because on ~celery.py~ we told Celery the prefix was ~CELERY~

With this, Celery is fully configured. 🎉 

**** Django
:PROPERTIES:
:ID:       dd40e5c4-7d82-4d3c-b5da-a7c8751b9f70
:END:

First, let's create a ~core~ app. This is going to be used for everything common
in the app
#+begin_src bash
$ python manage.py startapp core
#+end_src

On ~core/models.py~, lets set the following models:
#+begin_src python
"""
Models
"""
import uuid

from django.db import models


class TimeStampMixin(models.Model):
    """
    A base model that all the other models inherit from.
    This is to add created_at and updated_at to every model.
    """

    id = models.UUIDField(primary_key=True, default=uuid.uuid4)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        """Setting up the abstract model class"""

        abstract = True


class BaseAttributesModel(TimeStampMixin):
    """
    A base model that sets up all the attibutes models
    """

    name = models.CharField(max_length=255)
    outside_url = models.URLField()

    def __str__(self):
        return self.name

    class Meta:
        abstract = True
#+end_src

Then, let's create a new app for our books:
#+begin_src bash
python manage.py startapp books
#+end_src

And on ~books/models.py~, let's create the following models:
#+begin_src python
"""
Books models
"""
from django.db import models

from core.models import TimeStampMixin, BaseAttributesModel


class Author(BaseAttributesModel):
    """Defines the Author model"""


class People(BaseAttributesModel):
    """Defines the People model"""


class Subject(BaseAttributesModel):
    """Defines the Subject model"""


class Book(TimeStampMixin):
    """Defines the Book model"""

    isbn = models.CharField(max_length=13, unique=True)
    title = models.CharField(max_length=255, blank=True, null=True)
    pages = models.IntegerField(default=0)
    publish_date = models.CharField(max_length=255, blank=True, null=True)
    outside_id = models.CharField(max_length=255, blank=True, null=True)
    outside_url = models.URLField(blank=True, null=True)
    author = models.ManyToManyField(Author, related_name="books")
    person = models.ManyToManyField(People, related_name="books")
    subject = models.ManyToManyField(Subject, related_name="books")

    def __str__(self):
        return f"{self.title} - {self.isbn}"
#+end_src

~Author~, ~People~, and ~Subject~ are all ~BaseAttributesModel~, so their fields
come from the class we defined on ~core/models.py~.

For ~Book~ we add all the fields we need, plus a ~many_to_many~ with Author,
People and Subjects. Because:

- /Books can have many authors, and many authors can have many books/
Example: [[https://www.epicreads.com/blog/ya-books-multiple-authors/][27 Books by Multiple Authors That Prove the More, the Merrier]]
- /Books can have many persons, and many persons can have many books/
Example: Ron Weasley is in several /Harry Potter/ books
- /Books can have many subjects, and many subjects can have many books/
Example: A book can be a /comedy/, /fiction/, and /mystery/ at the same time

Let's create ~books/serializers.py~:
#+begin_src python
"""
Serializers for the Books
"""
from django.db.utils import IntegrityError
from rest_framework import serializers

from books.models import Book, Author, People, Subject
from books.tasks import get_books_information


class AuthorInBookSerializer(serializers.ModelSerializer):
    """Serializer for the Author objects inside Book"""

    class Meta:
        model = Author
        fields = ("id", "name")


class PeopleInBookSerializer(serializers.ModelSerializer):
    """Serializer for the People objects inside Book"""

    class Meta:
        model = People
        fields = ("id", "name")


class SubjectInBookSerializer(serializers.ModelSerializer):
    """Serializer for the Subject objects inside Book"""

    class Meta:
        model = Subject
        fields = ("id", "name")


class BookSerializer(serializers.ModelSerializer):
    """Serializer for the Book objects"""

    author = AuthorInBookSerializer(many=True, read_only=True)
    person = PeopleInBookSerializer(many=True, read_only=True)
    subject = SubjectInBookSerializer(many=True, read_only=True)

    class Meta:
        model = Book
        fields = "__all__"


class BulkBookSerializer(serializers.Serializer):
    """Serializer for bulk book creating"""

    isbn = serializers.ListField()

    def create(self, validated_data):
        return_dict = {"isbn": []}
        for isbn in validated_data["isbn"]:
            try:
                Book.objects.create(isbn=isbn)
                return_dict["isbn"].append(isbn)
            except IntegrityError as error:
                pass

        return return_dict

    def update(self, instance, validated_data):
        """The update method needs to be overwritten on
        serializers.Serializer. Since we don't need it, let's just
        pass it"""
        pass


class BaseAttributesSerializer(serializers.ModelSerializer):
    """A base serializer for the attributes objects"""

    books = BookSerializer(many=True, read_only=True)


class AuthorSerializer(BaseAttributesSerializer):
    """Serializer for the Author objects"""

    class Meta:
        model = Author
        fields = ("id", "name", "outside_url", "books")


class PeopleSerializer(BaseAttributesSerializer):
    """Serializer for the Author objects"""

    class Meta:
        model = People
        fields = ("id", "name", "outside_url", "books")


class SubjectSerializer(BaseAttributesSerializer):
    """Serializer for the Author objects"""

    class Meta:
        model = Subject
        fields = ("id", "name", "outside_url", "books")

#+end_src
The most important serializer here is ~BulkBookSerializer~. It's going to get an
ISBN list and then bulk create them in the DB.

On ~books/views.py~, we can set the following views:
#+begin_src python
"""
Views for the Books
"""
from rest_framework import viewsets, mixins, generics
from rest_framework.permissions import AllowAny

from books.models import Book, Author, People, Subject
from books.serializers import (
    BookSerializer,
    BulkBookSerializer,
    AuthorSerializer,
    PeopleSerializer,
    SubjectSerializer,
)


class BookViewSet(
    viewsets.GenericViewSet,
    mixins.ListModelMixin,
    mixins.RetrieveModelMixin,
):
    """
    A view to list Books and retrieve books by ID
    """

    permission_classes = (AllowAny,)
    queryset = Book.objects.all()
    serializer_class = BookSerializer


class AuthorViewSet(
    viewsets.GenericViewSet,
    mixins.ListModelMixin,
    mixins.RetrieveModelMixin,
):
    """
    A view to list Authors and retrieve authors by ID
    """

    permission_classes = (AllowAny,)
    queryset = Author.objects.all()
    serializer_class = AuthorSerializer


class PeopleViewSet(
    viewsets.GenericViewSet,
    mixins.ListModelMixin,
    mixins.RetrieveModelMixin,
):
    """
    A view to list People and retrieve people by ID
    """

    permission_classes = (AllowAny,)
    queryset = People.objects.all()
    serializer_class = PeopleSerializer


class SubjectViewSet(
    viewsets.GenericViewSet,
    mixins.ListModelMixin,
    mixins.RetrieveModelMixin,
):
    """
    A view to list Subject and retrieve subject by ID
    """

    permission_classes = (AllowAny,)
    queryset = Subject.objects.all()
    serializer_class = SubjectSerializer


class BulkCreateBook(generics.CreateAPIView):
    """A view to bulk create books"""

    permission_classes = (AllowAny,)
    queryset = Book.objects.all()
    serializer_class = BulkBookSerializer
#+end_src

Easy enough, endpoints for getting books, authors, people and subjects and an
endpoint to post ISBN codes in a list.

We can check swagger to see all the endpoints created:

[[/2020-11-29-115634.png]]

Now, *how are we going to get all the data?* 🤔 

*** Creating a Celery task
:PROPERTIES:
:ID:       ac678884-7d0e-46fd-91e2-ec0e0edd12a9
:END:
Now that we have our project structure done, we need to create the asynchronous
task Celery is going to run to populate our fields.

To get the information, we are going to use the [[https://openlibrary.org/dev/docs/api/books"""][OpenLibrary API]].

First, we need to create ~books/tasks.py~:
#+begin_src python
"""
Celery tasks
"""
import requests
from celery import shared_task

from books.models import Book, Author, People, Subject


def get_book_info(isbn):
    """Gets a book information by using its ISBN.
    More info here https://openlibrary.org/dev/docs/api/books"""
    return requests.get(
        f"https://openlibrary.org/api/books?jscmd=data&format=json&bibkeys=ISBN:{isbn}"
    ).json()


def generate_many_to_many(model, iterable):
    """Generates the many to many relationships to books"""
    return_items = []
    for item in iterable:
        relation = model.objects.get_or_create(
            name=item["name"], outside_url=item["url"]
        )
        return_items.append(relation)
    return return_items


@shared_task
def get_books_information(isbn):
    """Gets a book information"""

    # First, we get the book information by its isbn
    book_info = get_book_info(isbn)

    if len(book_info) > 0:
        # Then, we need to access the json itself. Since the first key is dynamic,
        # we get it by accessing the json keys
        key = list(book_info.keys())[0]
        book_info = book_info[key]

        # Since the book was created on the Serializer, we get the book to edit
        book = Book.objects.get(isbn=isbn)

        # Set the fields we want from the API into the Book
        book.title = book_info["title"]
        book.publish_date = book_info["publish_date"]
        book.outside_id = book_info["key"]
        book.outside_url = book_info["url"]

        # For the optional fields, we try to get them first
        try:
            book.pages = book_info["number_of_pages"]
        except:
            book.pages = 0

        try:
            authors = book_info["authors"]
        except:
            authors = []

        try:
            people = book_info["subject_people"]
        except:
            people = []

        try:
            subjects = book_info["subjects"]
        except:
            subjects = []

        # And generate the appropiate many_to_many relationships
        authors_info = generate_many_to_many(Author, authors)
        people_info = generate_many_to_many(People, people)
        subjects_info = generate_many_to_many(Subject, subjects)

        # Once the relationships are generated, we save them in the book instance
        for author in authors_info:
            book.author.add(author[0])

        for person in people_info:
            book.person.add(person[0])

        for subject in subjects_info:
            book.subject.add(subject[0])

        # Finally, we save the Book
        book.save()

    else:
        raise ValueError("Book not found")
#+end_src

So when are we going to run this task? We need to run it in the *serializer*.

On ~books/serializers.py~:
#+begin_src python
from books.tasks import get_books_information
...
class BulkBookSerializer(serializers.Serializer):
    """Serializer for bulk book creating"""

    isbn = serializers.ListField()

    def create(self, validated_data):
        return_dict = {"isbn": []}
        for isbn in validated_data["isbn"]:
            try:
                Book.objects.create(isbn=isbn)
                # We need to add this line
                get_books_information.delay(isbn)
                #################################
                return_dict["isbn"].append(isbn)
            except IntegrityError as error:
                pass

        return return_dict

    def update(self, instance, validated_data):
        pass
#+end_src

To trigger the Celery tasks, we need to call our function with the ~delay~
function, which has been added by the ~shared_task~ decorator. This tells Celery
to start running the task in the background since we don't need the result
right now.

*** Docker configuration
:PROPERTIES:
:ID:       3d992b01-11f9-488c-9781-dcddfcf6fe88
:END:
There are a lot of moving parts we need for this to work, so I created a
~docker-compose~ configuration to help with the stack. I'm using the package
[[https://github.com/joke2k/django-environ][django-environ]] to handle all environment variables.

On ~docker-compose.yml~:
#+begin_src yaml
version: "3.7"

x-common-variables: &common-variables
  DJANGO_SETTINGS_MODULE: "app.settings"
  CELERY_BROKER_URL: "redis://redis:6379"
  DEFAULT_DATABASE: "psql://postgres:postgres@db:5432/app"
  DEBUG: "True"
  ALLOWED_HOSTS: "*,test"
  SECRET_KEY: "this-is-a-secret-key-shhhhh"

services:
  app:
    build:
      context: .
    volumes:
      - ./app:/app
    environment:
      <<: *common-variables
    ports:
      - 8000:8000
    command: >
      sh -c "python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8000"
    depends_on:
      - db
      - redis

  celery-worker:
    build:
      context: .
    volumes:
      - ./app:/app
    environment:
      <<: *common-variables
    command: celery --app app worker -l info
    depends_on:
      - db
      - redis

  db:
    image: postgres:12.4-alpine
    environment:
      - POSTGRES_DB=app
      - POSRGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres

  redis:
    image: redis:6.0.8-alpine

#+end_src
This is going to set our app, DB, Redis, and most importantly our celery-worker
instance. To run Celery, we need to execute:
#+begin_src bash
$ celery --app app worker -l info
#+end_src

So we are going to run that command on a separate docker instance

*** Testing it out
:PROPERTIES:
:ID:       50e6cc5d-848b-4d73-a4b6-1bfd23bf86f4
:END:
If we run
#+begin_src bash
$ docker-compose up
#+end_src

on our project root folder, the project should come up as usual. You should be
able to open http://localhost:8000/admin and enter the admin panel.

To test the app, you can use a curl command from the terminal:
#+begin_src bash
curl -X POST "http://localhost:8000/books/bulk-create" -H  "accept: application/json" \
    -H  "Content-Type: application/json" -d "{  \"isbn\": [ \"9780345418913\", \
    \"9780451524935\", \"9780451526342\", \"9781101990322\", \"9780143133438\"   ]}"
#+end_src
[[/2020-11-29-124654.png]]

This call lasted 147ms, according to my terminal.

This should return instantly, creating 15 new books and 15 new Celery tasks, one
for each book. You can also see tasks results in the Django admin using the
~django-celery-results~ package, check its [[https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html#django-celery-results-using-the-django-orm-cache-as-a-result-backend][documentation]].

[[/2020-11-29-124734.png]]

Celery tasks list, using ~django-celery-results~

[[/2020-11-29-124751.png]]

Created and processed books list

[[/2020-11-29-124813.png]]

Single book information

[[/2020-11-29-124834.png]]

People in books

[[/2020-11-29-124851.png]]

Authors

[[/2020-11-29-124906.png]]

Themes

And also, you can interact with the endpoints to search by author, theme,
people, and book. This should change depending on how you created your URLs.

*** That's it!
:PROPERTIES:
:ID:       e7cf2cfe-25ae-472d-b4a2-d154834cce98
:END:
This surely was a *LONG* one, but it has been a very good one in my opinion.
I've used Celery in the past for multiple things, from sending emails in the
background to triggering scraping jobs and [[https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html#using-custom-scheduler-classes][running scheduled tasks]] (like a [[https://en.wikipedia.org/wiki/Cron][unix
cronjob]]) 

You can check the complete project in my GitLab here: https://gitlab.com/rogs/books-app

If you have any doubts, let me know! I always answer emails and/or messages.
** How I got a residency appointment thanks to Python, Selenium and Telegram :python:selenium:telegram:
:PROPERTIES:
:EXPORT_FILE_NAME: how-i-got-a-residency-appointment-thanks-to-python-and-selenium
:EXPORT_DATE: 2020-08-02
:ID:       b7da6c10-ca61-4839-9074-039e11a4475d
:END:
Hello everyone

As some of you might know, I'm a Venezuelan 🇻🇪 living in Montevideo, Uruguay 🇺🇾.
I've been living here for almost a year, but because of the pandemic my
residency appointments have slowed down to a crawl, and in the middle of the
quarantine they added a new appointment system. Before, there were no
appointments, you just had to get there early and wait for the secretary to
review your files and assign someone to attend you. But now, they had
implemented an appointment system that you could do from the comfort of your own
home/office. There was just one issue: *there were never appointments available*.

That was a little stressful. I was developing a small /tick/ by checking the
site multiple times a day, with no luck. But then, I decided I wanted to do a
bot that checks the site for me, that way I could just forget about it and let
the computers do it for me.

*** Tech
:PROPERTIES:
:ID:       b762da5f-9a5a-41ec-982d-ea864a661f5b
:END:
**** Selenium
:PROPERTIES:
:ID:       12fcf6c3-a167-4d7d-971e-614b1944078d
:END:
I had some experience with Selenium in the past because I had to run automated
tests on an Android application, but I had never used it for the web. I knew it
supported Firefox and had an extensive API to interact with websites. In the
end, I just had to inspect the HTML and search for the "No appointments
available" error message. If the message wasn't there, I needed a way to be
notified so I can set my appointment as fast as possible.
**** Telegram Bot API
:PROPERTIES:
:ID:       221b1f01-dfa7-46ae-b162-6299c8d69159
:END:
Telegram was my goto because I have a lot of experience with it. It has a
stupidly easy API that allows for superb bot management. I just needed the bot
to send me a message whenever the "No appointments available" message wasn't
found on the site.

*** The plan
:PROPERTIES:
:ID:       422aac40-f61b-4b7c-bd98-f68c2a0340da
:END:
Here comes the juicy part: How is everything going to work together?

I divided the work into four parts:
1) Inspecting the site
2) Finding the error message on the site
3) Sending the message if nothing was found
4) Deploy the job with a cronjob on my VPS
  
*** Inspecting the site
:PROPERTIES:
:ID:       df519909-0814-435d-9bf2-bf21b27328aa
:END:
Here is the site I needed to inspect:
- On the first site, I need to click the bottom button. By inspecting the HTML,
  I found out that its name is ~form:botonElegirHora~
  [[/2020-08-02-171251.png]]
- When the button is clicked, it loads a second page that has an error message
  if no appointments are found. The ID of that message is ~form:warnSinCupos~.
  [[/2020-08-02-162205.png]]
 
*** Using Selenium to find the error message
:PROPERTIES:
:ID:       aa6b4101-d8ab-4540-bfad-f6b70feb0e05
:END:
First, I needed to define the browser session and its settings. I wanted to run
it in headless mode so no X session is needed:
#+BEGIN_SRC python
from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True
d = webdriver.Firefox(options=options)
#+END_SRC

Then, I opened the site, looked for the button (~form:botonElegirHora~) and
clicked it
#+BEGIN_SRC python
# This is the website I wanted to scrape
d.get('https://sae.mec.gub.uy/sae/agendarReserva/Paso1.xhtml?e=9&a=7&r=13')
elem = d.find_element_by_name('form:botonElegirHora')
elem.click()
#+END_SRC

And on the new page, I looked for the error message (~form:warnSinCupos~)
#+BEGIN_SRC python
try:
    warning_message = d.find_element_by_id('form:warnSinCupos')
except Exception:
    pass
#+END_SRC

This was working exactly how I wanted: It opened a new browser session, opened
the site, clicked the button, and then looked for the message. For now, if the
message wasn't found, it does nothing. Now, the script needs to send me a
message if the warning message wasn't found on the page.

*** Using Telegram to send a message if the warning message wasn't found
:PROPERTIES:
:ID:       2a0fb5d3-d316-4ad0-9d46-58960c0ecb5e
:END:
The Telegram bot API has a very simple way to send messages. If you want to read
more about their API, you can check it [[https://core.telegram.org/][here]].

There are a few steps you need to follow to get a Telegram bot:
1) First, you need to "talk" to the [[https://core.telegram.org/bots#6-botfather][Botfather]] to create the bot.
2) Then, you need to find your Telegram Chat ID. There are a few bots that can help
  you with that, I personally use ~@get_id_bot~.
3) Once you have the ID, you should read the ~sendMessage~ API, since that's the
   only one we need now. You can check it [[https://core.telegram.org/bots/api#sendmessage][here]].

So, by using the Telegram documentation, I came up with the following code:
#+BEGIN_SRC python
import requests

chat_id = # Insert your chat ID here
telegram_bot_id = # Insert your Telegram bot ID here
telegram_data = {
    "chat_id": chat_id
    "parse_mode": "HTML",
    "text": ("<b>Hay citas!</b>\nHay citas en el registro civil, para "
             f"entrar ve a {SAE_URL}")
}
requests.post('https://api.telegram.org/bot{telegram_bot_id}/sendmessage', data=telegram_data)
#+END_SRC

*** The complete script
:PROPERTIES:
:ID:       4fdd292f-661f-424d-9c07-6fbcfa34fad7
:END:
I added a few loggers and environment variables and voilá! Here is the complete code:
#+BEGIN_SRC python
#!/usr/bin/env python3

import os
import requests
from datetime import datetime

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

from dotenv import load_dotenv

load_dotenv() # This loads the environmental variables from the .env file in the root folder

TELEGRAM_BOT_ID = os.environ.get('TELEGRAM_BOT_ID')
TELEGRAM_CHAT_ID = os.environ.get('TELEGRAM_CHAT_ID')
SAE_URL = 'https://sae.mec.gub.uy/sae/agendarReserva/Paso1.xhtml?e=9&a=7&r=13'

options = Options()
options.headless = True
d = webdriver.Firefox(options=options)
d.get(SAE_URL)
print(f'Headless Firefox Initialized {datetime.now()}')
elem = d.find_element_by_name('form:botonElegirHora')
elem.click()
try:
    warning_message = d.find_element_by_id('form:warnSinCupos')
    print('No dates yet')
    print('------------------------------')
except Exception:
    telegram_data = {
        "chat_id": TELEGRAM_CHAT_ID,
        "parse_mode": "HTML",
        "text": ("<b>Hay citas!</b>\nHay citas en el registro civil, para "
                 f"entrar ve a {SAE_URL}")
    }
    requests.post('https://api.telegram.org/bot'
                  f'{TELEGRAM_BOT_ID}/sendmessage', data=telegram_data)
    print('Dates found!')
d.close() # To close the browser connection
#+END_SRC

Only one more thing to do, to deploy everything to my VPS

*** Deploy and testing on the VPS
:PROPERTIES:
:ID:       f57694d6-904b-4c51-8560-2a1ad562e991
:END:
This was very easy. I just needed to pull my git repo, install the
~requirements.txt~ and set a new cron to run every 10 minutes and check the
site. The cron settings I used where:
#+BEGIN_SRC bash
*/10 * * * * /usr/bin/python3 /my/script/location/registro-civil-scraper/app.py >> /my/script/location/registro-civil-scraper/log.txt
#+END_SRC
The ~>> /my/script/location/registro-civil-scraper/log.txt~ part is to keep the logs on a new file.

*** Did it work?
:PROPERTIES:
:ID:       56cff142-4ab4-4f31-87f2-b4124e283158
:END:
Yes! And it worked perfectly. I got a message the following day at 21:00
(weirdly enough, that's 0:00GMT, so maybe they have their servers at GMT time
and it opens new appointments at 0:00).
[[/2020-08-02-170458.png]]

*** Conclusion
:PROPERTIES:
:ID:       1a6dcfca-aaf4-406f-8800-57ffa7832ddf
:END:
I always loved to use programming to solve simple problems. With this script, I
didn't need to check the site every couple of hours to get an appointment, and
sincerely, I wasn't going to check past 19:00, so I would've never found it by
my own.

My brother is having similar issues in Argentina, and when I showed him this, he
said one of the funniest phrases I've heard about my profession:

> /"Programmers could take over the world, but they are too lazy"/

I lol'd way too hard at that.

I loved Selenium and how it worked. Recently I created a crawler using Selenium,
Redis, peewee, and Postgres, so stay tuned if you want to know more about that.

In the meantime, if you want to check the complete script, you can see it on my
Gitlab: https://gitlab.com/rogs/registro-civil-scraper
* Announcements :announcements:
:PROPERTIES:
:ID:       0b36e39a-3389-4809-ae40-64ce37d452a1
:END:
All posts in here will have the category set to /announcements/.
** Removing comments from my blog
:PROPERTIES:
:ID:       9270279f-5911-4d91-a514-6be2a232ebc1
:EXPORT_FILE_NAME: removing-comments-from-my-blog
:EXPORT_DATE: 2023-01-14
:ID:       b7da6c10-ca61-4839-9074-039e11a4475d
:END:

I'm removing comments from my blog.

I've been thinking about this for a while, but I noticed that comments weren't being used and most posts
were not that interesting. Don't get me wrong, I really appreciate your awesome comments, but running
commento takes a lot of resources and I don't really see the full benefit of them.

From now on, if you want to leave a comment ("thank yous", suggestions, etc), you can send me an email.
You'll find my email addess on the [[/contact][Contact]] page.

You have a good and relevant comment, I'll update the relevant post accordingly.

* COMMENT Local Variables
:PROPERTIES:
:ID:       4a361a2c-2acc-4cb9-9683-d047323d091b
:END:
# Local Variables:
# eval: (org-hugo-auto-export-mode)
# End: