45 lines
1.2 KiB
Markdown
45 lines
1.2 KiB
Markdown
+++
|
|
title = "Volition"
|
|
author = ["Roger Gonzalez"]
|
|
date = 2020-07-01
|
|
lastmod = 2020-11-14T14:02:31-03:00
|
|
draft = false
|
|
weight = 1003
|
|
+++
|
|
|
|
## About the project {#about-the-project}
|
|
|
|
Volition is an app that wants to be the top selling place for a certain kind of
|
|
product. In order to achieve that, we had to develop a series of crawlers for
|
|
different vendors, in order to get all the data so the storefront could be
|
|
created .
|
|
|
|
|
|
## Tech Stack {#tech-stack}
|
|
|
|
- JavaScript
|
|
- TypeScript
|
|
- NodeJS
|
|
- PuppeteerJS
|
|
- Docker/docker-compose
|
|
- PostgreSQL
|
|
- Google Cloud
|
|
- Kubernetes
|
|
- Bash
|
|
- ELK (ElasticSearch, LogStash, Kibana)
|
|
|
|
|
|
## What did I work on? {#what-did-i-work-on}
|
|
|
|
- Team lead
|
|
- Moved the entire project to docker and docker-compose. Before it, the
|
|
development environment has pretty tricky to setup.
|
|
- Improved the old code, introducing standards with esLint and smoke tests.
|
|
- Configured a VPN and an Ubuntu VNC session in docker to help with the proxy
|
|
and the non-headless browser.
|
|
- Created new crawlers for the new vendors.
|
|
- Configured the new Kibana dashboard.
|
|
- Created a gatekeeper to check the crawlers status before going out to the
|
|
internet.
|
|
- Monitored and ran many crawlers.
|