blob: fce8e0685c0dbec3dfeb5135d9aff42353c249df (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
+++
title = "Volition"
author = ["Roger Gonzalez"]
date = 2020-07-01
lastmod = 2020-11-14T14:02:31-03:00
draft = false
weight = 1003
+++
## About the project {#about-the-project}
Volition is an app that wants to be the top selling place for a certain kind of
product. In order to achieve that, we had to develop a series of crawlers for
different vendors, in order to get all the data so the storefront could be
created .
## Tech Stack {#tech-stack}
- JavaScript
- TypeScript
- NodeJS
- PuppeteerJS
- Docker/docker-compose
- PostgreSQL
- Google Cloud
- Kubernetes
- Bash
- ELK (ElasticSearch, LogStash, Kibana)
## What did I work on? {#what-did-i-work-on}
- Team lead
- Moved the entire project to docker and docker-compose. Before it, the
development environment has pretty tricky to setup.
- Improved the old code, introducing standards with esLint and smoke tests.
- Configured a VPN and an Ubuntu VNC session in docker to help with the proxy
and the non-headless browser.
- Created new crawlers for the new vendors.
- Configured the new Kibana dashboard.
- Created a gatekeeper to check the crawlers status before going out to the
internet.
- Monitored and ran many crawlers.
|
