Author : saeednasehi

I want to scale my scrapy so I started to distribute my scrapy image on multiple servers. I have got a VPS with centos 7.3 and after configuring that, I used below docker-compose to bring it up: version: ‘2’ services: image: istresearch/scrapy-cluster:crawler-1.2.1 volumes: – logs:/var/log/scrapy-cluster – ./crawler/settings.py:/usr/src/app/crawling/localsettings.py network_mode: host environment: KAFKA_HOSTS: REMOTE_HOST:9092 ZOOKEEPER_HOSTS: REMOTE_HOST:2181 REDIS_HOST: ..

Read more