Category : hadoop

Thank you for reading! We are using docker-compose to start an instance of Hadoop on a local dev machine (Mac). Recently we began seeing the following error in the hadoop container, in the /var/log/hadoop-yarn-resource-manager.log file: 1-08-31 03:07:02,108 INFO [main] ipc.CallQueueManager (CallQueueManager.java:<init>(75)) – Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2021-08-31 03:07:02,198 INFO [main] service.AbstractService (AbstractService.java:noteFailure(272)) ..

Read more

I have a docker-compose with Flink (JobManager and TaskManager from Flink Playground) and HDFS(NameNode and DataNode). I want to make pipeline (Flink to HDFS) but have an Exception: Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘hdfs’. The scheme is not directly supported by Flink and no Hadoop file system to ..

Read more

Following these instructions, I get to the point where I want to execute pyspark. First, some perhaps useful information about what is going on: [email protected]:~/docker-hadoop-spark$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0d3a7c199e40 bde2020/spark-worker:3.0.0-hadoop3.2 "/bin/bash /worker.sh" 39 minutes ago Up 18 minutes 0.0.0.0:8081->8081/tcp spark-worker-1 c57ee3c4c30e bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /bin/u2026" 50 minutes ago Up ..

Read more