Category : hive

I have got the following setup for a local Hive-Server with Hadoop: version: "3" services: namenode: image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8 container_name: namenode restart: always ports: – 9870:9870 – 9000:9000 volumes: – ./hdfs/namenode:/hadoop/dfs/name environment: – CLUSTER_NAME=test env_file: – ./hadoop.env datanode: image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8 container_name: datanode restart: always volumes: – ./hdfs/datanode:/hadoop/dfs/data environment: SERVICE_PRECONDITION: "namenode:9870" env_file: – ./hadoop.env hive-server: image: bde2020/hive:2.3.2-postgresql-metastore ..

Read more

I want to setup a local hive server and found this repo: https://github.com/big-data-europe/docker-hive This is the yaml file I use. version: "3" services: namenode: image: bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8 volumes: – namenode:/hadoop/dfs/name environment: – CLUSTER_NAME=test env_file: – ./hadoop-hive.env ports: – "50070:50070" datanode: image: bde2020/hadoop-datanode:2.0.0-hadoop2.7.4-java8 volumes: – datanode:/hadoop/dfs/data env_file: – ./hadoop-hive.env environment: SERVICE_PRECONDITION: "namenode:50070" ports: – "50075:50075" hive-server: image: ..

Read more

Input Data wk_end_d Total Volume 2/6/2021 236255 2/13/2021 231962 2/20/2021 190785 2/27/2021 209750 3/6/2021 234347 3/13/2021 210586 3/20/2021 217937 3/27/2021 266082 4/3/2021 228382 4/10/2021 200622 4/17/2021 260464 4/24/2021 229509 5/1/2021 259193 Output how iam looking for is .How can I achieve this the calculations are interdependent |wk_end_d| |Total Volume| |New Volume| |Worked Volume| |Remaining Volume| ..

Read more

I’m trying to connect to hive-server in my docker container with the command beeline -u jdbc:hive2://localhost:10000, but i get the erros: [email protected]_server:/opt# beeline -u jdbc:hive2://localhost:10000 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] ..

Read more

Following these instructions, I get to the point where I want to execute pyspark. First, some perhaps useful information about what is going on: [email protected]:~/docker-hadoop-spark$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0d3a7c199e40 bde2020/spark-worker:3.0.0-hadoop3.2 "/bin/bash /worker.sh" 39 minutes ago Up 18 minutes 0.0.0.0:8081->8081/tcp spark-worker-1 c57ee3c4c30e bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /bin/u2026" 50 minutes ago Up ..

Read more

I am trying to load data into hive table running on docker container using the following: i get the error below:`Windows PowerShell Copyright (C) Microsoft Corporation. All rights reserved. Try the new cross-platform PowerShell https://aka.ms/pscore6 PS C:UsersJohn Mekubo> cd desktop PS C:UsersJohn Mekubodesktop> cd hive PS C:UsersJohn Mekubodesktophive> cd docker-hive PS C:UsersJohn Mekubodesktophivedocker-hive> docker-compose up ..

Read more

I’m working with a dockerized pyspark cluster which utilizes yarn. To improve the efficieny of the data processing pipelines I want to increase the amount of memory allocated to the pyspark executors and the driver. This is done by adding the following two key, value pairs to the REST post request, which is sent out ..

Read more