Priority error when pyspark save mode ‘overwrite’ in docker container

  apache-spark, docker, dockerfile, pyspark, python

I am saving csv files as stream with pyspark. When I saving files, I am using output mode is ‘overwrite’ and there is not any problem. But when I want to containerize my spark app is giving an error. I add code and the error below:

df.write.format("csv").mode("overwrite")

java.io.IOException: Unable to clear output directory file:/app/files prior to writing to it

I think the error is due to permissions. So I tried USER root in dockerfile but the error not fixed.

Source: Docker Questions

LEAVE A COMMENT