I am saving csv files as stream with pyspark. When I saving files, I am using output mode is ‘overwrite’ and there is not any problem. But when I want to containerize my spark app is giving an error. I add code and the error below:
java.io.IOException: Unable to clear output directory file:/app/files prior to writing to it
I think the error is due to permissions. So I tried USER root in dockerfile but the error not fixed.
Source: Docker Questions