Provide CPU time and memory to subprocess

  c#, docker, python, resources, subprocess

I have been researching over on how to provide Python subprocess it’s own time and memory.


import resource
import subprocess


def set_memory_time(seconds):
    limit_virtual_memory(seconds)
    usage_start = resource.getrusage(resource.RUSAGE_CHILDREN)
    print("usage_start ", usage_start)
    try:
        p = subprocess.check_output(
            ['docker exec -it cpp_compiler sh -c "g++ -o Test1 prog1.cpp && ./Test1 < input.txt"'],
            shell=True)
    except Exception as e:
        print(e)
    usage_end = resource.getrusage(resource.RUSAGE_CHILDREN)
    print("usage_end ", usage_end)
    cpu_time = usage_end.ru_utime - usage_start.ru_utime
    print("cpu_time ", cpu_time)


def limit_virtual_memory(seconds):
    max_virtual_memory = 10 * 1024 * 1024  # 10 MB
    usage_start = resource.getrusage(resource.RUSAGE_CHILDREN)
    resource.setrlimit(resource.RLIMIT_AS, (max_virtual_memory, resource.RLIM_INFINITY))
    resource.setrlimit(resource.RLIMIT_CPU, (seconds, usage_start.ru_utime + seconds))

Problem is that resource.setrlimit set limit for the main process and subprocess uses that limit. When limit exceeds it actually kills process as well.

  1. Overall goal that I am trying to achieve here is subprocess.check_output(['docker exec -it cpp_compiler sh -c "g++ -o Test1 prog1.cpp && ./Test1 < input.txt"'] this line should not more resource than allocated.
    Is there way to achieve this in python ?

  2. The problem I am trying to solve here is trying to allocate CPU time and memory limit for user submitted CPP code. CPP code will eventually run on docker container for sandboxing purpose, but want to have limit on the resource it uses.

It would be really helpful if someone can provide input on the problem and potential solution or correction in above code.

Thanks

Source: Docker Questions

LEAVE A COMMENT