shelljob package

Submodules

shelljob.fs module

A collection of filesystem related commands.

class shelljob.fs.NamedTempFile(suffix=None, prefix=None, dir=None)

Bases: object

Creates a temporary file for a ‘with’ block. The file is deleted when the block exits. This creates the file to ensure it exists/block a race, but does not write anything to it, nor does it keep it open. It is intended for times when you need a named file for subprocesses.

Example:

with fs.NamedTempFile() as nm:
        proc.call( "curl http://mortoray.com/ -o {}".format( nm ) )
        html = open(nm).read()
        print( len(html) )

shelljob.fs.find(path, include_dirs=True, include_files=True, name_regex=None, not_name_regex=None, whole_name_regex=None, not_whole_name_regex=None, exclude_root=False, relative=False, limit_depth=None)

Creates an iterator of files matching a variety of conditions.

Parameters

• path – which path to iterate

• include_dirs – include directories in output

• include_files – include files in output

• name_regex – optional regex string compared against basename of file

• not_name_regex – if specificed only produces names not matching this regex

• whole_name_regex – like name_regex but applies to whole path, not just basename

• not_whole_name_regex – like not_name_regex but applies to whole path

• exclude_root – do not include the intput ‘path’ itself in the output

• limit_depth – do not list items deeper than this level from root

• relative – filenames are relative to “path” as opposed to appended to path

Returns

a generator for the matched files

shelljob.job module

A basic monitor that provides a more convenient use of the process Groups.

class shelljob.job.FileMonitor(file_pattern='/tmp/job_{}.log', meta=True, **kwargs)

Bases: shelljob.job.Monitor

A monitor which writes output to log files. Simple textual feedback will also be reported to the console.

Parameters

• file_pattern – will be formatted with the job.id to produce filenames. These files are overwritten when a job starts and record the output of the job.

• meta – if True then meta information about the job will also be recorded to the logfile

• kwargs – the remaining arguments are passed to the Monitor constructor

gen_feedback()

Called whenever the Monitor things feedback should be generated (in addition to the other events). Generally this is called for each feedback_timeout period specified in the constructor.

get_log_name(job)

(Virtual) get the log of the log file to use.

job_finished(job)

Called when a job has completed.

job_output(job, line)

Called for each line of output from a job.

job_started(job)

Called just after a job is started.

class shelljob.job.Job(cmd)

Bases: object

STATUS_FINISHED = 2

Encapsulates information about a job.

STATUS_NONE = 0

STATUS_RUNNING = 1

get_ref_name()

A reference name suitable for using in identifiers and files.

class shelljob.job.Monitor(max_simul=16, feedback_timeout=5.0)

Bases: object

convert_cmds(in_cmds)

Converts a variable format input command list to a set of Jobs.

gen_feedback()

(Virtual) Called whenever the Monitor things feedback should be generated (in addition to the other events). Generally this is called for each feedback_timeout period specified in the constructor.

get_jobs()

Get the list of jobs.

job_finished(job)

(Virtual) Called when a job has completed.

job_output(job, line)

(Virtual) Called for each line of output from a job.

job_started(job)

(Virtual) Called just after a job is started.

run(cmds, shell=False)

Run a series of commands and wait for their completion.

Parameters

cmds – a list of cmds or Job’s for Group.run. This will be run in parallel obeying the ‘max_simul’ option provided to the constructor. Using Job’s directly allows you to associate additional data for use with each job – helpful for custom monitors.

shelljob.proc module

A mechanism to run subprocesses asynchronously and with non-blocking read.

exception shelljob.proc.BadExitCode(exit_code, output)

Bases: Exception

exception shelljob.proc.CommandException(msg)

Bases: Exception

class shelljob.proc.Group

Bases: object

Runs a subprocess in parallel, capturing it’s output and providing non-blocking reads (well, at least for the caller they appear non-blocking).

clear_finished()

Remove all finished processes from the managed list.

close()

Experimental closing of all handles, even if they haven’t finished. This likely doesn’t work on all platforms

count_running()

Return the number of processes still running. Note that although a process may be finished there could still be output from it in the queue. You should use ‘is_pending’ to determine if you should still be reading.

get_exit_codes()

Return a list of all processes and their exit code.

Returns

A list of tuples: ( handle, exit_code ) ‘handle’ as returned from ‘run’ ‘exit_code’ of the process or None if it has not yet finished

is_pending()

Determine if calling readlines would actually yield any output. This returns true if there is a process running or there is data in the queue.

readline(timeout=2.0)

Read a single line from any running process.

Note that this will end up blocking for timeout once all processes have completed. ‘readlines’ however can properly handle that situation and stop reading once everything is complete.

Returns

Tuple of ( handle, line ) or None if no output generated.

readlines(max_lines=1000, timeout=2.0)

Reads available lines from any of the running processes. If no lines are available now it will wait until ‘timeout’ to read a line. If nothing is running the timeout is not waited and the function simply returns.

When a process has been completed and all output has been read from it, a variable ‘group_ouput_done’ will be set to True on the process handle.

Parameters

• timeout – how long to wait if there is nothing available now

• max_lines – maximum number of lines to get at once

Returns

An array of tuples of the form: ( handle, line ) There ‘handle’ was returned by ‘run’ and ‘line’ is the line which is read. If no line is available an empty list is returned.

run(cmd, shell=False, encoding=None, on_error=None)

Adds a new process to this object. This process is run and the output collected.

Parameters

• cmd – the command to execute. This may be an array as passed to Popen, or a string, which will be parsed by ‘shlex.split’

• shell – should it be run in a shell (see call)

• encoding – should lines be decoded automatically. Be aware if the decoding fails then the streaming will be interrupted.

• on_error – Called with any exception generated in the processing.

Returns

the handle to the process return from Popen

exception shelljob.proc.Timeout(output)

Bases: Exception

shelljob.proc.call(cmd, encoding='utf-8', shell=False, check_exit_code=True, timeout=None, cwd=None)

Calls a subprocess and returns the output and optionally exit code.

Parameters

• encoding – convert output to unicode objects with this encoding, set to None to get the raw output

• check_exit_code – set to False to ignore the exit code, otherwise any non-zero result will throw BadExitCode.

• timeout – If specified only this amount of time (seconds) will be waited for the subprocess to return

Returns

If check_exit_code is False: list( output, exit_code ), else just the output

Module contents