coalib.processes package

Submodules

coalib.processes.BearRunning module

coalib.processes.BearRunning.get_global_dependency_results(global_result_dict, bear_instance)[source]

This method gets all the results originating from the dependencies of a bear_instance. Each bear_instance may or may not have dependencies.

Parameters:global_result_dict – The list of results out of which the dependency results are picked.
Returns:None if bear has no dependencies, False if dependencies are not met, the dependency dict otherwise.
coalib.processes.BearRunning.get_local_dependency_results(local_result_list, bear_instance)[source]

This method gets all the results originating from the dependencies of a bear_instance. Each bear_instance may or may not have dependencies.

Parameters:
  • local_result_list – The list of results out of which the dependency results are picked.
  • bear_instance – The instance of a local bear to get the dependencies from.
Returns:

Return none if there are no dependencies for the bear. Else return a dictionary containing dependency results.

coalib.processes.BearRunning.get_next_global_bear(timeout, global_bear_queue, global_bear_list, global_result_dict)[source]

Retrieves the next global bear.

Parameters:
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • global_bear_queue – queue (read, write) of indexes of global bear instances in the global_bear_list.
  • global_bear_list – A list containing all global bears to be executed.
  • global_result_dict – A Manager.dict that will be used to store global results. The list of results of one global bear will be stored with the bear name as key.
Returns:

(bear, bearname, dependency_results)

coalib.processes.BearRunning.run(file_name_queue, local_bear_list, global_bear_list, global_bear_queue, file_dict, local_result_dict, global_result_dict, message_queue, control_queue, timeout=0)[source]

This is the method that is actually runs by processes.

If parameters type is ‘queue (read)’ this means it has to implement the get(timeout=TIMEOUT) method and it shall raise queue.Empty if the queue is empty up until the end of the timeout. If the queue has the (optional!) task_done() attribute, the run method will call it after processing each item.

If parameters type is ‘queue (write)’ it shall implement the put(object, timeout=TIMEOUT) method.

If the queues raise any exception not specified here the user will get an ‘unknown error’ message. So beware of that.

Parameters:
  • file_name_queue – queue (read) of file names to check with local bears. Each invocation of the run method needs one such queue which it checks with all the local bears. The queue could be empty. (Repeat until queue empty.)
  • local_bear_list – List of local bear instances.
  • global_bear_list – List of global bear instances.
  • global_bear_queue – queue (read, write) of indexes of global bear instances in the global_bear_list.
  • file_dict – dict of all files as {filename:file}, file as in file.readlines().
  • local_result_dict – A Manager.dict that will be used to store local results. A list of all local results. will be stored with the filename as key.
  • global_result_dict – A Manager.dict that will be used to store global results. The list of results of one global bear will be stored with the bear name as key.
  • message_queue – queue (write) for debug/warning/error messages (type LogMessage)
  • control_queue – queue (write). If any result gets written to the result_dict a tuple containing a CONTROL_ELEMENT (to indicate what kind of event happened) and either a bear name (for global results) or a file name to indicate the result will be put to the queue. If the run method finished all its local bears it will put (CONTROL_ELEMENT.LOCAL_FINISHED, None) to the queue, if it finished all global ones, (CONTROL_ELEMENT.GLOBAL_FINISHED, None) will be put there.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
coalib.processes.BearRunning.run_bear(message_queue, timeout, bear_instance, *args, **kwargs)[source]

This method is responsible for executing the instance of a bear. It also reports or logs errors if any occur during the execution of that bear instance.

Parameters:
  • message_queue – A queue that contains messages of type errors/warnings/debug statements to be printed in the Log.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • bear_instance – The instance of the bear to be executed.
  • args – The arguments that are to be passed to the bear.
  • kwargs – The keyword arguments that are to be passed to the bear.
Returns:

Returns a valid list of objects of the type Result if the bear executed successfully. None otherwise.

coalib.processes.BearRunning.run_global_bear(message_queue, timeout, global_bear_instance, dependency_results)[source]

Runs an instance of a global bear. Checks if bear_instance is of type GlobalBear and then passes it to the run_bear to execute.

Parameters:
  • message_queue – A queue that contains messages of type errors/warnings/debug statements to be printed in the Log.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • global_bear_instance – Instance of GlobalBear to run.
  • dependency_results – The results of all the bears on which the instance of the passed bear to be run depends on.
Returns:

Returns a list of results generated by the passed bear_instance.

coalib.processes.BearRunning.run_global_bears(message_queue, timeout, global_bear_queue, global_bear_list, global_result_dict, control_queue)[source]

Run all global bears.

Parameters:
  • message_queue – A queue that contains messages of type errors/warnings/debug statements to be printed in the Log.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • global_bear_queue – queue (read, write) of indexes of global bear instances in the global_bear_list.
  • global_bear_list – list of global bear instances
  • global_result_dict – A Manager.dict that will be used to store global results. The list of results of one global bear will be stored with the bear name as key.
  • control_queue – If any result gets written to the result_dict a tuple containing a CONTROL_ELEMENT (to indicate what kind of event happened) and either a bear name(for global results) or a file name to indicate the result will be put to the queue.
coalib.processes.BearRunning.run_local_bear(message_queue, timeout, local_result_list, file_dict, bear_instance, filename)[source]

Runs an instance of a local bear. Checks if bear_instance is of type LocalBear and then passes it to the run_bear to execute.

Parameters:
  • message_queue – A queue that contains messages of type errors/warnings/debug statements to be printed in the Log.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • local_result_list – Its a list that stores the results of all local bears.
  • file_dict – Dictionary containing contents of file.
  • bear_instance – Instance of LocalBear the run.
  • filename – Name of the file to run it on.
Returns:

Returns a list of results generated by the passed bear_instance.

coalib.processes.BearRunning.run_local_bears(filename_queue, message_queue, timeout, file_dict, local_bear_list, local_result_dict, control_queue)[source]

Run local bears on all the files given.

Parameters:
  • filename_queue – queue (read) of file names to check with local bears.
  • message_queue – A queue that contains messages of type errors/warnings/debug statements to be printed in the Log.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • file_dict – Dictionary that contains contents of files.
  • local_bear_list – List of local bears to run.
  • local_result_dict – A Manager.dict that will be used to store local bear results. A list of all local bear results will be stored with the filename as key.
  • control_queue – If any result gets written to the result_dict a tuple containing a CONTROL_ELEMENT (to indicate what kind of event happened) and either a bear name(for global results) or a file name to indicate the result will be put to the queue.
coalib.processes.BearRunning.run_local_bears_on_file(message_queue, timeout, file_dict, local_bear_list, local_result_dict, control_queue, filename)[source]

This method runs a list of local bears on one file.

Parameters:
  • message_queue – A queue that contains messages of type errors/warnings/debug statements to be printed in the Log.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • file_dict – Dictionary that contains contents of files.
  • local_bear_list – List of local bears to run on file.
  • local_result_dict – A Manager.dict that will be used to store local bear results. A list of all local bear results will be stored with the filename as key.
  • control_queue – If any result gets written to the result_dict a tuple containing a CONTROL_ELEMENT (to indicate what kind of event happened) and either a bear name(for global results) or a file name to indicate the result will be put to the queue.
  • filename – The name of file on which to run the bears.
coalib.processes.BearRunning.send_msg(message_queue, timeout, log_level, *args, delimiter=' ', end='')[source]

Puts message into message queue for a LogPrinter to present to the user.

Parameters:
  • message_queue – The queue to put the message into and which the LogPrinter reads.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • log_level – The log_level i.e Error,Debug or Warning.It is sent to the LogPrinter depending on the message.
  • args – This includes the elements of the message.
  • delimiter – It is the value placed between each arg. By default it is a ‘ ‘.
  • end – It is the value placed at the end of the message.
coalib.processes.BearRunning.task_done(obj)[source]

Invokes task_done if the given queue provides this operation. Otherwise passes silently.

Parameters:obj – Any object.
coalib.processes.BearRunning.validate_results(message_queue, timeout, result_list, name, args, kwargs)[source]

Validates if the result_list passed to it contains valid set of results. That is the result_list must itself be a list and contain objects of the instance of Result object. If any irregularity is found a message is put in the message_queue to present the irregularity to the user. Each result_list belongs to an execution of a bear.

Parameters:
  • message_queue – A queue that contains messages of type errors/warnings/debug statements to be printed in the Log.
  • timeout – The queue blocks at most timeout seconds for a free slot to execute the put operation on. After the timeout it returns queue Full exception.
  • result_list – The list of results to validate.
  • name – The name of the bear executed.
  • args – The args with which the bear was executed.
  • kwargs – The kwargs with which the bear was executed.
Returns:

Returns None if the result_list is invalid. Else it returns the result_list itself.

coalib.processes.CONTROL_ELEMENT module

coalib.processes.LogPrinterThread module

class coalib.processes.LogPrinterThread.LogPrinterThread(message_queue, log_printer)[source]

Bases: threading.Thread

This is the Thread object that outputs all log messages it gets from its message_queue. Setting obj.running = False will stop within the next 0.1 seconds.

run()[source]

coalib.processes.Processing module

coalib.processes.Processing.autoapply_actions(results, file_dict, file_diff_dict, section, log_printer)[source]

Auto-applies actions like defined in the given section.

Parameters:
  • results – A list of results.
  • file_dict – A dictionary containing the name of files and its contents.
  • file_diff_dict – A dictionary that contains filenames as keys and diff objects as values.
  • section – The section.
  • log_printer – A log printer instance to log messages on.
Returns:

A list of unprocessed results.

coalib.processes.Processing.check_result_ignore(result, ignore_ranges)[source]

Determines if the result has to be ignored.

Any result will be ignored if its origin matches any bear names and its SourceRange overlaps with the ignore range.

Note that everything after a space in the origin will be cut away, so the user can ignore results with an origin like CSecurityBear (buffer) with just # Ignore CSecurityBear.

Parameters:
  • result – The result that needs to be checked.
  • ignore_ranges – A list of tuples, each containing a list of lower cased affected bearnames and a SourceRange to ignore. If any of the bearname lists is empty, it is considered an ignore range for all bears. This may be a list of globbed bear wildcards.
Returns:

True if the result has to be ignored.

coalib.processes.Processing.create_process_group(command_array, **kwargs)[source]
coalib.processes.Processing.execute_section(section, global_bear_list, local_bear_list, print_results, cache, log_printer, console_printer)[source]

Executes the section with the given bears.

The execute_section method does the following things:

  1. Prepare a Process - Load files - Create queues
  2. Spawn up one or more Processes
  3. Output results from the Processes
  4. Join all processes
Parameters:
  • section – The section to execute.
  • global_bear_list – List of global bears belonging to the section. Dependencies are already resolved.
  • local_bear_list – List of local bears belonging to the section. Dependencies are already resolved.
  • print_results – Prints all given results appropriate to the output medium.
  • cache – An instance of misc.Caching.FileCache to use as a file cache buffer.
  • log_printer – The log_printer to warn to.
  • console_printer – Object to print messages on the console.
Returns:

Tuple containing a bool (True if results were yielded, False otherwise), a Manager.dict containing all local results(filenames are key) and a Manager.dict containing all global bear results (bear names are key) as well as the file dictionary.

coalib.processes.Processing.fill_queue(queue_fill, any_list)[source]

Takes element from a list and populates a queue with those elements.

Parameters:
  • queue_fill – The queue to be filled.
  • any_list – List containing the elements.
coalib.processes.Processing.filter_raising_callables(it, exception, *args, **kwargs)[source]

Filters all callable items inside the given iterator that raise the given exceptions.

Parameters:
  • it – The iterator to filter.
  • exception – The (tuple of) exception(s) to filter for.
  • args – Positional arguments to pass to the callable.
  • kwargs – Keyword arguments to pass to the callable.
coalib.processes.Processing.get_cpu_count()[source]
coalib.processes.Processing.get_default_actions(section)[source]

Parses the key default_actions in the given section.

Parameters:section – The section where to parse from.
Returns:A dict with the bearname as keys and their default actions as values and another dict that contains bears and invalid action names.
coalib.processes.Processing.get_file_dict(filename_list, log_printer)[source]

Reads all files into a dictionary.

Parameters:
  • filename_list – List of names of paths to files to get contents of.
  • log_printer – The logger which logs errors.
Returns:

Reads the content of each file into a dictionary with filenames as keys.

coalib.processes.Processing.get_file_list(results)[source]

Get the set of files that are affected in the given results.

Parameters:results – A list of results from which the list of files is to be extracted.
Returns:A set of file paths containing the mentioned list of files.
coalib.processes.Processing.get_ignore_scope(line, keyword)[source]

Retrieves the bears that are to be ignored defined in the given line.

Parameters:
  • line – The line containing the ignore declaration.
  • keyword – The keyword that was found. Everything after the rightmost occurrence of it will be considered for the scope.
Returns:

A list of lower cased bearnames or an empty list (-> “all”)

coalib.processes.Processing.get_running_processes(processes)[source]
coalib.processes.Processing.instantiate_bears(section, local_bear_list, global_bear_list, file_dict, message_queue, console_printer)[source]

Instantiates each bear with the arguments it needs.

Parameters:
  • section – The section the bears belong to.
  • local_bear_list – List of local bear classes to instantiate.
  • global_bear_list – List of global bear classes to instantiate.
  • file_dict – Dictionary containing filenames and their contents.
  • message_queue – Queue responsible to maintain the messages delivered by the bears.
  • console_printer – Object to print messages on the console.
Returns:

The local and global bear instance lists.

coalib.processes.Processing.instantiate_processes(section, local_bear_list, global_bear_list, job_count, cache, log_printer, console_printer)[source]

Instantiate the number of processes that will run bears which will be responsible for running bears in a multiprocessing environment.

Parameters:
  • section – The section the bears belong to.
  • local_bear_list – List of local bears belonging to the section.
  • global_bear_list – List of global bears belonging to the section.
  • job_count – Max number of processes to create.
  • cache – An instance of misc.Caching.FileCache to use as a file cache buffer.
  • log_printer – The log printer to warn to.
  • console_printer – Object to print messages on the console.
Returns:

A tuple containing a list of processes, and the arguments passed to each process which are the same for each object.

coalib.processes.Processing.print_result(results, file_dict, retval, print_results, section, log_printer, file_diff_dict, ignore_ranges, console_printer)[source]

Takes the results produced by each bear and gives them to the print_results method to present to the user.

Parameters:
  • results – A list of results.
  • file_dict – A dictionary containing the name of files and its contents.
  • retval – It is True if no results were yielded ever before. If it is False this function will return False no matter what happens. Else it depends on if this invocation yields results.
  • print_results – A function that prints all given results appropriate to the output medium.
  • file_diff_dict – A dictionary that contains filenames as keys and diff objects as values.
  • ignore_ranges – A list of SourceRanges. Results that affect code in any of those ranges will be ignored.
  • console_printer – Object to print messages on the console.
Returns:

Returns False if any results were yielded. Else True.

coalib.processes.Processing.process_queues(processes, control_queue, local_result_dict, global_result_dict, file_dict, print_results, section, cache, log_printer, console_printer)[source]

Iterate the control queue and send the results received to the print_result method so that they can be presented to the user.

Parameters:
  • processes – List of processes which can be used to run Bears.
  • control_queue – Containing control elements that indicate whether there is a result available and which bear it belongs to.
  • local_result_dict – Dictionary containing results respective to local bears. It is modified by the processes i.e. results are added to it by multiple processes.
  • global_result_dict – Dictionary containing results respective to global bears. It is modified by the processes i.e. results are added to it by multiple processes.
  • file_dict – Dictionary containing file contents with filename as keys.
  • print_results – Prints all given results appropriate to the output medium.
  • cache – An instance of misc.Caching.FileCache to use as a file cache buffer.
Returns:

Return True if all bears execute successfully and Results were delivered to the user. Else False.

coalib.processes.Processing.simplify_section_result(section_result)[source]

Takes in a section’s result from execute_section and simplifies it for easy usage in other functions.

Parameters:section_result – The result of a section which was executed.
Returns:Tuple containing: - bool - True if results were yielded - bool - True if unfixed results were yielded - list - Results from all bears (local and global)
coalib.processes.Processing.yield_ignore_ranges(file_dict)[source]

Yields tuples of affected bears and a SourceRange that shall be ignored for those.

Parameters:file_dict – The file dictionary.

Module contents