log_writer module

class LogSimulator(config_path)[source]

Bases: threading.Thread

Simulates a real actively writen HTTP access log: writes lines in a given log file at given speeds during given times. All theses parameters are read in a given config file.

Note

See data/sim_config to understand how a config file should be written.

Variables:
  • config_path (string) – The path to the config file where the parameters and the command for the simulation are stored.
  • log_w (LogWriter) – A reference to the present LogWriter
  • name (string) –
  • total_nb_of_lines_previously_written (int) –
  • param_dict (dictionary) –
  • should_run (bool) – If False, the thread will shortly end stop its operation. Used to cleanly end the program
get_parameters()[source]

Reads the config file and extracts the parameters i.e. the lines with ‘=’. Sets LogSimulator.param_dict.

Returns:A dictionary with the parameters as key
Return type:dictionary
run()[source]

Reads the config file given, and executes the commands read with the read parameters.

started_first_writing()[source]
Returns:True if the LogWriter has started to write, else False
Return type:bool
state()[source]
Returns:Describes the present thread state
Return type:string
exception LogSimulatorConfigFileError[source]

Bases: exceptions.Exception

Raised when the path or the format of the config file is incorrect

class LogWriter(log_path, line_type=u'HTTP_slow', pace=3000, timeout=-1, erase_first=True, is_simulated=False)[source]

Bases: threading.Thread

Writes a given type of lines in the file given by the log_path at a given speed during a given time.

Warning

The time is the priority, hence of the I/O cannot follow the writing speed asked, the program will not write as many lines as asked, but it will end in time. A WARNING log message will be send.

Variables:
  • log_path (string) – The path to the log file, may already exist and will be erased or may not exist and will be create (see erase_first)
  • line_type (string) –

    The type of line that that should be writen in the log, can be:

    • 'line': lineX will be writen, where X is the number of the line writen (fastest)
    • 'HTTP_fast': an HTTP access line will be writen, but only the HTTP request and the bytes will be random (slower)
    • 'HTTP_slow': an HTTP access line will be writen, everything will be random (slowest)
  • pace (int) – The number of line that should be writen every second. If the IO stream cannot follow the pace, a log message is send to the Displayer.
  • timeout (int) – The number of seconds meanwhile the log should be writen, never stop if timeout == -1, minimum timeout is 1.
  • erase_first (bool) – If True, the log_path is erased before being writen (log file is opened in 'at' mode).
  • should_run (bool) – Used to get out of the writing while loop when the user want to stop the program.
  • is_simulated (LogSimulator or None) – A reference to the LogSimulator if this is a simulation, or None. Used to update LogSimulator.total_number...
  • nb_of_line_written (int) – The number of line written by the LogWriter. Only update every 100 lines !
  • started_writing (bool) – Becomes True after the first flush() ie after a first line has been written. NB: Could be deduced from nb_of_line_written...
  • name (string) –
run()[source]

Open the output log file and writes log lines in it n = pace every second, as long as execution time < timeout

Called when the thread is started.

generate_all_URL_possible(factor=2, max_depth=2)[source]

Generates a list of all the possible local URL with the given parameters

Note

number of section = len(section) * factor

max_depth is the highest number possible of /section/subsection/subsubsection/etc. /archive2/blog0/archive2/page1/ -> depth = 4

Examples

  • / <- min_depth without URL_end
  • /index.html <- min_depth with URL_end
  • /page0/
  • /archive0/blog1/
  • /blog0/picture.png
  • /page0/archive1/archive2/index.html
  • /archive2/archive0/archive2/archive2/picture.png <- max_depth = 4, with URL_end
random_HTTP_request(URL)[source]
Parameters:URL (string) – The URL that will be written in the HTTP request
Returns:A random W3C valid HTTP request
Return type:string

Examples

  • "HEAD /index.html HTTP/1.1"
  • "GET /page0/ HTTP/1.1"
  • "DELETE /archive0/blog1/ HTTP/1.1"
random_log_line_maker(line_type, **kwargs)[source]
Returns:

The right log line writer function
Return type:

function
Parameters:

line_type (string) –

The type of line that should be generated, can be

  • line: the string returned will be lineX\n where X is line_count given in keyword argument
  • HTTP_fast: the string returned will be a static HTTP access log line with random HTTP request
    and a accurate date, and random bytes
  • HTTP_slow: the string returned will be a fully random HTTP access line
Keyword Arguments:
 
  • factor (int) – Parameters used in the uniform_random_local_URL_maker() function
  • max_depth (int) – Parameters used in the uniform_random_local_URL_maker() function
uniform_random_local_URL_maker(factor=2, max_depth=2)[source]

Returns a uniform_random_local_URL function that returns equi-probable local URL, this is useful (I assure you) to avoid getting always the same most hit section in a simulation.

Note

number of section = len(section) * factor

max_depth is the highest number possible of /section/subsection/subsubsection/etc. /archive2/blog0/archive2/page1/ -> depth = 4

Examples

  • / <- min_depth without URL_end
  • /index.html <- min_depth with URL_end
  • /page0/
  • /archive0/blog1/
  • /blog0/picture.png
  • /page0/archive1/archive2/index.html
  • /archive2/archive0/archive2/archive2/picture.png <- max_depth = 4, with URL_end