API Reference

OS Helper

This module provides a collection of utility functions aimed at simplifying various common programming tasks, including file handling, system operations, string manipulation, folder management, and more. The functions are optimized for cross-platform compatibility and robust error handling.

Authors: - Warith Harchaoui, https://harchaoui.org/warith - Mohamed Chelali, https://mchelali.github.io - Bachir Zerroug, https://www.linkedin.com/in/bachirzerroug

Dependencies: - contextlib - glob - hashlib - json - logging - os - pathlib - shlex - shutil - sys - tempfile - requests - datetime (from datetime) - subprocess (from subprocess) - platform (from sys) - numpy - yaml - validators - zipfile - dotenv

os_helper.main.absolute2relative_path(path: str, base_path: str = None) → str

Convert an absolute path to a relative path, using a specified base path.

This function converts a given absolute path to a relative path, based on a provided base path. If no base path is given, it uses the current working directory as the base.

Parameters:

• path (str) – The absolute path to convert to a relative path.

• base_path (str, optional) – The base path to use for conversion. If None, uses the current working directory. Defaults to None.

Returns:

The relative path from the base path.

Return type:

str

Example

>>> absolute2relative_path("/home/user/project/file.txt", "/home/user")
'project/file.txt'

os_helper.main.asciistring(input_string: str, replacement_char: str = '-', lower: bool = True, allow_digits: bool = True) → str

Converts a given string into a “safe” ASCII string, replacing accented and non-ASCII characters with their ASCII equivalents. Non-ASCII characters that cannot be converted will be replaced with a specified character.

Parameters:

• input_string (str) – The input string to be converted.

• replacement_char (str, optional) – The character to replace non-ASCII or unwanted characters with. Defaults to ‘-‘.

• lower (bool, optional) – Whether to convert the string to lowercase. Defaults to True.

• allow_digits (bool, optional) – Whether to allow digits in the resulting string. Defaults to True.

Returns:

A “safe” ASCII string with unwanted characters replaced and case adjusted.

Return type:

str

Examples

>>> asciistring("MyFile@2024.txt")
'myfile-2024-txt'

>>> asciistring("Café-Con-Leche!", replacement_char="_")
'cafe_con_leche'

>>> asciistring("Special#File$2024", lower=False)
'Special-File-2024'

os_helper.main.check(condition: bool, msg: str = 'Something went wrong', error_code: int = 1) → None

Check a condition and log an error message if it is not met.

Parameters:

• condition (bool) – The condition to check. If False, logs the error message and exits.

• msg (str, optional) – The error message to log if the condition is not met. Defaults to “Something went wrong”.

• error_code (int, optional) – The exit code to use if the condition fails. Defaults to 1.

Example

>>> check(1 == 2, msg="Condition failed", error_code=2)

os_helper.main.checkfile(filepath: str, msg: str = '', check_empty: bool = False) → None

Check if a file exists and is optionally not empty.

Parameters:

• filepath (str) – The path to the file.

• msg (str, optional) – Message to log if the check fails. Defaults to “”.

• check_empty (bool, optional) – If True, also checks if the file is not empty. Defaults to False.

Example

>>> checkfile("example.txt", msg="File not found", check_empty=True)

os_helper.main.copyfile(a: str, b: str) → None

Copy a file from source to destination.

Parameters:

• a (str) – Source file path.

• b (str) – Destination file path.

Example

>>> copyfile("source.txt", "destination.txt")

os_helper.main.dir_exists(path: str, check_empty: bool = False) → bool

Check if a directory exists, with an option to check if it is empty.

Parameters:

• path (str) – The path to the directory.

• check_empty (bool, optional) – If True, checks if the directory contains files. Defaults to False.

Returns:

True if the directory exists (and is not empty, if check_empty is True).

Return type:

bool

Example

>>> dir_exists("/path/to/folder", check_empty=True)
True

os_helper.main.download_file(url: str, file_path: str = '')

download_file function is a shortcut function to download a file from a URL

Parameters:

• url (str) – URL to download the file from

• file_path (str) – File path to save the downloaded file

os_helper.main.emptystring(s: str) → bool

Check if a string is empty or None.

This utility function checks if the given string is either None or an empty string. It is useful for quickly validating user input or file paths.

Parameters:

s (str) – The string to check.

Returns:

True if the string is None or empty, False otherwise.

Return type:

bool

Example

>>> emptystring("")
True

>>> emptystring("hello")
False

os_helper.main.error(msg: str, error_code: int = 1) → None

Log an error message and exit the program with a specified error code.

Parameters:

• msg (str) – The error message to log.

• error_code (int, optional) – The exit code to use when terminating the program. Defaults to 1.

Example

>>> error("Critical failure", error_code=2)

os_helper.main.file_exists(file_path: str, check_empty: bool = False) → bool

Check if a file exists, with an option to check if it is empty.

Parameters:

• file_path (str) – The path to the file.

• check_empty (bool, optional) – If True, checks if the file is not empty. Defaults to False.

Returns:

True if the file exists (and is not empty, if check_empty is True).

Return type:

bool

Example

>>> file_exists("example.txt")
True

os_helper.main.folder_description(path: str, recursive: bool = True, index_html: bool = True, with_size: bool = True) → dict

Describe the contents of a folder, optionally recursively, and generate an HTML index.

Parameters:

• path (str) – The path to the folder to describe.

• recursive (bool, optional) – If True, includes subdirectories in the description. Defaults to True.

• index_html (bool, optional) – If True, generates an HTML index of the folder contents. Defaults to True.

• with_size (bool, optional) – If True, includes the size of each file in the description. Defaults to True.

Returns:

A dictionary describing the contents of the folder.

Return type:

dict

Example

>>> folder_description("/path/to/folder")
{'file1.txt': 1024, 'subfolder': None}

os_helper.main.folder_name_ext(path: str, checkpath: bool = False) → tuple

Decompose a file or folder path into three components: folder, basename, and extension.

This function splits a given path into its directory, filename (without extension), and the extension. It handles both single and multi-part extensions (e.g., .tar.gz).

Parameters:

• path (str) – The file or folder path to decompose.

• checkpath (bool, optional) – If True, checks if the path exists. Defaults to False.

Returns:

A tuple (folder, basename, extension) where: - folder: The absolute path of the directory containing the file or folder. - basename: The name of the file or folder without its extension. - extension: The extension of the file (empty if it’s a folder or no extension).

Return type:

tuple

Example

>>> folder_name_ext("/path/to/file.txt")
('/path/to', 'file', 'txt')

>>> folder_name_ext("/path/to/file.tar.gz")
('/path/to', 'file', 'tar.gz')

>>> folder_name_ext("/path/to/folder")
('/path/to', 'folder', '')

os_helper.main.format_size(size: int) → str

Helper function to format file size in human-readable form.

Parameters:

size (int) – Size of the file in bytes.

Returns:

Human-readable file size.

Return type:

str

os_helper.main.get_config(keys: List[str], config_type: str, path: str = None, env_files: List[str] = ['.env']) → dict

get_config is a helper function to load configuration from a file or environment variables.

This function attempts to load configuration settings from a file (JSON or YAML) or environment variables or .env. If a path or folder is provided, it will attempt to load the configuration from the file at that path. If no path is provided, it will look in environment or .env files.

Parameters:

• keys (list) – List of keys that must be present in the configuration file.

• config_type (str) – The type of configuration (for logging purposes).

• path (str, optional) – The path to the configuration file or folder. Defaults to None.

• env_files (list, optional) – List of .env files to check for configuration. Defaults to [“.env”].

Returns:

The loaded configuration dictionary keys and associated values.

Return type:

dict

os_helper.main.get_nb_workers() → int

Get the number of available CPU workers for parallel processing.

This function returns the number of CPU cores available on the machine. If the environment variable “NB_WORKERS” is set, its value will override the default CPU count, allowing you to control the number of workers manually.

Returns:

The number of available CPU cores, or the value of the “NB_WORKERS” environment variable (if set and valid).

Return type:

int

Example

>>> get_nb_workers()
8  # Depending on the number of CPU cores available

os_helper.main.get_user_ip() → Dict[str, str | None]

Fetches the user’s public IP addresses in both IPv4 and IPv6 formats, if available.

This function attempts to retrieve the user’s IP addresses using the ipify API. It tries separate endpoints for IPv4 and IPv6 and returns a dictionary containing both addresses, or None if an address could not be retrieved.

Returns:

dict of {str – A dictionary with the keys “ipv4” and “ipv6”. The values are either the IP addresses as strings or None if the address could not be fetched.

Return type:

Optional[str]}

os_helper.main.getpid() → str

Get the process ID of the current Python process.

Returns:

The process ID as a string.

Return type:

str

Example

>>> getpid()
'12345'

os_helper.main.hash_string(s: str, size: int = -1) → str

Generate a hash of a given string using the specified hashing algorithm.

This function hashes a string using the RIPMD-160 algorithm and returns the full hash or a truncated version based on the size parameter.

Parameters:

• s (str) – The input string to hash.

• size (int, optional) – If positive, truncates the hash to the specified length. Defaults to -1 (no truncation).

Returns:

The hashed string, optionally truncated to the specified size.

Return type:

str

Example

>>> hash_string("example")
'9c1185a5c5e9fc54612808977ee8f548b2258d31'
>>> hash_string("example", size=8)
'9c1185a5'

os_helper.main.hashfile(path: str, hash_content: bool = True, date: bool = False) → str

Generate a hash for a file’s content and/or its last modification date.

Parameters:

• path (str) – The path to the file to hash.

• hash_content (bool, optional) – If True, includes the file’s content in the hash. Defaults to True.

• date (bool, optional) – If True, includes the current date in the hash. Defaults to False.

Returns:

The resulting hash of the file.

Return type:

str

Example

>>> hashfile("example.txt")
'9c1185a5c5e9fc54612808977ee8f548b2258d31'

os_helper.main.hashfolder(path: str, hash_content: bool = True, hash_path: bool = False, date: bool = False) → str

Generate a hash for the contents of a folder and/or its path.

Parameters:

• path (str) – The path to the folder to hash.

• hash_content (bool, optional) – If True, includes the folder’s contents in the hash. Defaults to True.

• hash_path (bool, optional) – If True, includes the folder’s path in the hash. Defaults to False.

• date (bool, optional) – If True, includes the current date in the hash. Defaults to False.

Returns:

The resulting hash of the folder and/or its contents.

Return type:

str

Example

>>> hashfolder("/path/to/folder")
'9c1185a5c5e9fc54612808977ee8f548b2258d31'

os_helper.main.info(msg: str) → None

Log an informational message: just a shortcut

Parameters:

msg (str) – The message to log.

Example

>>> info("Process completed successfully.")

os_helper.main.is_working_url(url: str) → bool

Check if a URL is valid and reachable.

This function validates a URL and attempts to send a HEAD request to check if the URL is reachable.

Parameters:

url (str) – The URL to check.

Returns:

True if the URL is valid and reachable, False otherwise.

Return type:

bool

Example

>>> is_working_url("https://www.google.com")
True

os_helper.main.join(*args) → str

Join multiple path elements into a single path and convert it to an absolute path.

Parameters:

*args (strings) – Path elements to join.

Returns:

The absolute path after joining the elements.

Return type:

str

Example

>>> join("folder1", "subfolder2", "file.txt")
'/absolute/path/to/folder1/subfolder2/file.txt'

os_helper.main.linux() → bool

Determine if the current operating system is Linux.

This function checks the platform identifier to determine if the script is running on a Linux OS.

Returns:

True if the operating system is Linux, False otherwise.

Return type:

bool

Example

>>> linux()
True  # if running on Linux

os_helper.main.macos() → bool

Determine if the current operating system is macOS.

This function checks the platform identifier to determine if the script is running on a macOS.

Returns:

True if the operating system is macOS, False otherwise.

Return type:

bool

Example

>>> macos()
True  # if running on macOS

os_helper.main.make_directory(folder_path: str, exist_ok: bool = True) → None

Create a directory, optionally ignoring if it already exists.

Parameters:

• folder_path (str) – Path to the directory to create.

• exist_ok (bool, optional) – If True, does not raise an error if the directory already exists. Defaults to True.

Example

>>> make_directory("/path/to/folder")

os_helper.main.now_string(format: str = 'log') → str

Get the current timestamp as a formatted string.

This function generates a string representation of the current date and time. The format can be adjusted for logging or file naming purposes.

Parameters:

format (str, optional) – The format of the timestamp. “log” (default) provides a log-friendly format (YYYY/MM/DD-HH:MM:SS). “filename” replaces special characters (e.g., slashes and colons) with hyphens to make the string file-system safe.

Returns:

A string representing the current date and time.

Return type:

str

Example

>>> now_string("log")
'2024/10/10-14:33:21'

>>> now_string("filename")
'2024-10-10-14-33-21'

os_helper.main.open(filename: str, mode: str = 'r', buffering: int = -1, encoding: str = 'utf-8', errors=None, newline=None, closefd: bool = True, opener=None)

UTF-8 enforced open function.

This function overrides the built-in open() function to enforce UTF-8 encoding by default when reading or writing text files. It falls back to the default behavior for binary files.

Parameters:

• filename (str) – The path to the file to be opened.

• mode (str, optional) – The mode in which the file is opened (e.g., “r” for reading, “w” for writing). Defaults to “r”.

• buffering (int, optional) – Buffering policy. Defaults to -1 (use system default).

• encoding (str, optional) – Encoding to use for text files. Defaults to “utf-8”.

• errors (str, optional) – Error handling strategy (e.g., “ignore”, “replace”). Defaults to None.

• newline (str, optional) – How to handle newlines. Defaults to None.

• closefd (bool, optional) – Whether to close the file descriptor after use. Defaults to True.

• opener (callable, optional) – Custom file opener. Defaults to None.

Returns:

The opened file object.

Return type:

file object

Example

>>> with open("example.txt", "w") as f:
...     f.write("Hello, world!")

os_helper.main.openfile(filename: str) → None

Open a file with the default application based on the operating system.

This function attempts to open a file using the system’s default application for the file type, based on the current operating system (Windows, Linux, or macOS).

Parameters:

filename (str) – The path to the file to open.

Example

>>> openfile("example.txt")

os_helper.main.os_path_constructor(ell: list) → str

Construct a path from a list of path elements.

Parameters:

ell (list) – List of path components.

Returns:

The constructed absolute path.

Return type:

str

Example

>>> os_path_constructor(["/home/user", "folder", "file.txt"])
'/home/user/folder/file.txt'

os_helper.main.path_without_home(path: str) → str

Convert an absolute path to be relative to the user’s home directory.

This function replaces the home directory portion of the given path with a tilde (~), making it shorter and easier to read while still maintaining functionality. It works on both Unix-like systems and Windows.

Parameters:

path (str) – The absolute file or directory path to convert.

Returns:

The path with the home directory replaced by ~, or the original path if the home directory is not part of the path or if the path is already relative to ~.

Return type:

str

Example

>>> path_without_home("/home/user/project/file.txt")
'~/project/file.txt'

>>> path_without_home("/Users/myuser/project/file.txt")
'~/project/file.txt'  # On macOS

>>> path_without_home("C:\Users\myuser\project\file.txt")
'~\project\file.txt'  # On Windows

os_helper.main.recursive_glob(root_dir: str, pattern: str) → list

Perform a recursive search for files matching a specified pattern.

This function searches for files within a directory (and its subdirectories) that match a given pattern using glob.

Parameters:

• root_dir (str) – The root directory to start the recursive search from.

• pattern (str) – The file matching pattern (e.g., “*.txt” for text files).

Returns:

A list of file paths that match the specified pattern.

Return type:

list

Example

>>> recursive_glob("/home/user", "*.txt")
['/home/user/file1.txt', '/home/user/docs/file2.txt']

os_helper.main.relative2absolute_path(path: str, checkpath: bool = False) → str

Convert a relative path to an absolute path.

Parameters:

• path (str) – The relative or absolute file path.

• checkpath (bool, optional) – If True, checks if the path exists. Defaults to False.

Returns:

The absolute path.

Return type:

str

Example

>>> relative2absolute_path("relative/path")
'/absolute/relative/path'

os_helper.main.remove_directory(folder_path: str) → None

Remove a directory and its contents.

Parameters:

folder_path (str) – Path to the directory to remove.

Example

>>> remove_directory("/path/to/folder")

os_helper.main.remove_files(files_list: list, verbose: bool = False) → None

Remove a list of files.

Parameters:

• files_list (list) – List of file paths to remove.

• verbose (bool, optional) – If True, logs the removal of each file. Defaults to False.

Example

>>> remove_files(["file1.txt", "file2.txt"], verbose=True)

os_helper.main.size_file(filepath: str) → int

Get the size of a file in bytes.

Parameters:

filepath (str) – The path to the file.

Returns:

The size of the file in bytes, or -1 if the file does not exist.

Return type:

int

Example

>>> size_file("example.txt")
1024

os_helper.main.system(cmd: str, expected_output: str = '', check_exitcode: bool = True, check_empty: bool = False) → dict

Run a system command and return its output.

Parameters:

• cmd (str) – The system command to run.

• expected_output (str, optional) – Expected output file or directory. If provided, checks for its existence after the command runs. Defaults to “”.

• check_exitcode (bool, optional) – If True, checks that the command returns a zero exit code. Defaults to True.

• check_empty (bool, optional) – If True, checks if the expected output is not empty. Defaults to False.

Returns:

A dictionary with the command’s output (‘out’) and error messages (‘err’).

Return type:

dict

Example

>>> system("ls -la", expected_output="/path/to/file")
{'out': '...', 'err': ''}

os_helper.main.temporary_filename(suffix: str = '', mode: str = 'wt', prefix: str = '') → str

Create a temporary file with a unique name that persists even after closing.

This function generates a temporary file with a unique name, which is removed after use. It can be used to safely create and use a temporary file for writing or reading data.

Parameters:

• suffix (str, optional) – File suffix (e.g., .txt). Defaults to “”.

• mode (str, optional) – Mode in which the file is opened (e.g., “wt” for writing text). Defaults to “wt”.

• prefix (str, optional) – Prefix for the file name. Defaults to “”.

Yields:

str – The name of the temporary file.

Example

>>> with temporary_filename(suffix=".txt") as temp_file:
...     with open(temp_file, "w") as f:
...         f.write("Temporary content")

os_helper.main.temporary_folder(prefix: str = '') → str

Create a temporary directory with a unique name that persists during the context.

This function generates a temporary folder with a unique name, which is removed after use. It can be used to safely create and work within a temporary directory.

Parameters:

prefix (str, optional) – Prefix for the folder name. Defaults to “”.

Yields:

str – The name of the temporary directory.

Example

>>> with temporary_folder(prefix="tempdir") as temp_directory:
...     print(f"Temporary folder created: {temp_directory}")
...     # Do work inside the temporary folder

os_helper.main.tic() → float

Start a timer and return the current time.

This function is used to capture the start time, which can then be passed to toc() to calculate the elapsed time.

Returns:

The current time in seconds since the epoch (the same as time.time()).

Return type:

float

Example

>>> start_time = tic()

os_helper.main.time2str(time: float, no_space: bool = False) → str

Convert time in seconds to a human-readable string format (hours, minutes, seconds).

Parameters:

• time (float) – Time in seconds.

• no_space (bool, optional) – If True, removes spaces between numbers and units (default is False).

Returns:

A formatted string representing the time in hours, minutes, and seconds.

Return type:

str

Examples

>>> time2str(3661)
'1 hr 1 min 1 sec'
>>> time2str(61)
'1 min 1 sec'
>>> time2str(61, no_space=True)
'1min1sec'

os_helper.main.toc(t: float) → float

Stop the timer and return the elapsed time in seconds.

Parameters:

t (float) – The start time captured by the tic() function.

Returns:

The elapsed time in seconds since tic() was called.

Return type:

float

Example

>>> start_time = tic()
>>> # some operations
>>> elapsed_time = toc(start_time)
>>> print(f"Elapsed time: {elapsed_time:.2f} seconds")

os_helper.main.unix() → bool

Determine if the current operating system is Unix-based (Linux or macOS).

This function checks if the operating system is either Linux or macOS, which are both Unix-based systems.

Returns:

True if the operating system is Unix-based (Linux or macOS), False otherwise.

Return type:

bool

Example

>>> unix()
True  # if running on Linux or macOS

os_helper.main.valid_config_file(a_path: str, keys: list, config_type: str) → dict

Check if a configuration file is valid by verifying it contains the required keys.

This function loads a configuration file (in JSON or YAML format) and checks if it contains all the specified keys.

Parameters:

• a_path (str) – The path to the configuration file.

• keys (list) – List of keys that must be present in the configuration file.

• config_type (str) – The type of configuration (for logging purposes).

Returns:

The loaded configuration dictionary if valid, otherwise None.

Return type:

dict or None

os_helper.main.verbosity(level: int = None) → int

Set or retrieve the current verbosity level for logging.

This function allows for dynamic adjustment of the verbosity level of the logging system, enabling different levels of logging details to be output (e.g., errors only, detailed debugging information, etc.). If no level is provided, it returns the current verbosity level.

Parameters:

• level (int, optional) – The verbosity level to set (0: None, 1: Error, 2: Info, 3: Debug). If None, retrieves the current verbosity level. Defaults to None.

• Levels (Verbosity)

• 0 (-)

• 1 (-)

• 2 (-)

• 3 (-)

Returns:

The current verbosity level after setting it (if a new level was provided).

Return type:

int

Example

>>> verbosity(2)  # Set verbosity to show info-level messages
>>> verbosity()   # Retrieve the current verbosity level (2)

os_helper.main.windows() → bool

Determine if the current operating system is Windows.

This function checks the platform identifier to determine if the script is running on a Windows OS.

Returns:

True if the operating system is Windows, False otherwise.

Return type:

bool

Example

>>> windows()
True  # if running on Windows

os_helper.main.zip_folder(folder_path: str, zip_file_path: str = '')

Zips the contents of a folder (including subdirectories) into a ZIP file.

Parameters:

• folder_path (str) – Path to the folder to be zipped.

• zip_file_path (str, optional) – Path to the output ZIP file. If not provided, defaults to folder_path.zip.

Return type:

None