.. core-db documentation master file, created by sphinx-quickstart on Wed Apr 2 23:12:04 2025. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. core-db =============================================================================== This project/library contains common elements related to database engines and provides clients to simplify the connections... =============================================================================== .. image:: https://img.shields.io/pypi/pyversions/core-db.svg :target: https://pypi.org/project/core-db/ :alt: Python Versions .. image:: https://img.shields.io/badge/license-MIT-blue.svg :target: https://gitlab.com/bytecode-solutions/core/core-db/-/blob/main/LICENSE :alt: License .. image:: https://gitlab.com/bytecode-solutions/core/core-db/badges/release/pipeline.svg :target: https://gitlab.com/bytecode-solutions/core/core-db/-/pipelines :alt: Pipeline Status .. image:: https://readthedocs.org/projects/core-db/badge/?version=latest :target: https://readthedocs.org/projects/core-db/ :alt: Docs Status .. image:: https://img.shields.io/badge/security-bandit-yellow.svg :target: https://github.com/PyCQA/bandit :alt: Security Documentation Contents ------------------------------------------------------------------------------- .. toctree:: :maxdepth: 1 :caption: Index: interfaces engines testing_clients Features ------------------------------------------------------------------------------- **Multiple Database Engine Support** - PostgreSQL (via psycopg) - MySQL (via PyMySQL) - Oracle (via oracledb) - MS SQL Server (via pyodbc) - MongoDB (via pymongo) - IBM DB2 (via ibm-db) - Snowflake (via snowflake-connector-python) **Security** - Parameterized queries with placeholders to prevent SQL injection - Automatic column name validation against injection patterns - String value escaping for safe SQL operations - Secure identifier validation (alphanumeric and underscore only) **SQL Operations** - SELECT queries with optional column specification - Batch INSERT operations with configurable chunk sizes (default: 500 records per batch) - DELETE operations with conditional clauses or primary key filtering - MERGE/UPSERT operations (database-specific implementations) - CREATE TABLE DDL generation with type mapping and optional PRIMARY KEY, UNIQUE, and NOT NULL constraints - Support for temporary table creation **Data Retrieval** - ``fetch_one()``: Fetch single record as tuple - ``fetch_record()``: Fetch single record as dictionary with column names - ``fetch_all()``: Fetch all records as iterator of tuples - ``fetch_records()``: Fetch all records as iterator of dictionaries - Automatic column metadata extraction **Connection Management** - Context manager support (``with`` statement) for automatic cleanup - Automatic connection establishment and closure - Automatic commit on exit - Connection testing capabilities - Factory pattern for dynamic client instantiation **Type Handling** - Python-to-SQL type mapping (int, float, str, bool, dict, list) - JSON type support for dict/list serialization - Database-specific type conversions - Timestamp/epoch conversion functions **ETL Support** - Abstract base classes for database-based ETL processes - Batch-based record processing with configurable batch sizes - Connection lifecycle management within ETL workflows - Integration with core-etl framework - Dynamic query generation for incremental loads **Developer Experience** - Comprehensive type hints throughout the codebase - Detailed docstrings with usage examples - Abstract interfaces for custom implementations - Multi-row INSERT for optimal performance - Standardized interface across all database engines Installation ------------------------------------------------------------------------------- Install from PyPI using pip: .. code-block:: bash pip install core-db uv pip install core-db # Or using UV... pip install -e ".[dev]" # For development... Setting Up Environment ------------------------------------------------------------------------------- 1. Install required libraries: .. code-block:: bash pip install --upgrade pip pip install virtualenv 2. Create Python virtual environment: .. code-block:: bash virtualenv --python=python3.12 .venv 3. Activate the virtual environment: .. code-block:: bash source .venv/bin/activate Install packages ------------------------------------------------------------------------------- .. code-block:: bash pip install . pip install -e ".[dev]" Check tests and coverage ------------------------------------------------------------------------------- .. code-block:: shell python manager.py run-tests python manager.py run-coverage Contributing ------------------------------------------------------------------------------- Contributions are welcome! Please: 1. Fork the repository 2. Create a feature branch 3. Write tests for new functionality 4. Ensure all tests pass: ``python manager.py run-tests`` 5. Run linting: ``pylint core_db`` 6. Run security checks: ``bandit -r core_db`` 7. Submit a pull request License ------------------------------------------------------------------------------- This project is licensed under the MIT License. See the LICENSE file for details. Links ------------------------------------------------------------------------------- * **Documentation:** https://core-db.readthedocs.io/en/latest/ * **Repository:** https://gitlab.com/bytecode-solutions/core/core-db * **Issues:** https://gitlab.com/bytecode-solutions/core/core-db/-/issues * **Changelog:** https://gitlab.com/bytecode-solutions/core/core-db/-/blob/master/CHANGELOG.md * **PyPI:** https://pypi.org/project/core-db/ Support ------------------------------------------------------------------------------- For questions or support, please open an issue on GitLab or contact the maintainers. Authors ------------------------------------------------------------------------------- * **Alejandro Cora González** - *Initial work* - alek.cora.glez@gmail.com