Python Third-Party Library Selection Practice: Deep Thoughts and Experience Sharing from a Data Analyst-Jade Guqin Logic

Opening Thoughts

As a Python programmer, do you often encounter this dilemma: faced with numerous third-party libraries, how do you choose? Let's discuss this topic today.

I recently encountered an interesting case. A novice colleague working on a data analysis project asked me: "Teacher, I see pandas is popular online, but I found numpy seems to be able to do the same tasks and runs faster, so which one should I use?"

This question got me thinking deeply. Indeed, in the Python ecosystem, libraries with similar functionalities often appear, and how to choose is truly a topic worth discussing. Today, I'll share with you how to select and use Python third-party libraries based on my years of practical experience.

Environment Setup

Before starting our formal discussion, let's look at how to set up a clean, controllable Python development environment. Did you know that environment management is actually an important topic that many beginners tend to overlook?

Let's look at a basic environment setup example:

python3 -m venv data_analysis_env
source data_analysis_env/bin/activate  # Linux/Mac




pip install pandas numpy matplotlib scikit-learn


pip freeze > requirements.txt

This code demonstrates the basic environment setup process for a data analysis project. First, use Python's built-in venv module to create a virtual environment, which helps avoid dependency conflicts between different projects. Then install necessary libraries through pip, and finally export all dependencies in the current environment to a requirements.txt file. This file is very important as it serves as the project's "recipe," helping other developers quickly recreate your development environment.

Many people may have heard about the importance of virtual environments, but few actually pay attention to it. I previously encountered a real case: several projects in a team shared one Python environment, and when one project needed to upgrade a library version, it directly caused other projects to stop working. If virtual environments had been used at that time, this problem would not have occurred at all.

Library Selection Criteria

When it comes to selection criteria, I think we need to consider several dimensions.

Functional Completeness

The first dimension is functional completeness. Let me use the comparison between pandas and numpy as an example:

import numpy as np
import pandas as pd


numpy_array = np.array([[1, 2, 3], [4, 5, 6]])
print("Numpy result:")
print(numpy_array.mean(axis=0))


pandas_df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=['A', 'B', 'C'])
print("
Pandas result:")
print(pandas_df.mean())


print("
Pandas descriptive statistics:")
print(pandas_df.describe())

This code demonstrates the differences between numpy and pandas in data processing. Although both can calculate averages, pandas provides more built-in statistical functions, such as the describe() method which can provide multiple statistical indicators at once. Additionally, pandas' DataFrame structure supports column names, missing value handling, and many other features that are very practical in data analysis. From the code, we can see that although pandas is built on numpy, it provides higher-level abstractions and richer functionality in data analysis scenarios.

Performance Considerations

The second dimension is performance. Let's do a simple performance test:

import time
import numpy as np
import pandas as pd


size = 1000000
data = np.random.randn(size)


start_time = time.time()
np_mean = np.mean(data)
np_time = time.time() - start_time


df = pd.Series(data)
start_time = time.time()
pd_mean = df.mean()
pd_time = time.time() - start_time

print(f"Numpy calculation time: {np_time:.6f} seconds")
print(f"Pandas calculation time: {pd_time:.6f} seconds")
print(f"Pandas to Numpy time ratio: {pd_time/np_time:.2f}")

This code demonstrates the performance difference between numpy and pandas in large-scale data calculations through an actual performance test. By creating a dataset with 1 million random numbers and calculating the average using both libraries, it compares their execution times. This test reveals an important fact: in pure numerical computation scenarios, numpy is usually faster than pandas because pandas brings some performance overhead while providing more functionality. However, this performance difference is often acceptable in actual projects because the convenience and functionality provided by pandas usually compensate for this slight performance loss.

Community Activity

The third dimension is community activity. This point is particularly important but often overlooked. An active community means: - You can more easily find solutions to problems - The library will be continuously updated and maintained - Security vulnerabilities can be fixed promptly

Let's write a simple function to get basic information about libraries:

import pkg_resources
import requests
from datetime import datetime

def get_package_info(package_name):
    try:
        # Get locally installed version
        version = pkg_resources.get_distribution(package_name).version

        # Get information from PyPI
        response = requests.get(f"https://pypi.org/pypi/{package_name}/json")
        data = response.json()

        latest_version = data['info']['version']
        release_date = data['releases'][latest_version][0]['upload_time']

        # Calculate days since last update
        release_date = datetime.strptime(release_date, "%Y-%m-%dT%H:%M:%S")
        days_since_update = (datetime.now() - release_date).days

        return {
            'installed_version': version,
            'latest_version': latest_version,
            'days_since_last_update': days_since_update,
            'home_page': data['info']['home_page'],
            'project_urls': data['info']['project_urls']
        }
    except Exception as e:
        return f"Error getting info for {package_name}: {str(e)}"


libraries = ['pandas', 'numpy', 'requests']
for lib in libraries:
    info = get_package_info(lib)
    print(f"
{lib} library information:")
    for key, value in info.items():
        print(f"{key}: {value}")

This code creates a function to get detailed information about Python packages, including local installed version, latest version, last update time, etc. This information can help us evaluate a library's maintenance status and community activity. The function retrieves package metadata through PyPI's API interface and calculates the time since the last update. This tool can help us make more informed decisions when choosing libraries. For example, if a library hasn't been updated for a long time, even if its functionality is powerful, we should carefully consider whether to use it, as this might indicate potential security risks and compatibility issues.

Security Considerations

Speaking of this, we must address the topic of security. Many people only focus on functionality and performance when choosing third-party libraries while overlooking security. Let me share a simple security checking tool:

import subprocess
import json

def check_package_security(requirements_file):
    try:
        # Run safety check
        result = subprocess.run(
            ['safety', 'check', '-r', requirements_file, '--json'],
            capture_output=True,
            text=True
        )

        # Parse results
        vulnerabilities = json.loads(result.stdout)

        if not vulnerabilities:
            print("No security vulnerabilities found")
            return

        print(f"Found {len(vulnerabilities)} security issues:
")

        for vuln in vulnerabilities:
            print(f"Package name: {vuln['package']}")
            print(f"Current version: {vuln['installed_version']}")
            print(f"Vulnerability description: {vuln['description']}")
            print(f"Recommended update to: {vuln['fixed_version']}
")

    except Exception as e:
        print(f"Error during check: {str(e)}")


check_package_security("requirements.txt")

This code creates a function to check for security vulnerabilities in project dependencies. It uses the safety tool (a Python package security checking tool) to scan all dependency packages listed in the requirements.txt file and report any security issues found. The function outputs detailed information about each discovered vulnerability, including the affected package name, currently installed version, vulnerability description, and recommended version to update to. This tool is particularly suitable for regular running during project development and maintenance to ensure the third-party libraries being used are secure.

Dependency Management Practices

In actual projects, dependency management is a very important topic. Let me share a best practice for managing dependencies:

from pathlib import Path
import toml
import subprocess

def create_project_structure():
    """Create project basic structure and configuration files"""
    # Create project directories
    project_structure = {
        'src': {},
        'tests': {},
        'docs': {},
        'config': {}
    }

    for directory in project_structure:
        Path(directory).mkdir(exist_ok=True)

    # Create pyproject.toml
    pyproject_content = {
        'build-system': {
            'requires': ['poetry-core>=1.0.0'],
            'build-backend': 'poetry.core.masonry.api'
        },
        'tool': {
            'poetry': {
                'name': 'my-project',
                'version': '0.1.0',
                'description': 'Project description',
                'authors': ['Your Name <[email protected]>'],
                'dependencies': {
                    'python': '^3.8',
                    'pandas': '^1.3.0',
                    'numpy': '^1.21.0'
                },
                'dev-dependencies': {
                    'pytest': '^6.2.5',
                    'black': '^21.7b0',
                    'mypy': '^0.910'
                }
            }
        }
    }

    with open('pyproject.toml', 'w') as f:
        toml.dump(pyproject_content, f)

    return "Project structure created"

def install_dependencies():
    """Install project dependencies"""
    try:
        # Install poetry
        subprocess.run(['pip', 'install', 'poetry'], check=True)

        # Use poetry to install dependencies
        subprocess.run(['poetry', 'install'], check=True)

        return "Dependencies installation completed"
    except subprocess.CalledProcessError as e:
        return f"Error during installation: {str(e)}"


print(create_project_structure())
print(install_dependencies())

This code demonstrates how to use the Poetry tool to manage Python project dependencies. It first creates a standard project structure including source code, tests, documentation, and configuration directories. Then it creates a pyproject.toml file, which is Poetry's configuration file used to declare project metadata and dependencies. This approach has many advantages over traditional requirements.txt: it can precisely control dependency versions, automatically handle dependency conflicts, and supports separation of development and production dependencies. The use of this tool reflects best practices in modern Python project management.

Practical Case Analysis

Let's look at how to comprehensively apply this knowledge through an actual data analysis project:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
import matplotlib.pyplot as plt
import seaborn as sns
from typing import Tuple, Dict
import logging

class DataAnalysisProject:
    def __init__(self):
        self.logger = self._setup_logging()
        self.data = None
        self.model = None

    def _setup_logging(self):
        """Configure logging system"""
        logging.basicConfig(
            level=logging.INFO,
            format='%(asctime)s - %(levelname)s - %(message)s',
            handlers=[
                logging.FileHandler('analysis.log'),
                logging.StreamHandler()
            ]
        )
        return logging.getLogger(__name__)

    def load_data(self, file_path: str) -> None:
        """Load dataset"""
        try:
            self.data = pd.read_csv(file_path)
            self.logger.info(f"Successfully loaded data, shape: {self.data.shape}")
        except Exception as e:
            self.logger.error(f"Failed to load data: {str(e)}")
            raise

    def preprocess_data(self) -> Tuple[np.ndarray, np.ndarray]:
        """Data preprocessing"""
        try:
            # Handle missing values
            self.data = self.data.fillna(self.data.mean())

            # Feature engineering
            X = self.data.drop('target', axis=1)
            y = self.data['target']

            # Data splitting
            X_train, X_test, y_train, y_test = train_test_split(
                X, y, test_size=0.2, random_state=42
            )

            self.logger.info("Data preprocessing completed")
            return (X_train, X_test, y_train, y_test)

        except Exception as e:
            self.logger.error(f"Data preprocessing failed: {str(e)}")
            raise

    def train_model(self, X_train: np.ndarray, y_train: np.ndarray) -> None:
        """Train model"""
        try:
            self.model = RandomForestClassifier(n_estimators=100, random_state=42)
            self.model.fit(X_train, y_train)
            self.logger.info("Model training completed")
        except Exception as e:
            self.logger.error(f"Model training failed: {str(e)}")
            raise

    def visualize_results(self, feature_names: list) -> None:
        """Visualize results"""
        try:
            # Feature importance visualization
            importance = self.model.feature_importances_
            indices = np.argsort(importance)[::-1]

            plt.figure(figsize=(10, 6))
            plt.title("Feature Importance Ranking")
            plt.bar(range(len(importance)), importance[indices])
            plt.xticks(range(len(importance)), 
                      [feature_names[i] for i in indices], 
                      rotation=45)
            plt.tight_layout()
            plt.savefig('feature_importance.png')
            self.logger.info("Results visualization completed")

        except Exception as e:
            self.logger.error(f"Visualization failed: {str(e)}")
            raise


if __name__ == "__main__":
    project = DataAnalysisProject()
    project.load_data("dataset.csv")
    X_train, X_test, y_train, y_test = project.preprocess_data()
    project.train_model(X_train, y_train)
    project.visualize_results(project.data.columns.tolist())

This code demonstrates a complete data analysis project structure, integrating multiple popular data science libraries. It implements a complete machine learning workflow including data loading, preprocessing, model training, and results visualization. This example shows how to combine different libraries in actual projects: pandas for data processing, scikit-learn for machine learning, matplotlib for visualization. Additionally, the code implements comprehensive logging systems and exception handling mechanisms, which are essential in production environments. Through this example, we can see how different libraries work together and how to apply the various best practices discussed earlier in actual projects.

Experience Summary

Through these examples and discussions, I want to summarize several key experiences in selecting and using third-party libraries:

When choosing libraries, consider functional completeness, performance, and community activity comprehensively. Don't pursue performance while ignoring maintainability, and don't choose overly complex solutions just for comprehensive functionality.
Security cannot be ignored. Regularly check the security status of dependencies and update versions promptly. Though tedious, these tasks are very important.
Establish good dependency management mechanisms from the project's beginning. Using virtual environments and modern package management tools can make projects easier to maintain and collaborate on.

Finally, I want to say that choosing third-party libraries is not a one-time deal but an ongoing process. You need to regularly evaluate the libraries used in your project to ensure they are still the best choice. If you find better alternatives, don't let inertia prevent you from making improvements.

Do you find these suggestions helpful? Feel free to share your thoughts and experiences in the comments section.