trailpack.validation.standard_validator ======================================= .. py:module:: trailpack.validation.standard_validator .. autoapi-nested-parse:: Standard validator for Trailpack data packages. Validates metadata, resources, fields, and data quality against the Trailpack standard specification. Classes ------- .. autoapisummary:: trailpack.validation.standard_validator.StandardValidator trailpack.validation.standard_validator.ValidationResult Module Contents --------------- .. py:class:: StandardValidator(version: str = '1.0.0') Validate data packages against Trailpack standards. The StandardValidator checks data packages for: - Metadata completeness (required and recommended fields) - Resource definitions (proper schema, formats, name sanitization) - Field definitions (types, units, constraints) - Data quality (missing values, duplicates, type consistency) - Schema matching (column types match field definitions) All numeric fields must have units specified, even for dimensionless quantities: - Measurements: Use appropriate SI or domain units (kg, m, °C, etc.) - Counts/IDs: Use dimensionless unit (http://qudt.org/vocab/unit/NUM) - Percentages: Use percent or dimensionless unit **Resource Name Sanitization:** Resource names must match ^[a-z0-9\-_.]+$. The validator automatically: - Detects invalid resource names - Suggests sanitized alternatives - Can auto-sanitize names with sanitize_resource_name() **Automatic Inconsistency Export:** When type inconsistencies are detected during validation (e.g., mixed types in a column), each inconsistent value is tracked and automatically exported to 'data_inconsistencies.csv' when the ValidationResult is printed. This provides a detailed breakdown for data cleaning workflows. .. rubric:: Example >>> validator = StandardValidator("1.0.0") >>> result = validator.validate_metadata(metadata) >>> if result.is_valid: ... print("✅ Valid!") ... else: ... print(result) >>> # Validate with schema (auto-exports inconsistencies.csv if errors found) >>> result = validator.validate_data_quality(df, schema=schema) >>> print(result) # Shows errors and exports CSV automatically >>> # Sanitize resource names >>> clean_name = validator.sanitize_resource_name("My File!") >>> print(clean_name) # "my_file" Initialize validator with a specific standard version. :param version: Standard version to validate against (default: "1.0.0") .. py:method:: _determine_level(result: ValidationResult) -> str Determine validation level based on errors and warnings. :param result: ValidationResult to evaluate :returns: Validation level badge string .. py:method:: _load_standard(version: str) -> Dict[str, Any] Load the standard specification from YAML. .. py:method:: _validate_data_against_schema(df: pandas.DataFrame, schema: Dict[str, Any], quality_spec: Dict[str, Any]) -> ValidationResult Validate DataFrame against field schema definitions. Checks that actual column types match declared field types, and that numeric fields have proper units defined. :param df: DataFrame to validate :param schema: Schema dictionary with field definitions :param quality_spec: Quality specification from standard :returns: ValidationResult with schema validation errors .. py:method:: _validate_field_value(field_name: str, value: Any, field_def: Dict[str, Any]) -> ValidationResult Validate a specific field value against its definition. :param field_name: Name of the field :param value: Value to validate :param field_def: Field definition from standard :returns: ValidationResult with validation errors .. py:method:: get_help_url(topic: str) -> Optional[str] Get help URL for a specific topic. :param topic: Topic name (e.g., 'frictionless_spec', 'qudt_units') :returns: URL string or None if not found .. py:method:: sanitize_resource_name(name: str) -> str Sanitize resource name to match the required pattern ^[a-z0-9\-_.]+$. The resource name must only contain: - Lowercase letters (a-z) - Numbers (0-9) - Hyphens (-) - Underscores (_) - Dots (.) :param name: Raw name string to sanitize :returns: Sanitized name matching the required pattern .. rubric:: Example >>> validator = StandardValidator() >>> validator.sanitize_resource_name("My Resource Name!") 'my_resource_name' >>> validator.sanitize_resource_name("Test@123") 'test123' .. py:method:: validate_all(metadata: Dict[str, Any], df: Optional[pandas.DataFrame] = None, mappings: Optional[Dict[str, Any]] = None) -> ValidationResult Validate everything: metadata, data quality, and mappings. :param metadata: Data package metadata dictionary :param df: Optional DataFrame to validate data quality :param mappings: Optional field mappings to validate :returns: ValidationResult with all validation results .. py:method:: validate_and_sanitize_resource_name(name: str, auto_fix: bool = False) -> Tuple[bool, str, Optional[str]] Validate a resource name and optionally sanitize it. :param name: Resource name to validate :param auto_fix: If True, return sanitized name; if False, just validate :returns: Tuple of (is_valid, original_or_sanitized_name, suggestion) - is_valid: Whether the original name is valid - original_or_sanitized_name: Original name if valid/not auto_fix, sanitized if auto_fix - suggestion: Sanitized name suggestion if original is invalid, None otherwise .. rubric:: Example >>> validator = StandardValidator() >>> is_valid, name, suggestion = validator.validate_and_sanitize_resource_name("Invalid Name!") >>> print(f"Valid: {is_valid}, Suggestion: {suggestion}") Valid: False, Suggestion: invalid_name >>> is_valid, name, _ = validator.validate_and_sanitize_resource_name("valid-name") >>> print(f"Valid: {is_valid}, Name: {name}") Valid: True, Name: valid-name .. py:method:: validate_data_quality(df: pandas.DataFrame, schema: Optional[Dict[str, Any]] = None) -> ValidationResult Validate data quality of a DataFrame. Data quality checks are logged as informational messages, not errors: - Missing data: Percentage of nulls per column - Duplicates: Percentage of duplicate rows Type consistency checks RAISE ERRORS (not just logged): - Mixed types: Columns with multiple Python types (e.g., strings and integers mixed) - Schema matching: Column types must match field definitions - Unit requirements: Numeric fields must have units (including dimensionless for IDs/counts) **Automatic Inconsistency Export:** When type inconsistencies are detected (mixed types in columns), each inconsistent value is tracked with its row number, column, actual type, and expected type. These inconsistencies are automatically exported to 'data_inconsistencies.csv' when the ValidationResult is printed. You can also manually export to a custom location using `result.export_inconsistencies_to_csv("custom_path.csv")`. :param df: DataFrame to validate :param schema: Optional schema with field definitions to validate against. Should contain 'fields' list with field definitions including: - name: Field name matching column name - type: Field type (string, integer, number, boolean, etc.) - unit: Unit definition (required for numeric fields) - description: Field description :returns: - errors: Type consistency violations (mixed types, schema mismatches) - info: Data quality metrics (nulls, duplicates) - inconsistencies: List of dicts with details about each inconsistent value (automatically exported to CSV when result is printed) :rtype: ValidationResult with .. rubric:: Example >>> schema = { ... "fields": [ ... { ... "name": "id", ... "type": "integer", ... "description": "Unique identifier", ... "unit": {"name": "dimensionless", "path": "http://qudt.org/vocab/unit/NUM"} ... }, ... { ... "name": "mass", ... "type": "number", ... "description": "Mass measurement", ... "unit": {"name": "kg", "path": "http://qudt.org/vocab/unit/KiloGM"} ... } ... ] ... } >>> result = validator.validate_data_quality(df, schema=schema) >>> # result.errors will contain type/schema mismatches and mixed type violations >>> # result.info will contain data quality observations (nulls, duplicates) .. note:: Identifier fields (with "id", "index", "identifier" in name or description) are automatically recognized and should use dimensionless units. :returns: ValidationResult with quality issues .. py:method:: validate_field_definition(field: Dict[str, Any]) -> ValidationResult Validate a field (column) definition. :param field: Field dictionary from schema :returns: ValidationResult with validation errors and warnings .. py:method:: validate_metadata(metadata: Dict[str, Any]) -> ValidationResult Validate metadata against required and recommended fields. :param metadata: Data package metadata dictionary :returns: ValidationResult with validation errors and warnings .. py:method:: validate_resource(resource: Dict[str, Any]) -> ValidationResult Validate a resource (data file) definition. Automatically checks and suggests sanitized names for invalid resource names. :param resource: Resource dictionary from metadata :returns: ValidationResult with validation errors and warnings .. py:attribute:: standard .. py:attribute:: version :value: '1.0.0' .. py:class:: ValidationResult Result of a validation check. Contains three types of messages: - errors: Type consistency violations that fail validation - warnings: Recommended fields or practices that should be addressed - info: Data quality metrics and informational messages A validation is considered valid (passed) if there are no errors, regardless of warnings or info messages. **Data Inconsistency Tracking:** When type inconsistencies are detected (e.g., mixed types in a column), each inconsistent value is tracked in the `inconsistencies` list. This list is automatically exported to 'data_inconsistencies.csv' when the result is printed or converted to string. The CSV file contains the row, column, value, actual type, and expected type for each inconsistency. .. attribute:: errors List of error messages (type consistency violations) .. attribute:: warnings List of warning messages (recommended practices) .. attribute:: info List of informational messages (data quality metrics) .. attribute:: level Validation compliance level (if assigned) .. attribute:: inconsistencies List of dicts with inconsistent value details .. py:method:: add_error(message: str, field: Optional[str] = None) Add an error message. .. py:method:: add_inconsistency(row: int, column: str, value: Any, actual_type: str, expected_type: str) Track a data type inconsistency for later export. This method is called automatically during validation when mixed types are detected in a column. Each inconsistent value (one that doesn't match the most common type in the column) is recorded with its location and type information. The inconsistencies are automatically exported to CSV when the ValidationResult is printed or can be manually exported using export_inconsistencies_to_csv(). :param row: Row index of the inconsistent value :param column: Column name where the inconsistency was found :param value: The actual inconsistent value :param actual_type: Python type name of the value (e.g., 'int', 'str') :param expected_type: Expected type based on most common type in column .. py:method:: add_info(message: str, field: Optional[str] = None) Add an info message. .. py:method:: add_warning(message: str, field: Optional[str] = None) Add a warning message. .. py:method:: export_inconsistencies_to_csv(output_path: str = 'data_inconsistencies.csv') Export data type inconsistencies to a CSV file for analysis. Creates a CSV file with details about each value that has an inconsistent type compared to the expected type in its column. This is useful for data cleaning workflows where you need to identify and fix specific problematic values. The CSV includes columns: row, column, value, actual_type, expected_type This method is called automatically when printing the ValidationResult if inconsistencies exist, but can also be called manually to export to a custom location. :param output_path: Path to the output CSV file. Defaults to "data_inconsistencies.csv" in the current working directory. :returns: Path to the created CSV file (str), or None if no inconsistencies to export. .. rubric:: Example >>> result = validator.validate_data_quality(df, schema) >>> if result.inconsistencies: ... csv_path = result.export_inconsistencies_to_csv("issues.csv") ... print(f"Found {len(result.inconsistencies)} issues in {csv_path}") .. py:method:: get_summary() -> str Get a summary of the validation result. .. py:attribute:: errors :type: List[str] :value: [] .. py:property:: has_warnings :type: bool Check if there are any warnings. .. py:attribute:: inconsistencies :type: List[Dict[str, Any]] :value: [] .. py:attribute:: info :type: List[str] :value: [] .. py:property:: is_valid :type: bool Check if validation passed (no errors). .. py:attribute:: level :type: Optional[str] :value: None .. py:attribute:: warnings :type: List[str] :value: []