Introduction to Sparvi

Data Profiling Guide

Validation Rules Guide

Monitors

Anomaly Detection Guide

Sparvi Cloud Overview

Sparvi Cloud Database Connections

## What is Sparvi?

Sparvi is a comprehensive data observability platform that helps you catch data issues early and resolve what impacts business.

### Sparvi Cloud
A full-featured data observability platform for data teams, featuring:
- Interactive data exploration and lineage discovery
- Monitors that watch row counts, column statistics, and custom SQL on a schedule
- Segmented metrics so per-region or per-tenant problems never hide in the average
- Issue management with stakeholder notifications
- Team collaboration and automated monitoring
- Enterprise security with SSO and role-based access

Sparvi Cloud helps data teams maintain high-quality data, prevent issues before they impact business operations, and build confidence in their data assets.

## Core Concepts

### Data Observability

Data observability provides comprehensive visibility into your data's health, quality, and relationships across your organization. Sparvi implements data observability through:

- **Monitors**: Schedule a metric, built-in (row count, null %, distinct count, min, max, avg, stddev) or any custom SQL, and let Sparvi raise issues when its value deviates
- **Data Profiling**: Automated analysis of data structure, completeness, and statistical patterns
- **Lineage Discovery**: Understanding how data flows through your systems and who depends on it
- **Issue Management**: Tracking and resolving data problems with stakeholder awareness
- **Business Impact Analysis**: Understanding which teams and processes are affected by data issues

### Monitors

A monitor is the unit of observability in Sparvi. Each monitor:

- Targets one of three source types: **built-in table metric** (row count), **built-in column metric** (null %, distinct, min, max, avg, stddev), or **custom SQL** (any numeric query)
- Runs in one of two evaluation modes: **ML / statistical** (z-score, IQR, or moving-average detection against a learned baseline) or **threshold** (warning and critical levels you set yourself)
- Runs on its own schedule, every N minutes/hours, daily, weekly, or manual-only
- Can optionally be **segmented** by a dimension column so each segment value gets its own baseline and its own alert path

Anomalies in Sparvi are simply what happens when a monitor in statistical mode flags a value that falls outside the learned baseline.

### Segmented Metrics

Any monitor can be segmented by a dimension column, for example, row count per region, signup count per channel, p95 latency per tenant, or revenue per product line. Sparvi tracks each segment independently:

- Each segment has its own baseline (for statistical evaluation) or its own threshold (for threshold evaluation)
- Issues name the offending segment so on-call work is actionable on the first read
- The dashboard collapses per-segment values into one overall number using your choice of rollup (sum, average, min, or max) and shows all four side by side in the detail view

### Validation Rules

Validation rules enforce hard contracts on your data. Sparvi Cloud provides:

- **SQL-Based Rules**: Define validation criteria using familiar SQL syntax
- **Health Dashboards**: Visual monitoring of validation performance over time
- **Automated Scheduling**: Run validations on configurable schedules
- **Issue Integration**: Failed validations automatically create tracked issues with business context

Validations answer pass/fail questions on known rules; monitors track numeric metrics over time and alert on deviation. Most teams use both.

### Metadata Management

Sparvi automatically collects and tracks metadata changes across your data infrastructure:

- **Schema Change Detection**: Identify new, modified, or deleted columns
- **Relationship Discovery**: Find dependencies between tables, views, and downstream systems
- **Historical Tracking**: Maintain a complete audit trail of all metadata changes
- **Impact Analysis**: Understand which systems and teams are affected by changes

## Architecture Overview

### Sparvi Cloud Architecture
- **Web Application**: React-based frontend with comprehensive dashboards and workflows
- **API Layer**: RESTful APIs for all platform functionality
- **Database Connectors**: Native integrations with Snowflake, BigQuery, and dbt Core (Redshift and dbt Cloud coming H2 2026)
- **Automation Engine**: Scheduled monitor evaluation, validation runs, and metadata collection
- **Notification System**: Multi-channel alerting with Azure Communication Services
- **Enterprise Security**: Multi-factor authentication, role-based access, user management

## Key Features

- **Monitors**: ML or threshold detection on row counts, column statistics, or custom SQL, segmented when you want it
- **Data Explorer**: Navigate database schemas, tables, and columns with comprehensive metadata
- **Lineage Discovery**: Automatically map data relationships and business dependencies
- **Issue Management**: Track and resolve data quality problems with business impact context
- **Validations**: Custom SQL rules with scheduling and alerting
- **Team Collaboration**: User management, notifications, and shared workflows
- **Enterprise Integrations**: Snowflake, BigQuery, dbt Core, automated scheduling


Learn about Sparvi's core concepts and architecture

Data profiling is the process of examining data to understand its structure, content, quality, and patterns. With Sparvi, you can easily profile your database tables to gain deep insights into your data.

## Understanding Sparvi Profiles

When you profile a table with Sparvi, it computes a variety of metrics and statistics that give you a comprehensive view of your data. A profile includes:

### Basic Metrics

- **Row count**: Total number of rows in the table
- **Column count**: Total number of columns
- **Duplicate count**: Number of duplicate rows
- **Timestamp**: When the profile was created

### Completeness Metrics

For each column, Sparvi calculates:

- **Null count**: Number of NULL values
- **Null percentage**: Percentage of NULL values
- **Distinct count**: Number of unique values
- **Distinct percentage**: Percentage of unique values

### Numeric Column Statistics

For numeric columns, Sparvi calculates:

- **Min/Max**: Minimum and maximum values
- **Mean/Median**: Average and middle values
- **Standard Deviation**: Measure of data dispersion
- **Quartiles**: 25th, 50th, and 75th percentiles
- **Outliers**: Values that fall significantly outside the normal range

### Text Column Statistics

For text columns, Sparvi analyzes:

- **Min/Max/Avg Length**: Character length statistics
- **Pattern Recognition**: Common patterns in the data
- **Format Consistency**: Checks for consistent formatting
- **Empty String Detection**: Identifies empty strings (different from NULLs)

### Date Column Statistics

For date and timestamp columns, Sparvi calculates:

- **Min/Max Date**: Earliest and latest dates
- **Date Range**: Span of time represented
- **Temporal Patterns**: Distribution across time periods
- **Future Dates**: Detection of dates in the future
- **Unrealistic Dates**: Identification of dates that are unreasonably old

### Feeding Profiles into Monitors

Profile statistics are also the source data for column-metric monitors. The same null %, distinct count, min, max, avg, and stddev that the profile surfaces can be promoted to a monitor in two clicks, with ML or threshold evaluation. Segment any of them by a dimension column to track quality per region, product, or tenant.

This is how the typical Sparvi feedback loop closes:

- **Statistical outliers**, surface in the profile; promote to a column-metric monitor with statistical evaluation to alert on future drift
- **Distribution shifts**, visible in the histogram; promote to an avg or stddev monitor
- **Completeness issues**, visible in the null % column; promote to a null % monitor with a threshold ("never above 5%") or a statistical baseline
- **Schema shifts**, picked up automatically by Sparvi's schema-change tracking; no monitor required

## Using Data Profiling in Sparvi Cloud

Sparvi Cloud provides automated data profiling through the web interface:

1. **Connect Your Database**: Add your database connection in the Connections section
2. **Navigate to Data Explorer**: Browse your database schemas and tables
3. **View Profiles**: Click on any table to see its comprehensive data profile
4. **Automated Scheduling**: Configure automated profiling schedules for continuous monitoring
5. **Historical Comparison**: Track changes over time with automatic historical comparisons
6. **Promote to monitors**: Add a monitor on any column statistic so future drift creates an issue automatically

## Analyzing Profiles in Sparvi Cloud

Sparvi Cloud provides comprehensive profile analysis through interactive dashboards and visualizations.

### Viewing Profile Metrics

In the Data Explorer, when you view a table profile, Sparvi Cloud displays:

- **Table Overview**: Row count, column count, duplicate count, and profile timestamp
- **Column Completeness Dashboard**: Visual representation of null percentages and distinct value counts for all columns
- **Interactive Charts**: Sortable and filterable views of all column statistics

### Identifying Data Quality Issues

Sparvi Cloud automatically highlights problematic columns:

- **High Null Rates**: Columns with elevated null percentages are flagged with warning indicators
- **Low Cardinality Issues**: Columns with very low distinct value percentages are identified
- **Sorting and Filtering**: Sort columns by null percentage, distinct count, or other metrics to quickly find issues

### Numeric Column Analysis

For numeric columns, Sparvi Cloud provides:

- **Statistical Summary**: View min, max, average, median, and standard deviation
- **Distribution Visualizations**: Histograms and box plots showing data distribution
- **Outlier Detection**: Automatically identified outliers with example values
- **Trend Analysis**: Historical comparison of numeric statistics over time

### Monitor Outcomes Alongside Profiles

For tables that have monitors attached, the profile view shows each monitor's current value, baseline (or threshold), and last evaluation timestamp. Click into a monitor to see its trend chart, segmented breakdown (if segmented), and full issue history.

- **Monitor list**: Every monitor on the table, with status (healthy, warning, critical)
- **Severity levels**: Critical and warning issues derived from monitor evaluations
- **Schema change tracking**: Automatic detection of schema shifts with before/after comparison
- **Filterable views**: Filter monitors by status, evaluation mode, or affected column

### Dashboard and Reporting

The Sparvi Cloud dashboard provides comprehensive visualizations:

- **Monitor trends over time**: Line charts showing each monitor's value, baseline, and crossings
- **Severity breakdown**: Visual distribution of critical and warning issues
- **Issue type analysis**: Charts showing whether issues come from monitors, validations, or schema changes
- **Table health heatmap**: Color-coded view of data quality across all tables
- **Export options**: Download reports and share with your team

## Best Practices

1. **Profile Regularly**: Run profiles on a regular schedule to track changes over time
2. **Use Historical Comparisons**: Always compare current profiles with previous ones
3. **Start with Basic Metrics**: Begin analysis with row counts and null percentages
4. **Focus on Outliers**: Pay special attention to statistical outliers
5. **Look for Patterns**: Analyze patterns in text data for inconsistencies
6. **Check Date Ranges**: Ensure date ranges fall within expected bounds
7. **Track Schema Changes**: Monitor changes to table structures over time
8. **Automate Alerting**: Set up alerts for significant changes in key metrics
9. **Tune monitors**: Adjust sensitivity, baseline window, or thresholds on the monitors derived from your profiles so signal matches your tolerance for noise
10. **Document Findings**: Keep records of data quality issues for future reference

## Next Steps

After profiling your data, consider:

1. Setting up [Validation Rules](/docs/validation-rules) based on profile insights
2. Exploring the [Anomaly Detection Guide](/docs/anomaly-detection), how Sparvi monitors flag values that fall outside a learned baseline, including segment-aware detection
3. Connecting your databases with [Sparvi Cloud Database Connections](/docs/sparvi-cloud-databases)

Learn how to profile your data with Sparvi

Validation rules are the heart of data quality enforcement with Sparvi. They allow you to define specific expectations about your data and verify if those expectations are met.

## Understanding Validation Rules

A validation rule in Sparvi consists of:

1. **Name**: A unique identifier for the rule
2. **Description**: A human-readable explanation of what the rule checks
3. **Query**: The SQL query that performs the validation check
4. **Operator**: How to compare the query result with the expected value
5. **Expected Value**: The value that indicates a passing validation

For example, a rule might check if there are any orders with zero or negative amounts by counting rows where amount is less than or equal to zero. The rule passes if the count equals 0, meaning there are no such orders.

## Validation Rule Anatomy

### Query

The `query` is a SQL statement that returns a single value (typically a COUNT or other aggregate). This value will be compared against the expected value using the specified operator.

Common validation queries include:
- Checking for null values in required columns
- Validating data formats (emails, dates, etc.)
- Ensuring values fall within reasonable date ranges
- Verifying referential integrity between related tables

### Operators

Sparvi supports several comparison operators:

- **equals**: The query result must exactly match the expected value
- **greater_than**: The query result must be greater than the expected value
- **less_than**: The query result must be less than the expected value
- **between**: The query result must fall between two values (requires an array for expected_value)
- **not_equals**: The query result must not match the expected value (opposite of equals)

### Expected Value

The `expected_value` field defines what the query result should be for the validation to pass. The type depends on the operator:

- For `equals`, `greater_than`, `less_than`, and `not_equals`: A single value (number, string, or boolean)
- For `between`: An array of two values `[min, max]`

## Types of Validation Rules

### Data Integrity Rules

These rules check basic integrity constraints:

- **Primary Key Uniqueness**: Ensure all IDs are unique with no duplicates
- **Foreign Key Validity**: Verify that all foreign key references point to valid records in related tables
- **Not Null Constraints**: Check that required fields contain values

### Data Quality Rules

These rules check for data quality issues:

- **Format Validation**: Ensure emails, phone numbers, and other formatted fields follow valid patterns
- **Range Checks**: Verify numeric values fall within reasonable bounds (e.g., ages between 18 and 120)
- **Date Validity**: Confirm dates are not in the future when they shouldn't be
- **Pattern Consistency**: Check that text fields match expected patterns

### Business Logic Rules

These rules enforce business-specific requirements:

- **Conditional Requirements**: Ensure premium users have required information like phone numbers
- **Transaction Verification**: Check that high-value orders or transactions have been properly verified
- **Workflow Validation**: Verify records progress through business processes correctly
- **Relationship Rules**: Ensure related entities maintain expected relationships

### Distribution Rules

These rules check data distributions:

- **Value Distribution**: Ensure no single value dominates the data (e.g., no status represents more than 80% of records)
- **Balance Checks**: Verify data is distributed reasonably across categories
- **Ratio Validation**: Check that proportions between different values remain within expected ranges

## Working with Validation Rules in Sparvi Cloud

### Creating Validation Rules

In Sparvi Cloud, you can create validation rules through the intuitive web interface:

1. **Navigate to Validation Rules**: Access the Validation Rules section from the main navigation
2. **Create New Rule**: Click "Create Rule" to open the rule builder
3. **Define Rule Properties**:
   - Enter a unique name for your rule
   - Write a clear description of what the rule checks
   - Write or select a SQL query that performs the validation
   - Choose the comparison operator (equals, greater_than, less_than, between, not_equals)
   - Set the expected value that indicates a passing validation
4. **Test Your Rule**: Run a test to verify the rule works as expected
5. **Save and Activate**: Save the rule and add it to your validation suite

### Generating Default Rules

Sparvi Cloud can automatically generate sensible validation rules based on your table structure and data patterns:

- Primary key uniqueness checks
- Foreign key validity checks
- Null checks for non-nullable columns
- Format validation for emails, phone numbers, etc.
- Range checks for numeric columns
- Date range validations
- Pattern consistency for text columns

Use the "Generate Rules" feature in the Validation Rules section to create a starting set of rules for your tables.

### Running Validation Rules

In Sparvi Cloud, validation rules can be configured and run through the web interface:

1. **Create Rules**: Define validation rules in the Validation Rules section
2. **Schedule Runs**: Configure automated schedules for your validation rules
3. **Monitor Results**: View validation results in the Health Dashboards
4. **Track Issues**: Failed validations automatically create tracked issues with business context

## Validation Rule Best Practices

### Writing Effective Rules

1. **Use Clear Names**: Rule names should indicate what is being checked
2. **Write Descriptive Explanations**: Descriptions should explain the business reasoning
3. **Keep Queries Simple**: Focus each rule on a single validation check
4. **Validate at the Right Level**: Some checks need row-level detail, others need aggregates
5. **Consider Performance**: Avoid complex joins in validation queries for large tables
6. **Use Parameterized Thresholds**: Set realistic thresholds based on your data volume

### Organizing Validation Rules

Sparvi Cloud helps you organize your rules effectively:

- **Categories and Tags**: Group rules by category (data integrity, data quality, business logic)
- **Table Association**: Link rules to specific tables or schemas
- **Search and Filter**: Quickly find rules using search and filtering
- **Rule Collections**: Create collections of related rules for specific data domains

### Validation Rule Management

As your validation suite grows in Sparvi Cloud:

1. **Rule History**: Track changes to rules with automatic version history
2. **Documentation**: Add notes and documentation directly to each rule
3. **Testing**: Test rules on sample data before activating them in production
4. **Organization**: Group related rules by domain, table, or business function
5. **Prioritization**: Set severity levels (critical, warning, info) for different validations

## Advanced Validation Techniques

### Dynamic Expected Values

Sometimes you need to compare against a dynamic baseline rather than a fixed value. Sparvi Cloud supports:

- **Time-Based Comparisons**: Compare today's metrics against yesterday's or last week's values
- **Relative Thresholds**: Set validation thresholds based on historical averages or trends
- **Adaptive Baselines**: Use calculated values that adjust based on recent data patterns

For example, you can create rules that ensure today's order volume doesn't drop more than 50% compared to yesterday's volume.

### Conditional Validations

Some validations only apply under certain conditions. In Sparvi Cloud you can create:

- **Context-Aware Rules**: Validations that apply only to specific subsets of data
- **If-Then Logic**: Rules that check conditions only when certain criteria are met
- **Segmented Validation**: Different validation thresholds for different data segments

For example, you might validate that international orders have tax IDs, while domestic orders don't require this field.

### Statistical Validations

For data with natural variation, Sparvi Cloud supports statistical approaches:

- **Outlier Detection**: Identify values beyond a certain number of standard deviations from the mean
- **Distribution Checks**: Verify data follows expected statistical distributions
- **Tolerance Ranges**: Set acceptable ranges based on statistical measures rather than fixed values

For example, you can create rules that allow a small number of outliers (e.g., up to 10 values beyond 3 standard deviations) while still flagging when too many extreme values appear.

## Validation Rule Monitoring

Sparvi Cloud provides comprehensive monitoring of validation rule performance:

**Health Dashboards**: Track validation pass rates over time with visual trends

**Failure Tracking**: See which validations are failing most frequently

**Historical Trends**: Understand how data quality evolves over time

**Alert Configuration**: Set up notifications when validation rules start failing

**Team Visibility**: Share validation results with your team through dashboards and issues

## Next Steps

After mastering validation rules:

1. Learn about [Anomaly Detection](/docs/anomaly-detection), how Sparvi monitors flag metrics that fall outside a learned baseline
2. Explore [Data Profiling](/docs/data-profiling) to understand your data patterns and the statistics that feed column-metric monitors
3. Check out [Sparvi Cloud Overview](/docs/sparvi-cloud-overview) to learn about all platform features

Monitors are how Sparvi watches the metrics that matter. A monitor combines a **source** (what to measure), an **evaluation mode** (how to decide when to alert), and a **schedule** (when to run). When a monitor falls outside its expected range, Sparvi raises an issue with the metric, the segment (if applicable), and the historical trend.

## Anatomy of a Monitor

Every monitor has four parts:

1. **Source**, what numeric value to read from your warehouse
2. **Evaluation mode**, ML / statistical (learns a baseline) or threshold (warning + critical lines you set)
3. **Schedule**, interval, daily, weekly, or manual-only
4. **Segmentation (optional)**, split the metric by a dimension column so each segment gets its own baseline and its own alert path

## Source Types

There are three:

### Built-in table metric

A single value computed across the whole table. The only built-in table metric today is `row_count`. Use this for volume monitoring, partial loads, missed runs, duplicated batches.

### Built-in column metric

A single value computed across one column. Available metrics:

- `null_pct`, percent of values that are NULL
- `distinct_count`, number of distinct values
- `distinct_pct`, distinct / total ratio
- `min`, `max`, `avg`, `stddev`, numeric statistics on numeric columns

No SQL required, pick the table, the column, and the metric.

### Custom SQL

Write any query that returns a numeric column. Alias the numeric column as `value` (or whatever you specify in the **value column** field) and Sparvi reads the metric from there. Use this for:

- Business metrics: revenue, signups, churn rate, MRR
- Freshness: `DATEDIFF('minute', MAX(updated_at), CURRENT_TIMESTAMP()) AS value`
- Cross-table checks: JOINs and CTEs that no built-in metric covers
- Per-source health: split a single table's metric by upstream provider

Custom SQL monitors can also be **segmented**: alias a dimension column as `segment` (or your chosen name) and Sparvi runs one evaluation per segment value.

## Evaluation Modes

### ML / Statistical

Sparvi learns a baseline from history and alerts when the latest value deviates beyond your configured sensitivity. Three methods:

- **Z-score**, number of standard deviations from baseline mean. Good general-purpose default.
- **IQR (Interquartile Range)**, robust to outliers in the baseline window. Good when occasional legitimate spikes would otherwise inflate the standard deviation and mask future anomalies.
- **Moving average**, compares the latest value to a rolling mean. Good when the metric trends naturally over time.

Sensitivity ranges from 0.5 (less sensitive, fewer alerts) to 2.0 (more sensitive, more alerts). The baseline window defaults to 14 days and can be set anywhere from 1 to 90 days. The **minimum data points** guard prevents firing until enough history has been collected.

This is where Sparvi's anomaly detection lives, an "anomaly" is just what happens when a statistical monitor's latest value falls outside the learned baseline.

### Threshold

Set explicit warning and critical lines. The monitor alerts when the value crosses them. Pick a direction:

- **Below**, alert when the value drops below the line (e.g. revenue per region must stay above $50K/day)
- **Above**, alert when the value goes above the line (e.g. p95 latency must stay under 300ms)

You can set warning only, critical only, or both. When both are set, Sparvi enforces consistency (for `below`, critical must be ≤ warning; for `above`, critical must be ≥ warning).

Use threshold mode when you have a hard contract. Use statistical mode when "normal" drifts with the business.

## Segmented Metrics

Any monitor can be segmented by a dimension column. With segmentation enabled:

- Sparvi computes the metric once per segment value
- Each segment has its own baseline (statistical) or its own threshold check
- Issues name the segment so the alert is actionable on the first read
- The dashboard collapses per-segment results into one **overall** value using your chosen rollup: sum, avg, min, or max
- The detail view shows all four rollups side by side and lets you sort segments by deviation

For built-in metrics, choose any column on the same table. For custom SQL, alias your dimension column in the query (for example `region AS segment`) and name it in the **segment column** field.

### When to segment

Segment when an aggregate would hide the problem. Common patterns:

- Revenue, signups, or orders **per region** or **per channel**
- Latency, error rate, or throughput **per tenant**
- Row counts **per upstream source**
- Quality metrics **per environment** (dev, staging, prod)
- Distributions **per product line** or **per app version**

## Scheduling

Each monitor picks its own cadence:

- **Interval**, every N minutes or hours. Minimum total interval is 5 minutes.
- **Daily**, at a specific hour and minute in any IANA timezone
- **Weekly**, choose the day of week and time
- **Manual**, only runs when you click Run Now

Monitors also have an **active** flag. Toggle it off to pause without losing configuration or history; toggle it on to resume.

## Issues and Alerts

When a monitor fires, Sparvi creates an issue containing:

- The monitor name and its current value
- The baseline (statistical) or threshold (threshold mode)
- The segment value, if the monitor is segmented
- The historical trend so context is one click away
- AI-suggested resolution based on similar past issues
- Downstream lineage so you can see which dashboards or pipelines depend on the affected data

Alerts route via Slack, email, or PagerDuty according to your notification preferences. Each user can choose between immediate alerts and a daily digest.

## Best Practices

1. **Start with row counts**, every critical table deserves a row count monitor with statistical evaluation. It is the cheapest monitor and the highest-signal one for catching pipeline regressions.
2. **Use threshold mode for contracts**, SLAs and business minimums belong in threshold monitors, not statistical ones.
3. **Segment when an aggregate would hide the problem**, revenue per region, latency per tenant, row counts per source.
4. **Set sensible schedules**, hourly tables can be monitored every 10–60 minutes; reference tables that change daily do not need sub-hour cadence.
5. **Tune sensitivity after a week**, collect a baseline first, then dial sensitivity up or down based on how often you actually act on alerts.
6. **Pause noisy monitors instead of deleting them**, the active flag preserves configuration so you can come back after the upstream stabilizes.

## Next Steps

1. Read the [Anomaly Detection Guide](/docs/anomaly-detection) for a deeper look at statistical evaluation
2. Set up [Validation Rules](/docs/validation-rules) for hard pass/fail contracts
3. Explore [Data Profiling](/docs/data-profiling) to find the column statistics worth monitoring


Track row counts, column statistics, or custom SQL on a schedule, with ML or threshold detection, segmented when you need it

Anomaly detection in Sparvi happens through **monitors**. A monitor watches a metric on a schedule, learns a baseline from history, and flags an anomaly when the latest value falls outside what is normal. This guide explains how the statistical evaluation mode works, when to use it, and how it pairs with thresholds and segmented metrics.

## How Anomaly Detection Works

Every Sparvi monitor has an **evaluation mode**. There are two:

- **ML / statistical**, Sparvi learns a baseline from your historical data and alerts when the latest value deviates beyond your configured sensitivity. This is the mode that produces anomalies.
- **Threshold**, You set explicit warning and critical lines. The monitor alerts when the value crosses them. No anomaly framing, it is either inside or outside the contract.

Statistical mode supports three detection methods:

- **Z-score**, How many standard deviations the latest value sits from the baseline mean. Good general-purpose default.
- **IQR (Interquartile Range)**, Robust to outliers in the baseline window. Good when the metric has occasional legitimate spikes you do not want the baseline to absorb.
- **Moving average**, Compares the latest value to a rolling mean. Good when the metric trends naturally over time and a static mean would drift.

Each monitor also has a configurable **sensitivity** (0.5–2.0; higher fires on smaller deviations), a **baseline window** in days (default 14), and a **minimum data points** guard so detection does not fire on a thin history.

## What You Can Detect

Anomaly detection runs on whatever metric the monitor is configured to track. Sparvi's three source types cover most cases:

### Row count anomalies

A built-in table metric monitor on `row_count` plus statistical evaluation catches:

- Partial loads (today's volume sits well below baseline)
- Duplicate runs (volume sits well above baseline)
- Failed pipelines (volume flatlines)
- Upstream outages (volume drops on one source but not others)

Example: `orders.row_count` averages 10K rows/hour over the last 14 days; today's hour reads 500. Sparvi raises an issue with the metric, the baseline, and the deviation.

### Column metric anomalies

Built-in column-metric monitors (null %, distinct count, distinct %, min, max, average, standard deviation) plus statistical evaluation catch:

- Null spikes (null % on email column jumps from 2% to 15% after an upstream change)
- Distribution shifts (average order value jumps from $50 to $5,000)
- Cardinality drops (distinct count on customer_id falls off, an ID column is now constant)
- Range escapes (min or max of a numeric column exits its historical band)

### Custom SQL anomalies

A custom SQL monitor plus statistical evaluation lets you detect anomalies on any metric you can express as a numeric query: revenue per hour, p95 latency, churn rate, freshness, signups by source. The baseline is learned from the values your query has returned over time.

### Segment-level anomalies

Any monitor can be **segmented** by a dimension column. With segmentation enabled:

- Each segment value (e.g. each region, each tenant) has its own baseline
- Statistical evaluation runs per segment so a deviation in one slice fires its own alert
- Issues name the segment so on-call work is actionable on the first read

This is the difference between "revenue is down 5% overall" (probably noise) and "EMEA revenue is down 62% while AMER and APAC are flat" (immediately actionable).

## Configuring a Monitor for Anomaly Detection

In Sparvi Cloud, create a monitor:

1. **Basics**, Name, optional display name, description, and active flag.
2. **What to measure**, Pick a source type:
   - **Table metric (row count)**, choose a table.
   - **Column metric**, choose a table, a column, and a metric (null %, distinct count, distinct %, min, max, avg, stddev).
   - **Custom SQL**, paste a query that returns a numeric column, and name the alias of that column.
3. **Segment by (optional)**, Pick a column to segment by. For built-in metrics, pick any column on the same table. For custom SQL, alias a dimension column in the query (for example `region AS segment`) and reference it in the segment column field.
4. **How to alert**, Choose **ML / Statistical**, then pick a detection method (z-score, IQR, or moving average), a sensitivity, and a baseline window.
5. **Schedule**, Run every N minutes/hours, daily at a specific local time, weekly on a chosen day, or manually.

Save the monitor. Sparvi collects baseline data on the configured schedule and starts firing alerts once the minimum-data-points guard has been satisfied.

## Anomaly Types in Detail

### Volume anomalies

Row-count monitors detect unusual changes in table volume. Severity is derived from how far the value sits from the baseline.

Example: "orders row count down 92% from baseline (12,418 → 1,012)", critical severity.

### Null rate anomalies

Null % monitors on a column flag unexpected upstream changes, ETL bugs, or data source issues.

Example: "users.email null % rose from 2% to 15% over the last 6 hours", warning severity.

### Distribution anomalies

Column metric monitors on `avg`, `stddev`, `min`, or `max` catch shifts in value distributions that point to calculation errors, unit changes, or data corruption.

Example: "orders.amount avg jumped from $52.18 to $5,218, likely cents-to-dollars conversion regression", critical severity.

### Freshness anomalies

A custom SQL monitor on `DATEDIFF('minute', MAX(updated_at), CURRENT_TIMESTAMP())` plus statistical evaluation surfaces pipeline failures, source outages, and scheduling issues.

Example: "orders freshness lag is 4 hours; baseline is under 10 minutes", critical severity.

### Uniqueness anomalies

`distinct_count` and `distinct_pct` monitors catch unintended duplicates, ID collisions, and cardinality drift.

Example: "user_id distinct % fell from 100% to 84%, duplicates introduced", critical severity.

### Segment anomalies

Anomalies on segmented monitors are scoped to a single segment value. The issue names the segment so the alert is immediately actionable.

Example: "revenue per region, EMEA down 62% from baseline; AMER and APAC normal", critical severity, scoped to `region = 'EMEA'`.

## Working with Anomalies in Sparvi Cloud

### Filtering and viewing

In the dashboard you can:

- Filter by severity (critical, warning)
- Filter by monitor or by table
- Filter by segment value (for segmented monitors)
- View the trend chart for any monitor, with the baseline band overlaid

### Taking action

When a monitor fires, Sparvi automatically creates an issue with:

1. The metric name, the current value, the baseline, and the deviation
2. The segment value (if the monitor is segmented)
3. The historical trend so you can see context at a glance
4. AI-suggested resolution based on similar past issues
5. Downstream lineage so you can see which dashboards or pipelines depend on the affected data

You can assign the issue, comment in context, and resolve it without leaving Sparvi.

## Tuning Sensitivity

Some advice for getting useful signal:

1. **Start with the defaults**, sensitivity 1.0, baseline window 14 days, z-score detection. Adjust only after you have a few days of data and can see which alerts you actually act on.
2. **Lower sensitivity for noisy metrics**, if a monitor pages weekly on benign blips, drop sensitivity toward 0.5 (less sensitive) or switch to IQR.
3. **Raise sensitivity for critical columns**, payment amounts, primary keys, and similar should fire on smaller deviations.
4. **Use threshold mode when you have a hard contract**, "p95 latency must stay below 300ms" is not an anomaly question; it is a threshold. Flip the monitor to threshold mode and set the line.
5. **Segment when an aggregate would hide the problem**, per-region revenue, per-tenant latency, per-source row counts. Per-segment baselines surface localized issues that overall baselines smooth out.
6. **Set a meaningful minimum data points**, for a once-daily monitor, three is fine; for a once-an-hour monitor, you might want a week's worth before firing.

## Anomaly Detection vs. Validation

Both create issues and route alerts, but they answer different questions:

- **Validation**, pass/fail check against a known rule you write in SQL. "Orders must have positive amounts." If a row violates the rule, the validation fails.
- **Anomaly detection (statistical monitor)**, track a metric over time and alert when it falls outside the baseline learned from history. "Order volume per hour should look like the last 14 days of order volume per hour."

Use validations when you know the rule. Use monitors when you do not, or when the bar moves with the business. Most teams use both.

## Next Steps

After mastering anomaly detection:

1. Set up [Validation Rules](/docs/validation-rules) to enforce hard contracts alongside statistical monitors
2. Explore [Data Profiling](/docs/data-profiling) to see the column statistics that feed into column-metric monitors
3. Learn about [Sparvi Cloud Database Connections](/docs/sparvi-cloud-databases) to connect Snowflake, BigQuery, or dbt Core


Learn how Sparvi monitors detect anomalies using statistical evaluation


Sparvi Cloud is a comprehensive data observability platform designed for teams who need collaborative data quality monitoring, issue management, and business impact analysis.

## What Makes Sparvi Cloud Different

Unlike traditional data quality tools that just detect technical issues, Sparvi Cloud provides **business-aware data observability** with automatic stakeholder identification and impact analysis.

### Traditional Approach
```
❌ ORDERS table has 1,247 null customer_ids
Assigned to: @data-engineer
```

### Sparvi Cloud Approach
```
🚨 CRITICAL: ORDERS data quality issue
📊 BUSINESS IMPACT:
  • Sales Dashboard → 12 daily users affected
  • Commission Calculator → Finance team blocked
  • Customer Segmentation → Marketing campaigns paused

👥 AUTO-NOTIFIED:
  • @data-team (technical details)
  • @sales-team (dashboard broken)  
  • @finance-team (payroll delayed)
```

## Core Features

### Data Explorer
Navigate through your database schemas, tables, and columns with comprehensive metadata:
- **Hierarchical Navigation**: Browse databases → schemas → tables → columns
- **Data Preview**: View actual data samples and understand content
- **Metadata Insights**: Row counts, column types, and data patterns
- **Search and Filtering**: Find specific tables and objects quickly

### Data Lineage Discovery
Automatically discover and visualize data relationships:
- **Relationship Mapping**: Understand how data flows through your systems
- **Dependency Analysis**: See what depends on each data source
- **Business Impact**: Identify stakeholders affected by data issues
- **Confidence Scoring**: Know how reliable discovered relationships are

### Monitors
Watch the metrics that matter on the schedule that matters:
- **Three source types**: built-in table metrics (row count), built-in column metrics (null %, distinct count, min, max, avg, stddev), or any custom SQL
- **Two evaluation modes**: ML / statistical detection (z-score, IQR, moving average) that learns a baseline from history, or threshold mode with explicit warning and critical lines
- **Segmented metrics**: any monitor can be segmented by a dimension column so per-region or per-tenant problems never hide in the average
- **Flexible scheduling**: interval (every N minutes/hours), daily, weekly, or manual-only

### Issue Management System
Track and resolve data quality problems with business context:
- **Automatic Issue Creation**: From failed validations, monitor alerts, and schema changes
- **Business Impact Analysis**: Understand which teams and processes are affected
- **Stakeholder Notifications**: Alert the right people when issues affect their work
- **Resolution Tracking**: Monitor progress and collaborate on fixes

### Custom Validations
Create and manage data quality rules with comprehensive monitoring:
- **SQL-Based Rules**: Use familiar SQL syntax for custom validation logic
- **Health Dashboards**: Visual monitoring of validation performance over time
- **Automated Scheduling**: Run validations on configurable schedules
- **Rule Management**: Organize and maintain validation rule libraries

### Team Collaboration
Enterprise-ready user management and collaboration:
- **Organization Management**: Invite team members and manage access
- **Role-Based Access**: Control who can see and modify different data sources
- **Multi-Factor Authentication**: Secure access with MFA support
- **User Invitations**: Easy onboarding for new team members

## Database Support

Sparvi Cloud provides native integrations with:

### Snowflake
- **OAuth Authentication**: Secure, token-based authentication
- **Key-Pair Authentication**: Support for Snowflake key-pair authentication
- **Warehouse Management**: Specify compute warehouses for different operations
- **Role Support**: Use specific Snowflake roles for data access

### BigQuery
- **Service Account Authentication**: Connect via Google Cloud service account credentials
- **Project and Dataset Scoping**: Limit Sparvi to the projects and datasets it needs
- **Region Support**: Native handling of dataset locations

### dbt Core
- **Manifest Parsing**: Read your dbt project to map models, sources, and tests into Sparvi
- **Test Surfacing**: dbt test results appear next to Sparvi monitors and validations
- **Lineage Import**: Use your dbt graph as the starting point for cross-warehouse lineage

Redshift and dbt Cloud are coming in H2 2026.

## Getting Started

### 1. Sign Up and Connect
- Create your Sparvi Cloud account
- Connect your first database (Snowflake or PostgreSQL)
- Configure authentication (OAuth recommended for Snowflake)

### 2. Discover Your Data
- Run automatic data discovery to map your database structure
- Explore tables and understand data patterns
- Review automatically generated metadata

### 3. Set Up Monitoring
- Create validation rules for critical data quality checks
- Add monitors on the metrics that matter, row counts, column statistics, business KPIs via custom SQL, with ML or threshold evaluation
- Segment monitors by region, product, or tenant when an aggregate would hide the problem
- Set per-monitor schedules so noisy data is checked more often than slow-changing tables

### 4. Invite Your Team
- Add team members to your organization
- Configure roles and permissions
- Set up notification preferences

### 5. Monitor and Improve
- Review data quality dashboards regularly
- Respond to issues with business context
- Continuously refine validation rules and thresholds

## Pricing

Sparvi Cloud is a single flat plan at $999/month, every user on your team, every supported warehouse, every feature. A 14-day full-feature free trial is available; no credit card required. Enterprise quotes are available on request for customers with $1B+ revenue or specific SSO/audit/SLA requirements.

For current pricing and features, visit our [pricing page](/pricing).

## Enterprise Features

### Security and Compliance
- Multi-factor authentication (MFA)
- Role-based access control (RBAC)
- Audit logging and activity tracking
- SOC 2 compliance preparation

### Advanced Integrations
- REST API for custom integrations
- Webhook notifications for external systems
- Single Sign-On (SSO) support (coming soon)
- Custom notification channels

### Dedicated Support
- Priority support response times
- Dedicated customer success manager
- Custom onboarding and training
- Architecture review and optimization

## Next Steps

1. **Try Sparvi Cloud**: [Sign up for early access](/#waitlist)
2. **Schedule a Demo**: See Sparvi Cloud in action with your data
3. **Join Design Partner Program**: Get early access and influence our roadmap
4. **Read Getting Started Guide**: Learn how to connect your first database

## Need Help?

- **Documentation**: Browse our comprehensive guides
- **Support**: Contact our team for technical assistance
- **Community**: Join discussions with other data teams
- **Feedback**: Share your ideas and feature requests

Comprehensive data observability platform for teams


This guide covers how to connect your databases to Sparvi Cloud and configure authentication for secure access.

## Supported Databases

Sparvi Cloud provides native, optimized integrations with:

- **Snowflake**, Username/password and key-pair authentication
- **BigQuery**, Service account authentication, project and dataset scoping
- **dbt Core**, Manifest parsing to map models, sources, and tests into Sparvi

Redshift and dbt Cloud are coming in H2 2026.

## Snowflake Integration

Sparvi Cloud offers two authentication methods for Snowflake:

### Username/Password Authentication (Recommended)

Standard username and password authentication for Snowflake connections.

**Setup Steps:**
1. **In Sparvi Cloud**: Navigate to Connections → Add New Connection → Snowflake
2. **Choose Username/Password**: Select standard authentication
3. **Configure Connection**: Provide your Snowflake connection details:
   - Account identifier (e.g., `xy12345.us-east-1`)
   - Username
   - Password
   - Database name
   - Schema name (default: `PUBLIC`)
   - Warehouse name
4. **Test Connection**: Verify the connection works correctly

**Benefits:**
- Simple setup process
- Works with existing Snowflake accounts
- No additional configuration required
- Compatible with all Snowflake deployments

### Key-Pair Authentication

For environments where password authentication isn't preferred, Sparvi Cloud supports Snowflake key-pair authentication.

**Setup Steps:**
1. **Generate Key Pair**: Create RSA private/public key pair
2. **Configure Snowflake User**: Add public key to your Snowflake user account
3. **In Sparvi Cloud**: Choose "Key-Pair Authentication" option
4. **Upload Private Key**: Securely upload your private key file
5. **Configure Connection**: Provide connection details and user information
6. **Test Connection**: Verify authentication works

**Security Notes:**
- Private keys are encrypted at rest
- Keys are never logged or exposed
- Follows Snowflake security best practices
- Supports key rotation

### Required Snowflake Permissions

For both authentication methods, your Snowflake user needs these minimum permissions:

```sql
-- Database and schema access
GRANT USAGE ON DATABASE <database_name> TO ROLE <role_name>;
GRANT USAGE ON SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;

-- Table access (for all tables you want to monitor)
GRANT SELECT ON ALL TABLES IN SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;
GRANT SELECT ON FUTURE TABLES IN SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;

-- Information schema access (for metadata discovery)
GRANT USAGE ON DATABASE INFORMATION_SCHEMA TO ROLE <role_name>;

-- Warehouse usage
GRANT USAGE ON WAREHOUSE <warehouse_name> TO ROLE <role_name>;
```


## Connection Security

### Encryption at Rest
- All connection credentials are encrypted using AES-256 encryption
- Encryption keys are managed separately from data
- Regular key rotation following security best practices

### Encryption in Transit
- All database connections use SSL/TLS encryption
- Certificate verification enabled by default
- Support for custom certificates and CA bundles

### Access Controls
- Connection access controlled by organization membership
- Role-based permissions for connection management
- Audit logging for all connection-related activities

### Network Security
- Support for IP whitelisting
- VPC peering available for enterprise customers
- Private endpoint connections (coming soon)

## Connection Management

### Testing Connections
Before saving, Sparvi Cloud tests each connection:
- Verifies authentication credentials
- Checks database permissions
- Tests sample queries
- Validates SSL configuration

### Connection Health Monitoring
- Automatic connection health checks
- Alerts when connections become unavailable
- Connection performance monitoring
- Historical uptime tracking

### Credential Management
- Secure credential storage with encryption
- Support for credential rotation
- Integration with external secret managers (enterprise)
- Audit trail for credential changes

## Troubleshooting

### Common Snowflake Issues

**Username/Password Authentication Failed**
- Double-check username and password are correct
- Verify your Snowflake account identifier format
- Ensure user account is not locked or suspended
- Check that user has permission to use the warehouse

**Key-Pair Authentication Failed**
- Verify private key format (PKCS#8 or PKCS#1)
- Check that public key is correctly associated with user
- Ensure user account is not locked or suspended
- Verify role assignments and permissions

**Connection Timeout**
- Check network connectivity to Snowflake
- Verify warehouse is available and running
- Consider firewall and proxy configurations
- Try connecting from Snowflake web interface


## Best Practices

### Security
- Always use SSL/TLS encryption for production connections
- Implement least-privilege access principles
- Regular credential rotation (quarterly recommended)
- Monitor connection access logs
- Use key-pair authentication for enhanced security (Snowflake)

### Performance
- Choose appropriate Snowflake warehouse sizes
- Configure connection pooling settings
- Monitor query performance and resource usage
- Use efficient sampling strategies for large tables
- Schedule resource-intensive operations during off-peak hours

### Monitoring
- Set up connection health alerts
- Monitor data freshness and availability
- Track query performance metrics
- Review access patterns and usage
- Regular permission audits

## Enterprise Features

### Advanced Authentication
- Single Sign-On (SSO) integration (coming soon)
- LDAP/Active Directory integration
- Multi-factor authentication enforcement
- Custom authentication providers

### Network Security
- VPC peering for private connections
- IP whitelisting and access controls
- Private endpoints and service connections
- Custom DNS and network routing

### Compliance
- SOC 2 Type II certification
- GDPR compliance features
- Data residency options
- Audit logging and reporting

## Next Steps

1. **Connect Your Database**: Start with our connection wizard
2. **Verify Permissions**: Ensure your database user has required access
3. **Test Data Discovery**: Run initial metadata discovery
4. **Set Up Monitoring**: Configure validation rules and create monitors (ML or threshold, segmented when useful) on the metrics that matter
5. **Invite Team Members**: Add colleagues and configure roles

Need help connecting your database? [Contact our support team](/#contact) for personalized assistance.

How to connect Snowflake databases to Sparvi Cloud

Documentation

Sparvi Documentation