What would disprove this analysis? (Criterion 1)

AI customer profiling produces equitable outcomes across demographic groups without fairness constraints

What would disprove this analysis? (Criterion 2)

Dynamic pricing based on AI profiling benefits consumers overall compared to uniform pricing

What would disprove this analysis? (Criterion 3)

Proxy discrimination in AI models is detectable and correctable through standard auditing practices

When should you avoid ai customer profiling discrimination?

Your AI profiling model uses variables that correlate strongly with protected characteristics. You can't explain why different customers receive different prices or service levels

What are alternatives?

Transparent tiered pricing: Published price tiers based on objective, non-discriminatory criteria. Fairness-constrained AI: Models trained with explicit fairness constraints that limit disparate impact. Opt-in personalization: Let customers choose whether to share data for personalized pricing

Catalog

A030

AI & Automation

AI Customer Profiling Discrimination

HIGH(80%)

February 2026

4 sources

Context

Companies use AI to profile customers for personalized pricing, credit decisions, insurance rates, and service tiers. The pitch is efficiency — serve each customer optimally based on their data. But AI profiling systematically discriminates through proxy variables. Zip code correlates with race. Browser type correlates with income. Purchase history correlates with health status. The AI doesn't need protected characteristics to discriminate — it finds proxies that achieve the same result while maintaining plausible deniability. Dynamic pricing charges higher prices to customers identified as less price-sensitive (often wealthier, often whiter neighborhoods). Credit algorithms deny loans to qualified borrowers in historically redlined areas. Insurance models charge more to people with certain shopping patterns. The discrimination is invisible because it's statistical, individualized, and hidden behind proprietary algorithms.

Hypothesis

What people believe

“AI improves customer targeting and delivers personalized experiences.”

Actual Chain

→

Proxy discrimination replaces explicit discrimination(Protected characteristics inferred from behavioral data)

└

Zip code proxies for race in credit and insurance decisions

└

Device and browser data proxies for income level

└

Discrimination becomes invisible and harder to prove legally

→

Dynamic pricing exploits information asymmetry(Same product, different prices based on profile)

└

Price-insensitive customers charged 10-30% more

└

Vulnerable populations pay more for essential services

└

Consumer trust erodes when differential pricing discovered

→

Feedback loops amplify existing inequality(Denied services → worse data → more denials)

└

Credit denials reduce credit history, leading to more denials

└

Higher insurance prices in poor areas reduce coverage, increasing risk

Impact

Metric	Before	After	Delta
Price variation for identical products	Minimal	10-30% based on profile	+20%
Credit approval disparity by zip code	Regulated (explicit)	Unregulated (proxy-based)	Shifted not reduced
Targeting efficiency	Demographic segments	Individual-level profiling	+500%
Discrimination detectability	Auditable (explicit criteria)	Opaque (algorithmic)	-80%

Navigation

Don't If

•Your AI profiling model uses variables that correlate strongly with protected characteristics
•You can't explain why different customers receive different prices or service levels

If You Must

1.Audit AI models for disparate impact across protected groups regularly
2.Remove proxy variables that correlate with race, gender, or other protected characteristics
3.Provide transparency into how pricing and service decisions are made
4.Implement fairness constraints in model training

Alternatives

Transparent tiered pricing — Published price tiers based on objective, non-discriminatory criteria
Fairness-constrained AI — Models trained with explicit fairness constraints that limit disparate impact
Opt-in personalization — Let customers choose whether to share data for personalized pricing

Falsifiability

This analysis is wrong if:

AI customer profiling produces equitable outcomes across demographic groups without fairness constraints
Dynamic pricing based on AI profiling benefits consumers overall compared to uniform pricing
Proxy discrimination in AI models is detectable and correctable through standard auditing practices

Sources

1.
ProPublica: Machine Bias
Landmark investigation showing AI systems discriminate through proxy variables
2.
White House OSTP: Blueprint for an AI Bill of Rights
Federal framework addressing algorithmic discrimination in automated decision systems
3.
Cathy O'Neil: Weapons of Math Destruction
Comprehensive analysis of how AI profiling amplifies inequality through feedback loops
4.
Federal Reserve: Fair Lending and AI
Regulatory guidance on proxy discrimination in AI-based credit decisions

A012 A014 A018