Articles
Objective Data: Definition, Examples, and Interview Use
Learn what objective data means, how it differs from subjective data, and how engineers use it in analytics and experimentation interviews.
Objective Data: Definition, Examples, and How Engineers Use It in Analytics Interviews
Objective data is evidence grounded in observable or verifiable measurement: counts, timestamps, sensor readings, experiment assignments, transactions, clinical measurements, system logs, and audited outcomes. Subjective data captures opinions, perceptions, preferences, or interpretations.
In analytics and engineering interviews, one trap comes up constantly: standardized collection does not guarantee objective truth. A biased survey, broken event log, miscalibrated sensor, or flawed classifier can be repeatable and still wrong. I separate three questions before trusting the numbers:
- Data quality: Was the data logged completely, consistently, and without avoidable errors?
- Measurement validity: Does the metric actually measure the concept we care about?
- Causal validity: Does the analysis support the decision or causal claim?
Objective data is useful only when all three hold up.
1. What "Objective Data" Means in Analytics & Experimentation Interviews
Objective data comes from observable events or measurements collected under explicit rules. That does not mean the whole pipeline is free of judgment. Subjectivity can enter through event taxonomy, survey wording, labeling guidelines, classifier thresholds, instrumentation choices, inclusion rules, and aggregation logic.
Objective data vs subjective data
| Data type | Example | Objective? | Caveat |
|---|---|---|---|
| Business records | Purchase amount, refund timestamp, shipment status | Usually yes | Definitions and system completeness still matter |
| Healthcare measurements | Blood pressure, lab value, medication dose | Usually yes | Sensor calibration, protocol, and timing matter |
| Research measurements | Reaction time, task completion, error count | Usually yes | Construct validity and sampling matter |
| Product logs | Impressions, clicks, hides, session starts | Mixed | Logs can capture retries, prefetches, bots, cached views, background events, or UI artifacts |
| System telemetry | Latency, crashes, memory, API errors | Mixed | Depends on sampling, clock sync, platform coverage, dropped logs, and crash definitions |
| Surveys | Satisfaction rating, preference ranking | Subjective response; objective metadata | Rating is subjective; response time, sample frame, and completion status are objective |
| Human labels | Spam label, task success label | Structured judgment | Requires guidelines, inter-rater checks, audit samples |
| Text, audio, video, sensor traces | Complaint mentions, transcript events, near-miss detection | Not automatically objective | Extraction adds classification error, model bias, thresholds, and privacy constraints |
The borderline cases are where interviewers usually push. A five-star rating is not an objective measure of quality; it is a subjective judgment stored in a structured field. Counting five-star ratings is objective. Inferring that "quality improved" still requires validity checks.
Core concepts candidates should know
- Accuracy: How close the measurement is to the truth.
- Precision: How consistent or granular the measurement is.
- Reliability: Whether repeated measurement gives similar results.
- Validity: Whether the metric measures the intended construct.
- Construct validity: Whether "meaningful social interaction" is captured by likes, comments, replies, conversation depth, or something else.
- Measurement error: Random or systematic difference between recorded data and reality.
- Proxy metric: A measurable stand-in for a harder concept.
- Reproducibility: Whether another analyst can recompute the result from the same spec.
2. Turning Product Goals Into Objective Metrics
Interview prompts often start vague: "increase connection," "improve quality," "measure demand," or "make robots useful." Start by pinning down the decision and the behavior that would prove success.
Clarification questions to ask
What user problem are we solving?
Who is eligible for the feature?
What behavior would prove success?
Is the goal short-term engagement, long-term retention, safety, quality, or revenue?
What harms are unacceptable?
Do we need a causal decision or descriptive sizing?
What time horizon matters?
Are there network effects, novelty effects, or marketplace tradeoffs?
Worked example: Meta-style social feed feature
Prompt: "We are launching a feed ranking change to increase meaningful social interactions."
A weak answer: "Measure likes and comments."
A better answer challenges the proxy.
Reactions, comments, and shares do not all mean the same thing. Better signals might include reciprocal replies, comment thread depth, close-friend interactions, conversations lasting multiple turns, and interactions not followed by hides, reports, or unfollows. Low-effort reactions, spammy comments, and engagement bait should be discounted or broken out.
| Metric type | Metric | Definition |
|---|---|---|
| Primary | Meaningful friend interactions per eligible user-day | Valid friend-content actions per daily active feed user |
| Secondary | Sessions with meaningful friend interaction | Feed sessions with at least one valid friend interaction / feed sessions |
| Input | Friend-authored impressions per user | Exposure to friend content |
| Input | Average friend-content rank position | Whether ranking increased friend visibility |
| Quality guardrail | Hides/reports per 1,000 impressions | Example block threshold: increase above +0.2 per 1,000 with CI excluding 0 |
| Ecosystem guardrail | Non-friend creator impressions per user | Example non-inferiority limit: no worse than -3% |
| Integrity guardrail | Flagged-account action share | Spam actions / total actions |
The denominator can flip the story. "Interactions per friend impression" can increase while total interactions per user drops if friend exposure drops. The reverse can also happen: increasing friend exposure may lower the per-impression rate while raising per-user interactions. For a launch decision, use a user-level primary metric tied to product value, then use exposure-normalized rates for diagnosis.
Experiment length depends on power, minimum detectable effect, metric latency, seasonality, novelty, network effects, and rare guardrails. "One week" is not a rule. A solid interview answer: run until the precomputed sample size is reached, include complete weekday/weekend cycles, wait for late events, and extend if guardrails are rare or novelty is likely.
If the rollout needs a holdout, say exactly what it is. For example, keep 2% of eligible users on control ranking for four to eight weeks, randomized by viewer or network cluster, monitor retention, hides, creator distribution, and interaction quality, and launch only if effects stay stable and no guardrail crosses its threshold.
3. Designing Robust Metric Definitions: Numerators, Denominators, Windows, and Segments
A metric spec should leave very little room for interpretation.
Metric name:
Product decision:
Construct measured:
Unit of analysis:
Numerator:
Denominator:
Attribution window:
Timezone:
Event names and versions:
Logging surface:
Eligibility:
Inclusion/exclusion rules:
Bot/spam filters:
Identity resolution:
Deduplication rule:
Late-arriving data policy:
Freshness SLA:
Privacy constraints:
Aggregation method:
Segments:
Guardrails:
Pre-registration status:
Known limitations:
Owner/source tables:
Friend interaction metric spec
Metric: Meaningful friend interactions per eligible user-day
Unit: user-day
Primary aggregation: mean of user-day interaction counts
Numerator: valid reactions + comments + shares on attributed friend-authored impressions
Denominator: eligible user-days, not impressions
Attribution: action within 30 minutes after impression_id exposure
Timezone: user local date, fallback UTC
Event versions: feed_impression v3+, feed_action v2+
Deduplication: one canonical row per event_id/idempotency_key
Filters: exclude bots, spam accounts, test users, invalid author_type
Late data: freeze after 48 hours; report late-event sensitivity
This is a count per user-day, so it can exceed 1. If you also report actions per impression, call it an action rate, not a percentage. One impression can have multiple actions.
SQL-style metric computation
WITH impressions AS (
SELECT *
FROM feed_impressions
WHERE event_date BETWEEN DATE '2026-01-01' AND DATE '2026-01-14'
AND event_version >= 3
AND surface = 'home_feed'
AND post_author_type = 'friend'
AND is_bot = FALSE
),
canonical_impressions AS (
SELECT *
FROM (
SELECT
i.*,
ROW_NUMBER() OVER (
PARTITION BY impression_id
ORDER BY server_received_at ASC, ingestion_id ASC
) AS rn
FROM impressions i
)
WHERE rn = 1
),
actions AS (
SELECT *
FROM feed_actions
WHERE event_date BETWEEN DATE '2026-01-01' AND DATE '2026-01-15'
AND event_version >= 2
AND action_type IN ('reaction', 'comment', 'share')
AND is_spam_action = FALSE
),
canonical_actions AS (
SELECT *
FROM (
SELECT
a.*,
ROW_NUMBER() OVER (
PARTITION BY COALESCE(event_id, idempotency_key)
ORDER BY server_received_at ASC, ingestion_id ASC
) AS rn
FROM actions a
WHERE COALESCE(event_id, idempotency_key) IS NOT NULL
)
WHERE rn = 1
),
attributed AS (
SELECT
i.experiment_group,
i.user_id,
i.user_local_date,
COUNT(a.event_id) AS friend_actions
FROM canonical_impressions i
LEFT JOIN canonical_actions a
ON a.user_id = i.user_id
AND a.impression_id = i.impression_id
AND a.client_ts >= i.client_ts
AND a.client_ts < i.client_ts + INTERVAL '30' MINUTE
GROUP BY 1, 2, 3
)
SELECT
experiment_group,
AVG(friend_actions) AS mean_friend_actions_per_user_day
FROM attributed
GROUP BY 1;
Robot-fleet metric caveat
"Customer-defined acceptance criteria" is not objective unless it is operationalized. Use definitions like these instead:
Requested tasks: all customer/system task requests
Accepted tasks: requests the robot accepted as eligible
Attempted tasks: accepted tasks with autonomous start signal
Completed tasks: barcode scanned, item placed in bin, image audit passed, and force threshold not exceeded
Refusal rate: rejected requests / requested tasks
Deferral rate: accepted but not attempted / accepted tasks
Autonomous completion rate: completed without teleoperation / attempted tasks
This prevents the system from looking good by excluding hard tasks before assignment.
4. Validating Whether Your Data Is Actually Objective
The thresholds below are interview examples, not universal constants. In production, they should be agreed on before the readout based on metric sensitivity.
| Check | Example decision logic |
|---|---|
| Assignment SRM | Block readout if allocation p-value < 0.001 in assignment population |
| Exposure imbalance | Investigate if treatment/control exposure differs unexpectedly by platform or segment |
| Key missingness | Block if user_id/impression_id missing >0.1% or differs by variant >0.02 percentage points |
| Duplicates | Investigate if duplicate rate >0.5%, differs by variant >0.1 pp, or moves primary metric by >25% of MDE |
| Event volume | Investigate unexplained day/platform shifts >5% |
| Late events | Freeze after SLA; run sensitivity for late arrivals |
| Bot/spam | Report metrics with and without filters; block if conclusion depends on filter choice |
SRM: assignment vs exposure population
WITH assigned AS (
SELECT user_id, experiment_group
FROM experiment_assignments
WHERE experiment_id = 'feed_friend_rank_v1'
),
counts AS (
SELECT experiment_group, COUNT(*) AS observed
FROM assigned
GROUP BY 1
),
tot AS (SELECT SUM(observed) AS n FROM counts),
expected AS (
SELECT 'control' AS experiment_group, 0.50 AS alloc UNION ALL
SELECT 'treatment', 0.50
)
SELECT
c.experiment_group,
c.observed,
t.n * e.alloc AS expected,
POWER(c.observed - t.n * e.alloc, 2) / (t.n * e.alloc) AS chi_component
FROM counts c
JOIN expected e USING (experiment_group)
CROSS JOIN tot t;
Sum chi_component to get χ². For two cells, χ² above 10.83 implies p < 0.001. Repeat by platform, geography, and app version. Also check the exposed analysis population separately; exposure imbalance may be product behavior, while assignment SRM is experiment failure.
Duplicate handling
Do not deduplicate by user_id + event_type + object_id + timestamp bucket unless it is a documented fallback. That approach can merge legitimate repeated actions and still miss retries. Prefer event IDs, idempotency keys, request IDs, ingestion IDs, and event-version-aware contracts.
SELECT
event_date,
platform,
experiment_group,
COUNT(*) AS rows,
COUNT(DISTINCT COALESCE(event_id, idempotency_key)) AS unique_events,
1.0 - COUNT(DISTINCT COALESCE(event_id, idempotency_key)) / COUNT(*) AS duplicate_rate
FROM feed_actions
WHERE event_date BETWEEN DATE '2026-01-01' AND DATE '2026-01-14'
GROUP BY 1, 2, 3;
Canonical-row rule: keep the earliest valid server-received row per stable event key, preserve legitimate repeat behavior with distinct event IDs, and compare metrics before and after deduplication.
Bot/spam example
SELECT
experiment_group,
COUNT(*) AS actions,
SUM(CASE WHEN account_age_days < 1
OR actions_per_minute > 30
OR spam_model_score >= 0.95
THEN 1 ELSE 0 END) AS flagged_actions
FROM feed_actions
GROUP BY 1;
Also check clock drift, timezone conversion, backfills, schema changes, client/server reconciliation, event firing QA, assignment stickiness, mutually exclusive variants, and exposure logging.
5. Objective Data From Non-Table Sources
Messy sources can produce objective variables, but only after the coding rules are defined and validated.
Example: demand for "friend-only feed filters"
Sources:
Search terms: "friends only", "show friends", "hide suggested", "hide pages"
Regex example: \b(friends only|show (me )?friends|hide suggested|hide pages)\b
Support taxonomy: feed_control_request, unwanted_recommendations, friend_content_request
Manual review: 200 stratified examples by language/platform
False positives: "hide from friends", "friend request page"
Normalization: mentions per 10,000 active users, deduped by user-week
Validation methods:
- Stratified sampling across geography, language, platform, and activity level.
- Precision/recall checks against human review.
- Inter-rater agreement for labels.
- Weighting to known active-user composition when appropriate.
- Confidence intervals for mention rates.
- Sensitivity analysis using stricter and looser classifiers.
- Deduplicate repeated complaints from the same user.
Privacy needs to be explicit: use approved data sources, respect consent and product policy, minimize fields, aggregate where possible, restrict raw text/audio/video access, define retention, and avoid using private content beyond the approved purpose.
6. Objective Data vs. Causal Evidence
Objective measurement tells you what was recorded. Causal evidence tells you what changed because of an intervention.
A causal answer should specify:
Estimand: effect of ranking change on mean friend interactions per eligible user-day
Randomization unit: viewer user, or network cluster if interference is high
Analysis unit: user-day
Population: eligible logged-in feed users
Treatment: ranking boost increasing friend-authored exposure
Assignment: intent-to-treat primary; treatment-on-treated secondary
Exposure dilution: some assigned users may not open feed
Power/MDE: precompute sample size for primary and key guardrails
Uncertainty: effect size, standard error, confidence interval
Multiple testing: pre-register primary metric and guardrail hierarchy
Observational comparison
SELECT
post_author_type,
feed_position_bucket,
post_type,
COUNT(a.event_id) * 1.0 / COUNT(i.impression_id) AS actions_per_impression
FROM canonical_impressions i
LEFT JOIN canonical_actions a
ON a.impression_id = i.impression_id
AND a.client_ts BETWEEN i.client_ts AND i.client_ts + INTERVAL '30' MINUTE
GROUP BY 1, 2, 3;
This is descriptive. You can reduce confounding with regression controls, matching, propensity weighting, fixed effects, or within-user comparisons. Still state the limitation: unobserved confounding can remain.
A/B plan
Randomize eligible viewers, log assignment and exposure, ramp 1% → 5% → 20% only after logging and guardrails pass, and consider cluster randomization or network-aware analysis if treated viewers affect creators or control viewers. A holdout is not a separate causal method; it is a control group maintained during an experiment or rollout.
7. Choosing Metrics and Guardrails for Real Interview Scenarios
Case 1: feed launch recommendation
Illustrative result table:
| Metric | Effect | 95% CI | Decision rule |
|---|---|---|---|
| Friend interactions per user-day | +2.0% | +0.8% to +3.2% | Must exceed +1% practical threshold |
| Hides per 1,000 impressions | +0.3 | +0.1 to +0.5 | Must not exceed +0.2 |
| Non-friend creator impressions/user | -5.0% | -6.2% to -3.8% | Must not be worse than -3% |
| Duplicate rate difference | +0.03 pp | stable | Must be <0.1 pp |
Recommendation: do not fully launch. The primary metric is practically and statistically positive, but hides breach the guardrail and creator distribution fails the non-inferiority limit. I would tighten the ranking constraints, inspect harmed segments, and rerun the experiment or continue at limited ramp with rollback thresholds.
Formula language for interviews:
Effect = treatment mean - control mean
95% CI = effect ± 1.96 × standard error
Launch if primary clears practical threshold and guardrails satisfy non-inferiority rules.
Case 2: notification-triggered survey
Separate subjective sentiment from objective metadata.
Subjective: answer to "How helpful was this notification?" on a 1–5 Likert scale
Objective metadata: sampled user, notification sent/opened, response timestamp, completion, breakoff, opt-out
Question wording: "How helpful was this notification for what you were doing today?"
Sampling frame: random eligible notification recipients, not just clickers
Treatment of Likert: report distribution and top-2-box, not only mean
Bias checks: response rate, respondent vs nonrespondent behavior, incentives, mode effects
Use a factorial design:
| Cell | Notification | Survey |
|---|---|---|
| A | No | No |
| B | Yes | No |
| C | No | Yes |
| D | Yes | Yes |
This separates notification effects from survey measurement effects.
Case 3: robot fleet analytics
Operationalize safety metrics:
Near miss: human/robot distance < defined threshold with closing velocity above threshold
Restricted-zone entry: robot pose enters geofenced area for >1 second
Human intervention: teleoperation command or supervisor override
Useful autonomous work: completed accepted task with barcode/image/weight audit and no teleoperation
Stop rule: any severe safety incident pauses ramp pending failure review
Review rare severe events individually. Averages are not enough for safety-critical launches.
8. Handling Bias, Network Effects, and Novelty Effects
| Threat | How to detect or quantify | Mitigation and limitation |
|---|---|---|
| Selection bias | Compare eligible, exposed, and analyzed users | Randomization helps; reweighting cannot fix unobserved confounding and can increase variance |
| Nonresponse bias | Respondent vs nonrespondent differences | Weighting, follow-up samples, response-rate reporting |
| Survivorship/attrition | Users disappearing after treatment | Include churn/return guardrails |
| Instrumentation bias | Variant-specific logging differences | A/A tests, client/server reconciliation |
| Carryover/learning | Effects persist after exposure | Washout periods, switchback caution |
| Regression to mean | Extreme users normalize | Randomized controls |
| Peeking/sequential testing | Repeated looks inflate false positives | Preplanned looks or corrected sequential methods |
| Spillover contamination | Treated users share with controls | Cluster randomization |
| Marketplace/network effects | Creator supply changes affect all viewers | Creator-viewer network analysis |
| Novelty | Early lift fades | Longer holdout |
Simpson's paradox example:
| Platform | Control rate | Treatment rate |
|---|---|---|
| iOS | 10% | 11% |
| Android | 4% | 5% |
Treatment improves both segments. But if treatment has many more Android users, the overall treatment rate can look lower. Segment before launch.
9. How to Communicate Objective Data in an Interview Answer
Use a structure that matches the prompt.
Metrics design:
Goal → user behavior → primary metric → inputs → guardrails → limitations
Experiment readout:
Design → data quality → effect/CI → guardrails → segments → decision
Data-quality prompt:
Failure mode → detection query → threshold → metric impact → prevention
Causal-inference prompt:
Estimand → identification strategy → assumptions → analysis → sensitivity
Sample answer: duplicate-observation tool
"I would treat duplicate observations as an experiment-validity risk. I would first quantify duplicate rates using stable event IDs or idempotency keys, segmented by day, platform, app version, and experiment group. I would not use timestamp buckets except as a documented fallback because they can merge real repeated actions.
I would build the tool if duplicates exceed an agreed threshold, differ by variant, move a key metric by more than 25% of the MDE, change the launch decision, or make guardrails worse even if significance remains. The system should include data contracts requiring event IDs, streaming validation, warehouse tests, alert thresholds, dashboards, clear ownership, and a backfill strategy. The decision is based on practical effect size and guardrails, not only whether p-values change."
10. Decision Checklist: Is the Evidence Strong Enough to Launch?
Metric movement
- Primary metric is pre-registered.
- Effect clears practical threshold, not just positive direction.
- Confidence interval is narrow enough for a decision.
- Metric maps to the product goal, not only an easy proxy.
Statistical discipline
- MDE and sample size were planned before readout.
- Multiple comparisons and metric fishing are controlled.
- Sequential peeking is avoided or corrected.
- Primary, secondary, and exploratory metrics are labeled.
Guardrail health
- Guardrails have denominators and non-inferiority thresholds.
- Rare severe events are reviewed separately.
- Segment regressions are explained or mitigated.
Data quality
- Assignment SRM passes.
- Exposure logging is correct.
- Missingness, duplicates, bots, late events, and schema changes are acceptable and balanced.
Rollout plan
Do not say "launch to 25%" without rationale. Say:
Ramp 1% after logging QA.
Move to 5% if guardrails stay within limits for 24–48 hours.
Move to 20% after full metric-latency window.
Hold or rollback if any severe incident occurs, key guardrail breaches, or metric effect falls below threshold.
Maintain holdout if novelty, retention, or network effects matter.
FAQ
What is objective data?
Objective data is evidence based on observable, verifiable measurement, such as transaction counts, timestamps, lab values, sensor readings, event logs, or audited task outcomes.
What is subjective data?
Subjective data captures perception or judgment, such as satisfaction ratings, opinions, interview responses, app review text, or human labels. Structured subjective data is still subjective in content.
Can subjective sources produce objective metrics?
Yes, with caveats. For example, "number of reviews mentioning battery drain per week" is an objective count, but the complaint itself is subjective and the classifier must be validated.
What are examples of objective data in business, healthcare, and research?
Business: orders, refunds, delivery times. Healthcare: temperature, blood pressure, lab results. Research: task time, error count, randomized assignment, observed completion.
Can objective data be wrong?
Yes. Broken logging, biased sampling, bad sensors, bot traffic, flawed labels, or invalid proxies can make objective-looking data misleading.
Can objective data prove causation?
Not by itself. Causation requires a credible design such as randomization, a valid holdout/control group, or a justified quasi-experiment with clear assumptions.