DeepFinder Performance Benchmark

2. Headline charts

Interpretation: depth 1 and 2 always return all paths the user has. From depth 3 onward, most users have more paths than we currently surface — they just don't see them.

3. What we measured

For each query, we recorded:

Sample size: 500 measurements per depth (1, 2, 3, 4) per run = 2,000 queries per run × 4 runs = 8,000 baseline queries, plus 2,400 sweep queries. Random synthetic contact as target each time.

Plain-English glossary

• p50 / p95 / p99: Sort all 500 latency measurements smallest to largest. p50 is the middle one (the typical case). p95 is the 25th-from-worst (what 95% of users will experience or better). p99 is the 5th-from-worst (the slow tail).
• Hop: One step through a work_overlap edge. "Walked 3 hops" = "we crossed 3 shared-employer connections between user and target."
• Branching factor: How many edges leave each contact in the graph. Higher = more paths, more work for the recursive walk.
• Truncation: Returning fewer paths than the SQL actually found. Caused by the public LIMIT=50.

4. The graph we tested against

We didn't have prod data on staging, so we generated a synthetic graph designed to mirror the largest real customer network: Inovia Capital (collection #1201).

How we sized it

Single read-only query on production aggregate stats:

SELECT
  COUNT(DISTINCT cwe.organization_id) AS distinct_orgs,
  COUNT(*)                            AS total_cwes,
  COUNT(DISTINCT cwe.contact_id)      AS distinct_contacts
FROM user_contact_collections ucc
JOIN contact_work_experiences cwe ON cwe.contact_id = ucc.contact_id
WHERE ucc.collection_id = 1201
  AND cwe.organization_id IS NOT NULL;

Topology generator

Distribution we got

5. Four independent runs

We ran the same benchmark four times to confirm reproducibility.

Run 1 (100 iterations × 4 depths)

Run 2 (500 iterations)

Run 3 (500 iterations)

Run 4 (500 iterations — buffer pool fully warm)

Interpretation

• Depths 1-2 are rock-solid stable — sub-millisecond variance across runs. Essentially "free" queries.
• Depth 3 has higher variance at p95/p99 — Run 3 saw p95=44.5 vs Run 2's 14.4 (3× spread). This is real: the cost depends on which random target was picked. A target with rich work history explores a much bigger frontier.
• Depth 4 is consistently around p95 70 ms, p99 100-120 ms. The truncation cap (50) actually stabilizes the upper tail because once we hit it the walk stops.
• Truncation rates are stable across runs: ~70% at depth 3, ~94% at depth 4.
• Run 4 was noticeably faster — buffer pool was fully warm by then (cumulative cache from runs 1-3).

The variance at depth 3 is a feature, not a bug — it's telling us "some users have richer networks than others, and DeepFinder's cost reflects that."

6. What this means in product terms

"Should we raise the depth cap from 3 to 4?"

Performance says yes — depth 4 is fast enough.

• Depth 4 p95: ~70 ms (margin to 250 ms cap: 3.5× slack)
• Depth 4 p99: ~100-120 ms (margin to 250 ms cap: 2× slack)
• No timeouts observed across 1,500 depth-4 queries

The decision is purely about whether 4-hop connections feel meaningful to users — not about whether the system can compute them.

"Are we losing user value because of the 50-path limit?"

Yes — at depth 3+, ~70-94% of the time. When a user hits this endpoint and we tell them "you have 50 paths to this person," in a network like Inovia they probably have many more we just clipped. Increasing the limit is safe from a perf standpoint at depth 3, but adds latency at depth 4 (see the limit sweep below).

Are the cap defaults right?

7. Parameter sweeps

MAX_EDGES_PER_HOP sweep (depth 3, 200 iterations each)

MAX_EDGES_PER_HOP bounds how many overlap edges DeepFinder follows from each contact during the recursive walk. Too low → miss real paths. Too high → frontier explodes.

What this tells us

1. The cap of 25 is not the binding constraint at all. Mean paths only nudges from 39 → 43 across all cap values — the 50-path public LIMIT clips long before the frontier cap kicks in.
2. Even MAX_EDGES_PER_HOP=10 returns ~95% as many paths as the default. If you wanted to cut perf cost, dropping to 10 is essentially free in user-visible terms.
3. The cap exists for adversarial cases, not the average case. A hyper-connected hub contact (think a hiring-manager with hundreds of overlaps) would push the recursive frontier to thousands without the cap. Our synthetic doesn't generate those, so we don't see the cap save us — but it's correct insurance.

Recommendation: keep at 25. Don't tighten (small risk of cutting real edges from hub contacts), don't loosen (no benefit because of LIMIT).

limit sweep (depths 3 and 4, 200 iterations each)

DEFAULT_LIMIT clips paths returned to the API caller. We tested 50, 100, 200, 500.

What this tells us

1. At depth 3, raising limit is essentially free. p95 stays around 12-15 ms across all values. Mean paths jumps 43 → 99 (limit 50 → 500) and truncation drops 67% → 2%. You can comfortably triple the limit at depth 3 with zero perf cost.
2. At depth 4, the limit is the perf knob. Higher limit = more rows to sort and serialize. limit=500 hits p95 312 ms — over the 250 ms budget. limit=200 lands at 201 ms (knife edge).
3. Diminishing returns are clear. limit 50 → 200 doubles paths returned; 200 → 500 only adds 16% more. Most users have under 100 useful paths even at depth 3.
4. A depth-aware limit could give the best of both worlds:
   • depth 1-3: limit=200 (truncation 12%, p95 still ~15 ms)
   • depth 4: limit=100 (truncation 90%, p95 ~115 ms — well under budget)

Recommendation: consider raising depth-3 limit to 200. Hold depth-4 limit at 50-100 if 4-hop ever ships.

8. Hypothesis verdicts

9. How to reproduce

One-command setup:

cd ~/Desktop/projects/getro
make up                              # Brings up dev env

# Seed (~5 min)
docker exec getro_backend bin/rails runner /tmp/perf_seed_500k_inovia.rb

# Backfill (~40 min — the longest step)
docker exec getro_backend bin/rails runner /tmp/perf_backfill_synthetic_only.rb

# Run benchmark (~7 min for 500 iter × 4 depths)
docker exec getro_backend bundle exec rake 'perf:deep_finder_load_test[500,4]'

The rake task accepts:

perf:deep_finder_load_test[iterations, max_depth, collection_id]

• iterations — queries per depth (default 100)
• max_depth — 1..4. Setting 4 lifts the production hard cap of 3 just for the test (restored on exit).
• collection_id — optional. If set, attaches synthetic contacts as shared UCC rows on an existing collection (e.g. 11 for the dev "GetroJobs staging environment"). If omitted, creates a synthetic collection.

Cleanup

All synthetic rows are tagged with prefixes (perf-synth- for contacts, PERF_SYNTH_ for orgs). Cleanup is a single multi-statement DELETE.

10. Impact: before vs after the cap changes

We shipped two changes informed by this benchmark. This section quantifies what users actually get from each.

Change A — DEFAULT_LIMIT 50 → 100

At depth 3, doubling the default limit nearly doubled the number of paths users see, and cut the truncation rate almost in half — with no measurable latency impact.

In plain English: before, two-thirds of users at depth 3 had paths we silently dropped. After, only about a third do — and the user still gets twice as many paths visibly. At depth 4, the change costs ~40 ms of latency at p95 (75.9 → 114.2 ms) — still well under the 250 ms budget — in exchange for nearly doubling visible paths.

Change B — MAX_DEPTH_HARD_CAP 3 → 4

Before this change, an API caller passing ?max_depth=4 got silently clamped to 3. After, callers can opt in to 4-hop searches when they want deeper reach.

In plain English: users who explicitly want "show me anyone within 4 hops" now get that — at a slight latency cost (114 ms vs 14 ms for 3-hop) but well within the timeout. Default behavior is unchanged, so existing integrations see zero impact.

Change C — MAX_LIMIT clamp at 500 (server-side guard)

A safety net we added because the API previously accepted any positive limit. A caller passing limit=10000 could force expensive sorting and risk hitting the SQL statement_timeout.

No user-visible change in normal cases — only blocks abusive / accidentally-huge requests.

11. Open follow-ups

• ☐ Add walk_rows instrumentation (EXPLAIN ANALYZE per query) to confirm H4 directly.
• ☐ Test with concurrent load (e.g. 50 parallel DeepFinder calls) to verify perf under contention.
• ☐ Run the same benchmark on staging once we mirror real Inovia data (Privacy/T015 review pending).
• ☑ Decide on LIMIT=50 increase — shipped (commit 29b04563: DEFAULT_LIMIT → 100, MAX_LIMIT clamp at 500).
• ☑ Decide whether to lift MAX_DEPTH_HARD_CAP — shipped (commit 29b04563: lifted from 3 to 4).

Companion docs: spec.md · performance.md · index.html · raw CSVs in backend/out/perf/