NYC DOB Contact Scraper

worker online
Completed

Unique records

10,894

122 pages processed

Materialized

12,200

Page$1221 · records 12201–12300

Duplicates skipped

1,306

deduped on record_key

Crawl speed

497.7/min

last 10 productive blocks

Last request

28.40 MB

ViewState

28.04 MB

Last pager

Page$1211

Last block

25m 59s ago

Last success

Heartbeat

3s ago

Block throughput
last 12 blocks
new records duplicatesoldest → latest
PagerRangeNewDupmsWhen
Page$121112,10112,110 / 12,200+8911880925m 59s ago
Page$120112,00112,010 / 12,100+8416900126m 13s ago
Page$119111,90111,910 / 12,000+8515881826m 27s ago
Page$118111,80111,810 / 11,900+919852626m 40s ago
Page$117111,70111,710 / 11,800+8317860326m 53s ago
Page$116111,60111,610 / 11,700+8713817827m 6s ago
Page$115111,50111,510 / 11,600+8515897827m 19s ago
Page$114111,40111,410 / 11,500+8812866727m 33s ago
Page$113111,30111,310 / 11,400+8911918727m 46s ago
Page$112111,20111,210 / 11,300+8614843228m 0s ago
Page$111111,10111,110 / 11,200+9010840628m 13s ago
Page$110111,00111,010 / 11,100+9010845928m 26s ago
1m 39s ago
lifecycle
worker boot (status=completed)
11m 23s ago
lifecycle
worker boot (status=completed)
25m 52s ago
lifecycle
stopped: server rejected request size; all prior records are saved
25m 52s ago
error
body_limit: HTTP 413, request=28.64 MB
43m 30s ago
command
resume: crawl resumed from checkpoint (materialized=0)
44m 16s ago
lifecycle
worker boot (status=idle)
53m 7s ago
command
reset_queue: queue reset: crawl state cleared, contacts preserved
53m 7s ago
lifecycle
queue reset: crawl state cleared (contacts preserved)
Scraped contacts
10,894 records

PostgreSQL is the source of truth · dashboard polls every 3s · a810-lmpaca.nyc.gov