# Scraper-CI: Building a Web Data Reliability & Crawl-Governance Control Plane With Bright Data ## The problem: a successful scraper can still be wrong A scraper returning HTTP 200 is not necessarily healthy. It can return thousands of records, finish without an exception, and still be producing bad data — a website changes its structure, a field disappears, pagination behaves differently, a page starts returning a different template, a required field goes empty. The scraper keeps running the whole time. From the infrastructure's perspective, everything looks fine: ```text HTTP 200 ↓ Records returned ↓ Job completed ``` From the application's perspective: ```text The data is wrong. ``` That gap is what I wanted to address with **Scraper-CI**: a web-data reliability and recovery control plane with Bright Data at its center. The goal was not to build another scraper. The goal was to build the layer around the scraper that can answer: * What does this source look like? * What should actually be crawled? * What should not be crawled? * Which acquisition capability makes sense? * Did the extraction still satisfy its contract? * If the data degraded, why? * Can the system recover? * Did the recovery actually fix the problem? * Can downstream applications consume the resulting data without knowing how it was acquired? The resulting lifecycle is: ```text PROFILE ↓ POLICY ↓ ROUTE ↓ ACQUIRE ↓ VALIDATE ↓ DIAGNOSE ↓ HEAL ↓ VERIFY ↓ CONSUME ``` This article explains how I built that system, why I designed the boundaries this way, and why treating "a scraper ran successfully" and "the scraper's data is trustworthy" as two separate questions turned out to be the central design decision. Consider a scraper that historically produces 100 records at 98% required-field completeness — healthy by any reasonable measure. Then the target website changes its layout, and the same scraper still returns 100 records, still returns HTTP 200, but now sits at 42% completeness. Nothing crashed. The record count didn't even move. But the dataset is clearly degraded, and no infrastructure-level signal would have caught it. **🔗 Scraper-CI GitHub Repository:** ([https://github.com/Jaival-Suthar/sci](https://github.com/Jaival-Suthar/sci)) That led to the central design principle behind Scraper-CI: > **A scraper should not be considered reliable merely because it runs. It should be considered reliable when the data it produces continues to satisfy the contract it was built to provide.** Scraper-CI therefore treats acquisition success and data correctness as separate concerns: ```text Acquisition succeeded ≠ Extraction is trustworthy ``` That distinction drives the rest of the architecture. * * * ## Bright Data is the acquisition engine at the heart of the system The first architectural decision was to **not rebuild the acquisition layer**. Bright Data already provides sophisticated web-acquisition infrastructure — Scraper Studio handles navigation, pagination, sitemap loading, browser/code execution, and structured extraction, along with its own self-healing capabilities. That's valuable, mature infrastructure, and Scraper-CI is built to sit directly on top of it rather than compete with it. Bright Data isn't a bolted-on dependency here — it's the acquisition core the rest of the system is designed around. ![](https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/c6ad15e5-fabd-48b6-8ba2-925e7c858511.png align="center") The boundary is deliberate: ```text Bright Data = acquisition (the core) Scraper-CI = intelligence + control + reliability + recovery Consumer = domain-specific interpretation ``` This separation became the foundation for everything else. * * * ## Source Intelligence: Profiling, Policy, and Scope ### Profile the source before deciding how to crawl it A website isn't simply a graph of URLs and links pointing to more URLs and links — it carries signals that help determine how it should be approached in the first place. Scraper-CI introduced a **source profiling layer** for exactly this reason. Given a target URL, the profiler inspects signals such as page type, platform, JavaScript dependency, internal and external links, canonical relationships, pagination, sitemap availability, repeated-content patterns, and domain/page relationships. ![](https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/8bab2d1e-be68-4ff0-800b-dc900a91183f.png align="center") The profile isn't a report for its own sake — it becomes input to the next question: **what should this acquisition be allowed to do?** ### Crawl policy: turning source understanding into boundaries A scraper can have powerful navigation primitives and still lack an explicit statement of intent. I wanted Scraper-CI to make crawl scope visible and bounded, so the source profile feeds into an inferred **crawl policy**: ```text Internal links ✓ Canonical links ✓ Pagination ✓ Sitemap ✓ Unrelated external ✕ ``` The policy also expresses bounded constraints — depth ≤ 2, pages ≤ 50, no external domains, that kind of thing. The important distinction is that the crawler *has* capabilities, while the policy defines the intended boundary of those capabilities. That's why I think of this as **crawl governance** rather than just another crawler implementation. ### Sitemap support: primitive versus abstraction There's an important distinction worth being precise about here. Bright Data already supports sitemap loading in Scraper Studio, so I wouldn't claim Bright Data lacks sitemap support — that would simply be wrong. Bright Data provides the primitive. Scraper-CI adds a different abstraction around it: sitemap discovery and sitemap-derived crawl scope become part of source intelligence and crawl-policy inference. ```text Bright Data ↓ Sitemap-loading capability Scraper-CI ↓ Detect sitemap-related source signals ↓ Use them as crawl intelligence ↓ Build bounded crawl intent ↓ Pass that intent into acquisition ``` The contribution isn't inventing sitemap support — it's the **control-plane abstraction around the primitive**. ### The policy participates in execution The inferred crawl policy isn't merely stored as metadata — it influences the acquisition request itself. ```text User extraction intent + Source-derived crawl intelligence ↓ Acquisition description ↓ Bright Data collector ``` The acquisition intent can carry constraints like depth ≤ 2, pages ≤ 50, internal = 1, canonical = 1, pagination = 1, sitemap = 1, external = 0. This creates two layers of protection: first, the acquisition system receives the intended crawl constraints up front (**acquisition guidance**); second, Scraper-CI independently evaluates the resulting acquisition against the expected state afterward (**independent validation**). The policy isn't just documentation — it participates in execution and is subsequently checked. * * * ## Acquisition Routing ### From profiling to acquisition routing Once the source has been profiled and the crawl policy inferred, the next question is which acquisition capability should handle this source. This became the **Capability Router**. Instead of assuming every URL gets the same treatment, the system evaluates source signals and produces an acquisition decision. ```text Source Profile │ ▼ Capability Router │ ├── primary capability ├── fallback capabilities ├── evidence └── confidence ``` A routing decision might look conceptually like: source signals → recommended capability → confidence: 78% → acquisition. That confidence score is deliberately not presented as a guarantee — it means the router found stronger evidence for this strategy, not that the scraper is guaranteed to succeed. Reliability is measured from actual extraction results, not from routing confidence. ### Capability abstraction Scraper-CI uses an abstraction around acquisition capabilities rather than hard-coding the rest of the control plane to one implementation. The current capability model includes pipeline, unlocker, browser, scraper\_studio, and search. Some are concrete execution paths, others are extensible capability contracts — that distinction is intentional, since I didn't want the architecture to pretend an executor exists when it doesn't. ```text Capability contract ↓ Concrete adapter ↓ Acquisition provider ``` This means additional acquisition mechanisms can be introduced later without rewriting source profiling, crawl policy, reliability, diagnosis, recovery, verification, or downstream consumers. The result: **acquisition can evolve without becoming the architecture.** * * * ## Reliability, Diagnosis, and Recovery ### Validate the data after acquisition Once Bright Data returns structured data, Scraper-CI asks a different question: is the result still trustworthy? A simplified pipeline runs schema validation, required-field validation, completeness, uniqueness, record-count drift, record comparison, historical drift, and health scoring — landing on HEALTHY, DEGRADED, or INVALID. The system never treats "records exist" as proof of correctness; it evaluates the records themselves against schema validity, required-field completeness, uniqueness, record-count drift, record-level differences, and historical reliability. The goal throughout is detecting **silent degradation** — the kind of failure that never throws an exception. ### Diagnosis: turning failure into an explanation A health score alone isn't enough. If the system reports "Health: 54, Status: DEGRADED," the next question is why. Scraper-CI's diagnosis layer classifies problems: missing required fields point to extraction degradation, schema mismatch points to contract violation, blocked acquisition points to an acquisition problem, high JavaScript dependency points to an execution strategy problem. This creates a clean separation — validation answers *what* is wrong, diagnosis answers *why* it's wrong, and the recovery planner answers *what to try next*. Diagnosis is persisted with the run and becomes input to recovery planning. ### Diagnosis-driven recovery A common automation pattern is to retry exactly the same thing after a failure, but that's not always useful. If the problem is a missing field, repeating the same acquisition may just reproduce the same missing field. If acquisition is blocked, the system may need a different capability entirely. If the page is JavaScript-heavy, the execution strategy needs to change. Scraper-CI makes recovery diagnosis-driven: ```text Diagnosis │ ├── Missing field → Self-healing ├── Schema failure → Self-healing ├── Blocked acquisition → Unlocker strategy ├── JS dependency → Browser strategy └── Unknown → Fallback strategy ``` The recovery planner is deterministic and independently testable. ### Automatic healing is not the end of the workflow When an acquisition finishes in a degraded or invalid state, Scraper-CI can enter an automatic recovery path. The current workflow allows up to two automatic healing attempts per incident: ```text Run → DEGRADED/INVALID → Healing attempt 1 → Re-run → Verify → still degraded? → Healing attempt 2 → Re-run → Verify ``` The important part is what happens *after* healing. A successful repair operation doesn't automatically mean the data is correct — Scraper-CI runs the extraction again and independently evaluates the result. The loop is observe → diagnose → heal → re-run → measure. ### Constrained recovery I didn't want recovery to become "something is broken, fix it." Instead, Scraper-CI generates a diagnosis-derived repair instruction, carrying target information, observed degradation reasons, current field completeness, the extraction contract, preservation rules, and scope constraints. The repair instructions emphasize preserving existing fields and meanings, not expanding crawl scope, not introducing unrelated navigation, and making the smallest extraction change necessary. The goal is to restore the extraction contract with the smallest reasonable change — a repair that fixes one field by dramatically changing what the scraper crawls isn't actually a good repair. ### Automatic healing versus manual correction There's an important distinction between failure recovery and intentional contract evolution. **Automatic healing** applies when something degraded and the system should restore the existing extraction contract. **Manual correction** applies when the extraction still works but the desired contract has intentionally changed — for example, adding an author field while preserving the existing extraction. That's not a failure; it's an intentional requirement change. Keeping these separate prevents every change in requirements from being treated as scraper failure. ### Verification is a separate phase One of the easiest mistakes in a self-healing system is treating the repair API response as proof that the repair worked. I deliberately avoided that — recovery is incomplete until the resulting extraction has been independently evaluated. ```text Existing state → Repair → New acquisition → Independent validation → Before/after comparison → Verification result ``` Verification evaluates schema pass, completeness pass, health improvement, regression state, record-level differences, recovery delta, and diagnosis state, landing on either REPAIR VERIFIED or REPAIR REJECTED. This distinction is fundamental: **a repair action is not the same thing as a repaired dataset.** * * * ## The Operator Control Plane ### A UI over the same domain logic All of this eventually needed a UI. The Scraper-CI dashboard is an operator control plane over the backend orchestration, with main actions covering Automated, Inspect, Profile, Status, Reliability, Diagnose, Correct data, Verify, and Open URL. These aren't independent frontend implementations — the frontend calls the backend API, which invokes the same domain and orchestration logic used by the CLI. The UI is a control surface, not the system itself. ![](https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/02fdf390-471e-443a-a50f-85e8d7d2734b.png align="center") ### Inspect and Profile answer different questions **Inspect** answers which acquisition capability to use — it exposes source intelligence, recommended capability, confidence, supporting evidence, and fallback capabilities. This is the routing view. **Profile** answers what the source looks like and what crawl boundaries can be inferred — it exposes source signals, relationships, sitemap information, pagination, internal/external structure, and the inferred crawl policy. This is the source-intelligence view. Keeping these separate makes the architecture understandable to an operator. ### Reliability history A single run tells you whether something worked today. It doesn't tell you whether the scraper is reliable. Scraper-CI therefore persists historical reliability state — run health, health score, record count, validation state, degradation, recovery attempts, verification state — turning reliability into a historical property rather than a single status field. That lets "did the latest run pass?" and "has this target been stable over time?" be answered separately. * * * ## Flight Intelligence: A Real Downstream Consumer ### The architecture To prove Scraper-CI wasn't just an abstract control plane, I built a concrete downstream application: **Flight Intelligence**. ```text Public aviation source → Bright Data → Scraper-CI → Validation → Trusted structured data → Flight Intelligence ``` Flight Intelligence doesn't need to understand the acquisition mechanics — how Bright Data collected the records, which capability was selected, how the collector was created, how crawl policy was inferred, how diagnosis works, or how healing works. It just consumes structured data. That separation was intentional. ### A consumer, not part of the control plane Flight Intelligence is domain-specific — it understands aircraft. Scraper-CI doesn't, and shouldn't. ```text Scraper-CI │ structured snapshot ▼ Flight Intelligence ├── aircraft counts ├── altitude intelligence ├── speed intelligence ├── aircraft identity └── historical activity ``` The consumer derives observations like aircraft count, airborne versus ground activity, average and maximum altitude and speed, altitude bands, high-altitude traffic, new and disappeared aircraft, cumulative distinct aircraft, and historical activity — none of which needs to leak back into Scraper-CI. That's what makes the consumer replaceable. ### Why this architecture matters The larger idea isn't "I built a flight dashboard" — it's that I built a reusable reliability layer that can feed different applications. Today it's Scraper-CI feeding Flight Intelligence. Tomorrow it could just as easily feed monitoring, analytics, research, alerting, RAG, agent workflows, or any other domain-specific application, without rebuilding the acquisition and reliability infrastructure each time. * * * ## Production and Testing ### Production architecture The deployed system is split into independent services: ![](https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/211dd781-8f59-41ad-9479-d7f0cc8f38bc.png align="center") The backend owns the Bright Data credential; the frontend never receives the API key. Persistent control-plane state lives in PostgreSQL. Flight Intelligence's production collection is scheduled separately through GitHub Actions, running every two hours (`0 */2 * * *`) — so the consumer isn't backed by a static demo dataset, but by a live, recurring pipeline: scheduled workflow → scrape-ci run → Bright Data → Scraper-CI → PostgreSQL → Flight Intelligence. ### Reproducibility and deterministic testing Web acquisition has a hard testing problem: the web itself is nondeterministic. A live website can change between two test runs, making it difficult to tell whether a failure came from the system or from the external source changing underneath it. Scraper-CI includes deterministic tests and benchmark scenarios covering acquisition behavior, API contracts, health scoring, completeness degradation, schema validation, capability routing, crawl policy inference, relationship classification, diagnosis, drift, recovery planning, recovery guards, reliability history, and repair verification — the goal being to keep the intelligence and reliability layers reproducible even when the external web isn't. * * * ## Lessons and Honest Boundaries ### What I learned **1) HTTP success is a weak signal.** An HTTP response tells you a request succeeded, not that the extracted dataset is correct — and that single fact changes the architecture considerably. **2) Crawl scope should be explicit.** Once a system starts following links, "what does this scraper crawl?" becomes an architectural question. Encoding scope as a policy makes that intent visible, testable, and enforceable. **3) Don't confuse primitives with policy.** Bright Data already provides powerful crawling and acquisition primitives. The interesting problem is deciding which primitives to use, how to bound them, what evidence supports the decision, and whether the resulting acquisition satisfied the intended contract. **4) Recovery without verification is dangerous.** A healing operation can report success while the resulting data remains wrong. Heal ≠ verify — recovery must end with independent verification. **5) Keep domain intelligence downstream.** Aircraft-specific logic doesn't belong in a generic scraping reliability layer, and neither does ecommerce, jobs, news, or finance logic. Scraper-CI produces trusted structured snapshots; consumers interpret them. That boundary is what makes the system reusable. ### What Scraper-CI does not claim Worth stating explicitly: Scraper-CI does **not** claim to have invented web scraping, sitemap parsing, pagination, browser automation, anti-bot infrastructure, proxy infrastructure, scraper self-healing, or structured extraction. Bright Data already provides significant, mature infrastructure in all of these areas — it's the acquisition core this entire system is built to sit on top of. The architectural contribution is the layer around it: source profiling, crawl policy, acquisition routing, reliability evaluation, diagnosis, recovery planning, and independent verification. The project is stronger for being honest about that boundary. * * * ## Putting It All Together ### The complete architecture ![](https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/52cf6243-73f5-452e-91c2-bdf5fc70efef.png align="center") This is the system I ended up building. ### The bigger idea The web is not a static API. Selectors change, schemas change, page relationships change, navigation changes, content changes. And the most dangerous failures are often not crashes — they're successful executions that produce increasingly incorrect data. That's why web acquisition needs a control plane that treats reliability as a lifecycle rather than a boolean: understand → bound → route → acquire → validate → diagnose → recover → verify → consume. Bright Data is the acquisition core at the heart of this system. Scraper-CI provides the intelligence, control, reliability, and recovery around it. Flight Intelligence demonstrates what can be built on top. That separation is ultimately the point. > **A scraper should not be considered reliable merely because it runs. It should be considered reliable when the data it produces continues to satisfy the contract it was built to provide.** * * * ## Project The complete project, implementation, deployment configuration, benchmarks, tests, and local setup are available in the [Scraper-CI GitHub repository](https://github.com/Jaival-Suthar/sci). ### 🎬 Watch the demos ▶ [**3-minute Scraper-CI demo on YouTube**](https://youtu.be/wOKMk2Btyd8) ▶ [**7-minute technical deep dive on YouTube**](https://youtu.be/7tQj2uWBRwM) ### 🚀 Live applications * [**Scraper-CI Control Plane**](https://scraper-ci-ui.onrender.com/) * [**Flight Intelligence**](https://flight-intelligence-rj3l.onrender.com/) ### 🔧 Backend * [**API**](https://sci-av37.onrender.com) * [**Health Check**](https://sci-av37.onrender.com/api/health) * [**OpenAPI / Swagger**](https://sci-av37.onrender.com/docs) > The two application links above are the primary entry points for the project. The backend is the shared control-plane API behind the Scraper-CI UI and the downstream consumer. If a Render service is sleeping or temporarily unavailable, a live URL may briefly return an availability error until the service wakes. * * * ## AI / coding-assistant disclosure This project was developed with assistance from **ChatGPT and OpenAI Codex**. AI assistance was used for coding support, implementation iteration, debugging, and development assistance. The system architecture, source-profiling model, crawl-policy design, acquisition abstraction, reliability model, recovery logic, verification lifecycle, downstream consumer architecture, and infrastructure decisions were designed, reviewed, integrated, and verified as part of the project development process. AI-assisted code was treated as implementation assistance rather than as an authority over the system design. * * * # Final takeaway Scraper-CI started with a simple observation: a scraper can succeed technically while failing silently in the data it produces. The solution wasn't to build another scraper. It was to build the layer that makes acquisition **observable, bounded, recoverable, and trustworthy**. That is the idea behind Scraper-CI.