Connectors and enrichment

The platform takes data in two ways. MI ingestion pulls source-of-truth volume, complaints, and conduct data from the AR’s own systems so the quarterly MI return is populated from a system of record rather than retyped from a spreadsheet. Enrichment pulls the AR firm’s static facts (controllers, charges, credit standing, FCA permissions) from external public and commercial APIs so the AR detail surfaces drift the moment it happens.

Both classes share one architecture: a connector definition, a sync record, idempotency keys, field-level provenance on the consumer side, and an audit event per sync.

Why bother

The alternative — and it is the alternative most principal firms run today — is the AR retyping numbers from a broker portal into a monthly spreadsheet that the principal pastes into another spreadsheet. The data is stale, partial, and unreproducible. PS22/11’s expectation that the principal “must evidence supervision” is not credible against numbers entered by hand.

Connector architecture replaces the spreadsheet email-tag with a deterministic feed. The MI return becomes a confirmation that the data the principal already has is correct, rather than the principal’s only source.

Connector shape

// lib/types.ts (excerpt)
export type ConnectorKind =
  | "crm-webhook"
  | "lender-portal"
  | "csv-upload"
  | "complaints-system"
  | "companies-house"
  | "creditsafe"
  | "fca-register";

export type ConnectorPurpose = "mi-ingestion" | "enrichment";

export type ConnectorStatus =
  | "connected"
  | "syncing"
  | "error"
  | "not-configured";

export interface DataConnector {
  id: Ulid;
  kind: ConnectorKind;
  purpose: ConnectorPurpose;
  label: string;
  description: string;
  status: ConnectorStatus;
  lastSyncAt: IsoTimestamp | null;
  cadenceLabel: string;
  enrichedFields: string[];
  arCoverage: number;
  errorMessage: string | null;
  providerDocsUrl: string | null;
}

In production the connector record is the configuration. Authenticated handles (webhook secrets, API keys, OAuth refresh tokens) live in a separate, encrypted credential store keyed by connector id. The connector record itself is safe to log and surface in the principal-admin UI.

MI ingestion connectors

CRM webhook

The AR’s CRM (the most common source of case-completion data) posts events to a per-tenant webhook on the platform. Volumes, complaints, and conduct events arrive in real time. The webhook is signed with HMAC over the request body using a secret rotated quarterly. Each event is idempotent on the CRM’s own case ID, so retries are safe.

Production endpoint: POST /api/connectors/crm/:tenantId/events with X-Connector-Signature: sha256=… header. Deliveries failing signature verification are logged but not processed; rate-limited at 1000/min/tenant at the edge.

Lender portal feeds

Pulls from the principal’s lender-portal aggregator on a 4-hourly schedule. Application volumes, completions, decline reasons, procuration-fee events. Replaces the most-spreadsheet part of the existing manual MI flow. Idempotent on the lender’s submission reference.

Production: scheduled job per tenant, cursor-based pagination, retried with exponential backoff on transient failures. Errors recorded as ConnectorSyncError audit events.

CSV upload (fallback)

For ARs without an integrated source system, drag-and-drop CSV upload of the monthly volume and complaints data. Schema published in the AR onboarding pack. Validation in-browser via Zod, server-side re-validation on upload, idempotent on (arId, period, fileSha256).

CSV is not a great answer but it’s a necessary one — some smaller ARs genuinely don’t have an integrated source system, and forcing them off CSV before they’re ready risks the AR pulling out of the network.

Complaints system

Bidirectional sync with the principal’s central complaints system. Customer complaints originating with an AR are tagged with originatingArId; outcome and remediation flow back so the DISP 1.10 half-yearly return aggregates cleanly at the principal level. See DISP 1 for the regulatory background.

Production: webhook on inbound events, polling on outbound updates (because the complaints system is usually the system of record and the platform is enriching, not driving). Idempotent on the complaints system’s case reference.

Enrichment connectors

These pull the AR firm’s static facts from external APIs. The values become first-class fields on the AR detail with field-level provenance (“Source: Companies House, synced 4h ago”) and last-sync timestamps. A daily reconciliation pass detects drift and writes a ConductEvent of type controller-change, permission-change, or credit-rating-change for any material movement.

Companies House

Public free API. Pulls the AR’s company filings, directors, controllers (PSC register), charges, and confirmation-statement cycle. Director changes flag automatically as a conduct event for compliance review. Provider docs: https://developer.company-information.service.gov.uk/.

Sync cadence: daily by default. The CHANGED filter on the streaming API can drop this to near-real-time for tenants where the change-detection latency matters.

Field map:

Field on AR detail	Companies House field
Company number	company_number
Registered office	registered_office_address
SIC codes	sic_codes[]
Directors	officers[] (active filter)
Persons with significant control	persons-with-significant-control[]
Charges	charges[] (outstanding)
Confirmation-statement due	confirmation_statement.next_due
Accounts due	accounts.next_due

CreditSafe

Commercial. Pulls the AR firm’s credit rating, CCJ history, payment-behaviour score, and risk-monitoring alerts. Drops in the AR’s credit standing as a SUP 12.4 due-diligence input on appointment and as ongoing monitoring data thereafter. Provider docs: https://www.creditsafe.com/gb/en/business/api.html.

Sync cadence: weekly default, real-time alert subscription for ratings movements and new CCJs.

Field map:

Field on AR detail	CreditSafe field
Credit rating	companyRating.commonValue
Credit limit	creditLimit.value
CCJ summary	legalAndJudicialInformation.ccjSummary
Payment-behaviour score	paymentData.dbt
Risk-monitoring alerts	monitoringAlerts[]

FCA Register

The FCA’s public Register API. Pulls each AR’s FRN status, permitted regulated activities, SMCR-certified individuals, and historic permission changes. Reconciles nightly so the in-product permissions table never drifts from the public Register. Provider docs: https://register.fca.org.uk/Services/V0.1/Help/Index.

This is the connector that catches scope changes the AR forgets to tell the principal about. A permission revoked on the FCA side without the AR notifying is a notification-failure event under SUP 12 — and the connector surfaces it within 24 hours.

Field map:

Field on AR detail	FCA Register field
FRN status	Status
Trading names	OtherNames[]
Permitted regulated activities	Permission[]
Approved persons / certified individuals	Individuals[]
Permission change history	PermissionHistory[]
Disciplinary history	DisciplinaryHistory[]

Field-level provenance

Every enriched field on the AR detail carries provenance. The component reads field.source (connector kind), field.lastSyncAt (timestamp), and field.confidence (where the source supplies one) and renders a small badge under the value. On hover the user sees the full source URL and the previous value if the field changed in the last 30 days.

interface EnrichedField<T> {
  value: T;
  source: ConnectorKind;
  sourceUrl: string;
  lastSyncAt: IsoTimestamp;
  confidence: number | null;
  previousValue: T | null;
  changedAt: IsoTimestamp | null;
}

This is the single most regulator-credible feature of the platform: an auditor pulling up an AR record sees not just “this is the controller”, but “this came from Companies House at 06:14 this morning, here is the previous value that changed last week, here is the conduct event we wrote when it changed”.

Idempotency and ordering

Every external event carries a stable identifier (the lender’s submission reference, the CRM’s case ID, Companies House’s transaction_id). Sync writes are upserts keyed on (connectorId, externalId). Out-of-order delivery is handled by recording the source’s own lastModifiedAt on the row and rejecting writes older than the recorded value.

For Companies House and FCA Register, the source is the system of record and the platform is the cache; for CRM and lender portals, the source is the source-of-truth and the platform aggregates. The data model is the same in both cases: the platform never overwrites a more-recent value with an older one.

Errors and degraded modes

Connector failures don’t block the platform. Each connector has a status field that becomes error on repeated sync failures, and the affected AR detail renders the affected fields with a “Stale, last sync 14h ago” badge instead of the live value. The principal-admin sees the error on the connectors settings page with the underlying message; alerting fires to the principal-admin’s email after a configurable failure window.

The system never serves silently-stale data: if a sync hasn’t run in over 2x the cadence interval, the field is rendered as stale.

Mock-vs-real boundary additions

Surface or concern	v1 demo	Production
Connector catalogue	lib/connectors.ts, mock-connected	Per-tenant connector configs in Postgres, credentials in encrypted store
MI ingestion	Fixture-set MI returns	Real-time CRM webhooks, scheduled lender-portal pulls, CSV upload with virus-scan
Companies House sync	Static enriched-field strings on AR detail	Daily delta pull, CHANGED stream subscription, controller-change conduct events
CreditSafe sync	Static enriched-field strings	Weekly default + real-time alert subscription, rating-change conduct events
FCA Register sync	Static enriched-field strings	Nightly reconciliation, permission-change conduct events within 24 hours
Field provenance	Implicit (“Auto-enriched: …”)	Per-field EnrichedField<T> shape with source URL and change history
Connector errors	Toggleable in the demo via the Connectors settings page	ConnectorSyncError audit, alerting to principal-admin, stale-field badging
Webhook security	Not applicable	HMAC-signed bodies, signature-rejection logs, key rotation

See also

• Data model — where enriched fields land in the persisted AR record.
• Mock-vs-real boundary — the wider seam between demo and production.
• Oversight tasks — the supervisory tasks that consume this data.
• SUP 12 (the AR rulebook) — the regulatory motivation for the FCA Register sync.