Custom data extractor

A custom data extractor lets WeTransform pull data out of your system, on demand, without you having to upload anything. When a transformation needs fresh data, we call your server; your server answers with a link to the file; we download it and carry on.

When should I use this?

Use it when the export takes time to produce, or when the data lives behind your own business logic. If the file is already reachable at a stable address, an URL or FTP source is simpler. If your system can push on its own schedule, uploading through the API is simpler still. The custom extractor exists for the case where we need to ask, and you need a moment to answer.

Lifecycle

Four steps. Two of them are yours.

PING — we POST a signed payload to your ping URL, telling you which customer needs data and where to answer.
You produce the file — take as long as you need. The request can return immediately; nothing is blocked on it.
PONG — you POST back to the collect URL we gave you, with a download link.
DOWNLOAD — we fetch that link and the transformation resumes on its own.

The PING response body is ignored. Answering 200 OK with an empty body is perfectly fine — what matters is the separate PONG call you make afterwards.

Setup

Create the extractor once, then point as many sources at it as you like.

1. Create the extractor

Go to Integrations → Custom data extractors.

Name and handle: how you and your senders recognise it.
Ping URL: the endpoint on your server we will call. It must be reachable from the public internet — private and internal addresses are refused.
Signature secret: generated for you, used to sign every PING. Keep it secret; you can regenerate it at any time.

2. Create the source through the API

Custom sources are created programmatically, not from the source picker — the picker has no way to know which extractor and which metadata a given source should carry. Use POST /sender/source:

POST /sender/source
Authorization: Bearer {your_api_token}
X-Customer: shop-42
Content-Type: application/json

{
  "name": "Daily orders",
  "data_extractor": "custom",
  "custom": {
    "handle": "my-export",
    "metadata": [
      { "name": "warehouse", "value": "lyon-1" }
    ]
  }
}

3. What goes in that call

handle — the handle you gave the extractor. You may send custom_data_extractor_id with its UUID instead; exactly one of the two is required.
metadata — arbitrary name/value pairs, sent back to you unchanged with every PING, so your server knows which export to produce (a shop id, a warehouse code, a report type). Optional, and free-form.
X-Customer — the customer this source belongs to, by external id, email or UUID. The same impersonation header you will use to answer the PING.
That customer must be allowed to use this extractor. If your role configuration restricts a customer to a subset of extractors, a source pointing at one they do not hold is refused.

1. Receiving the PING

We send a POST with a JSON body to your ping URL.

Payload

{
  "collect_url": "https://acme.send-a-file.io/en/send/acme/custom-data-extractor/orders/receive/6f1c...",
  "customer_id": "30bb4d27-cf83-4235-97af-9557bb5c623c",
  "customer_external_id": "shop-42",
  "customer_email": "ops@customer.example",
  "source_id": "53867fb0-5bf2-45c8-96e4-8dab89ffcc8b",
  "metadata": [
    { "name": "warehouse", "value": "lyon-1" }
  ]
}

collect_url — where you answer. It is single-use and tied to this one request; do not cache or reuse it.
customer_id / customer_external_id / customer_email — who the data is for. Use whichever identifier you already store.
source_id — the source being fed, stable across runs.
metadata — exactly what was configured on the source.

2. Verifying the signature

Every PING carries an X-Signature header: an HMAC-SHA256 of the raw request body, keyed with your extractor's signature secret.

Always verify the signature, and compare it in constant time. Your ping URL is public, so this is what proves the request came from us.

Implementation (PHP)

$secret = 'your_signature_secret';
$body = file_get_contents('php://input');
$signature = $_SERVER['HTTP_X_SIGNATURE'] ?? '';

$expected = hash_hmac('sha256', $body, $secret);

if (!hash_equals($expected, $signature)) {
    http_response_code(403);
    return;
}

$ping = json_decode($body, true);
// Queue your export, then answer the collect URL when it is ready.
http_response_code(200);

3. Answering with your file

Once your export is ready, POST the download link to the collect_url.

Request

POST {collect_url}
Authorization: Bearer {your_api_token}
X-Customer: shop-42
Content-Type: application/json

{
  "url": "https://files.yourcompany.example/exports/orders-2026-08-01.csv"
}

The collect URL is authenticated

The signature secret signs what we send you; it does not authenticate what you send back. Answer with your own WeTransform API token as a bearer token, plus an X-Customer header naming the customer from the PING — its customer_external_id, customer_email or customer_id, whichever you already have. One organisation-level token covers every customer, so there is nothing per-customer to store.

The url you return must be reachable by us and point directly at the file. It may be a signed, short-lived link — we fetch it immediately.

4. We download it

We fetch the URL, store the file as the source's content, and the transformation continues exactly as if the file had been uploaded by hand. Any format your template accepts works — CSV, Excel, XML, JSON — and the usual normalisation runs on it.

Answering the same collect URL twice is safe: the second call is ignored rather than producing a duplicate import.

Testing

Send a test PING

On the extractor's settings page, Test sends a real signed PING to your ping URL and shows you what your server answered — status code, body and any connection error. It is the fastest way to check reachability and your signature check.

The test payload carries placeholder identifiers (_test_source_id_) and a collect URL reserved for testing. No source is touched, so you can fire it as often as you like.

Guard against the test payload

Because a test PING is otherwise indistinguishable from a real one, check for source_id === "_test_source_id_" if you want to answer it without running a real export.

Errors & timeouts

Your server refuses the PING (4xx). A rejected signature, a wrong URL, a route that does not exist — a retry would hit the same wall, so the import stops immediately and the source shows the status your server returned. Fix the endpoint, then re-run the import.
Your server is unreachable, or answers 5xx. That may be a passing outage, so the source keeps waiting rather than failing. Re-run the import once your endpoint is healthy.
You never answer the collect URL. The source stays waiting for data — it does not fail on its own today, so if an export dies on your side, re-run the import rather than waiting.
The download link is dead or returns an error. The download is retried; a link that stays unreachable leaves the source waiting.
The organisation is not allowed to use custom extractors, or the extractor has no ping URL — nothing is sent at all. Check the extractor's settings first if you see no traffic whatsoever.

Every PING, PONG and DOWNLOAD is recorded with its full request and response for 30 days. If an integration misbehaves, that trail is what our support team reads — quote the source id when you get in touch.