API Fetcher

Import source data straight from your own API. Point WeTransform at the endpoint that lists your items, choose what to import, and the fetch is replayed on every refresh.

How It Works

Setup is a six-step guided wizard. Its central idea: you never type a JSON path by hand. The wizard performs real test calls against your API and everything — the records collection, the item id, the pagination fields, the columns — is picked by clicking the live response.

Once saved, a fetch run looks like this:

1. Authentication

Fetches run on a schedule with nobody watching, so the credentials must keep working on their own. Three strategies are supported:

2. Entry Point & Records

Give the address of the endpoint that lists your items (GET or POST). The wizard calls it once — a read-only test fetch, nothing is saved on either side — and shows you the live JSON response.

Then click the collection in that response that holds your items: each of its elements becomes one row of your source. Nested collections are selectable too — for example data to import orders, or data[].lines to import the order lines instead.

{
  "data": [
    {
      "id": 4821,
      "sku": "BRK-220",
      "customer": "Northwind Co",
      "total": 1290.50,
      "status": "shipped",
      "lines": [ { "ref": "BRK-220", "qty": 2 }, { "ref": "CAP-7", "qty": 5 } ]
    }
  ],
  "meta": { "page": 1, "per_page": 100, "total": 1387, "total_pages": 14 }
}

3. Item Details (optional)

Some APIs only return a summary in the list, and the full record needs a second call. When that is your case, enable the detail request: WeTransform will call a detail endpoint once per item.

The item id is a pill you insert anywhere in the detail URL. You pick the id field by clicking it in the list response — for example data[].id — and WeTransform substitutes each item's value at fetch time:

4. Pagination

Most list endpoints return one page at a time. Pick how WeTransform walks the rest — pagination values are read from the JSON response body only.

The total-count field (e.g. meta.total_pages) and the next-cursor field are, as everywhere else in the wizard, picked by clicking the live response — never typed by hand.

5. Picking Columns

The wizard pulls the first page (and, if configured, the first item's detail) and shows two checklists: the columns available from the list, and the columns available from the item detail. Tick the fields you want.

Below the checklists, a single joined preview table shows exactly what will land in your source — list columns and detail columns side by side, with real values from your API. What you see is what you import.

Finish with Save & run fetch: the configuration is stored and the first full import starts immediately.

Try It Live: The Movie Catalog

Want to walk through the wizard before wiring up your own API? We host a public demo API — a small movie catalog — with exactly the shape the API Fetcher expects: a paginated listing plus a per-movie detail endpoint. Point the wizard at it and click through every step end to end.

1. Authentication

Both endpoints require an API key. To keep the demo simple, any key whose last character is an even digit is accepted (e.g. demo-42); a missing or odd key returns 401. Choose API key in the Authentication step and send it as an Authorization header, an X-API-Key header, or a token query parameter.

2. Entry Point & Records

GET https://api.wetransform.com/en/documentation/api-fetcher/movies?page={page}

The records live under the results array — pick it in the Locate records step. Each item carries an id.

{
  "page": 1,
  "per_page": 10,
  "total_pages": 3,
  "total_results": 30,
  "results": [
    { "id": 1, "title": "The Shawshank Redemption", "year": 1994, "genre": "Drama", "rating": 9.3 },
    { "id": 2, "title": "The Godfather", "year": 1972, "genre": "Crime", "rating": 9.2 }
  ]
}

3. Item Details

Tick "Another request is needed to fetch more details" and use the item-relative token {id} in the detail URL. The catalog serves richer fields (runtime, cast, director, plot) that are not in the listing.

GET https://api.wetransform.com/en/documentation/api-fetcher/movies/{id}

{
  "id": 3,
  "runtime_minutes": 152,
  "director": "Christopher Nolan",
  "cast": ["Christian Bale", "Heath Ledger", "Aaron Eckhart"],
  "language": "English",
  "plot": "Batman faces the Joker, a criminal mastermind who plunges Gotham City into anarchy."
}

4. Pagination & Columns

• Use page number pagination on page. Pages 1–3 hold data; page 4 returns an empty results, so the fetch stops.
• Optionally stop on the total_results count (30).
• In Pick data, combine list columns (title, year, rating) with detail columns (runtime_minutes, cast) into one joined table — the detail call is what supplies runtime and cast.

Refresh & Monitoring

The fetched data becomes a regular source: it flows through parsing, matching and transformation like any uploaded file. The difference is the refresh — every source refresh replays the whole fetch (all pages, all detail calls) so your source always mirrors the API.

• Run history: each fetch logs its status, pages and rows fetched, and any error — with the failing request and response captured for debugging.
• Item limit: an optional maximum number of items bounds the fetch — handy to test a configuration on a small sample before importing everything.
• Reusable: a saved configuration can be reused to create new sources without going through the wizard again.