Advanced Analytica Advanced Analytica: IBOM
← All connectors
End-to-End Testing (datagen) logo

Source connector

End-to-End Testing (datagen)

Set up End-to-End Testing (datagen) as a source and move its data into your chosen destination.

Version 0.2.5 community alpha

DataGen

The DataGen source connector generates synthetic data for testing and development. You can test data destinations end to end. You can also test Airbyte configurations in speed mode without accessing an external data source.

Prerequisites

You don’t need any prerequisites. DataGen generates data locally and doesn’t connect to external systems.

Set up DataGen

  1. Log in to your Airbyte Cloud or Airbyte Open Source account.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select DataGen from the Source type dropdown.
  4. Enter a name for your DataGen source.
  5. Configure the data generation settings:
    • Data Generation Type: Choose Incremental, All Types, or Wide.
    • Max Record: Enter the total records to generate, from 1 to 100 billion. The default is 100.
    • Column Count (Wide mode only): Set the number of columns, from 1 to 1000. The default is 50.
    • Max Concurrency (optional): Set the maximum number of concurrent data generators. Leave this empty so Airbyte can manage performance automatically.
  6. Click Set up source.

Supported sync modes

The DataGen source connector supports the following sync mode:

FeatureSupported?
Full Refresh SyncYes
Incremental SyncNo

Data generation types

The connector supports three data generation patterns:

Incremental

Generates a stream named increment with one column named id. The column contains monotonically increasing integers. Use this mode to test incremental data loading and confirm that data arrives in the expected order.

All types

Generates a stream named all types with columns for these Airbyte data types:

  • id
  • string
  • boolean
  • number
  • big integer
  • big decimal
  • date
  • time, with and without time zones
  • timestamp, with and without time zones
  • JSON

Use this mode to test type handling and schema compatibility across different destinations.

Wide

Generates a stream named wide with 1–1000 configurable columns. The default is 50. Column 0 is always id, the integer primary key.

The remaining columns cycle through all 12 Airbyte data types:

  • integer
  • string
  • boolean
  • number
  • big integer
  • big decimal
  • date
  • time with timezone
  • time without timezone
  • timestamp with timezone
  • timestamp without timezone
  • JSON

Column names follow the pattern col_1_integer, col_2_string, and so on. Use this mode to test wide schema handling and destination performance with many columns.

Configuration

ParameterTypeDefaultDescription
Data Generation TypeenumIncrementalThe data generation pattern to use. Choose Incremental, All Types, or Wide.
Max Recordinteger100The total number of records to generate. Minimum 1, maximum 100 billion.
Max Concurrencyinteger(auto)Maximum number of concurrent data generators. Leave empty to let Airbyte optimize performance automatically.
Column Countinteger50Wide mode only. The number of columns to generate, including the id column. Minimum 1, maximum 1000.

IP allow list

If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.52026-10-2289643Upgrade to Bulk CDK 1.2.3: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86.
0.2.42026-10-1689066Bump Bulk CDK to base 1.0.6 / extract 1.1.21 (Kotlin 2.4 / K2 toolchain, Micronaut 4.10.31, Jackson 2.21.7).
0.2.32026-10-0988425Upgrade to Bulk CDK 1.1.14.
0.2.22026-07-1081656chore(source-datagen): dummy version bump for progressive rollout (autopilot) testing
0.2.12026-06-0979330Progressive rollout e2e test
0.2.02026-04-2875542Add wide schema flavor with configurable column count; fix null safety in partition reader; cache codec references; bump CDK to 1.1.6
0.1.62025-10-2368611Update cdk version
0.1.52025-10-2168581Update dataChannel version
0.1.42025-10-1668131Increment naming fix
0.1.32025-10-1668129Increment encoding fix
0.1.22025-10-1467720Removal of Array type
0.1.12025-10-1367110Addition of proto types
0.1.02025-09-2966331Creation of initial DataGen Source
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.