For paths, parameters, and response schemas, refer to the in-app documentation at the address above.

L’article API est apparu en premier sur ETL administration console.

Run a new job version safely while your scheduled task keeps using the current, stable version.

Prerequisites

• You have a scheduled task already running on version A.
• You’ve deployed job version B that you want to test.
• Role: Admin (to edit tasks and versions).

Steps

1. Pin the current scheduled task to its version

• Go to Task › Task setup of your scheduled task.
• Select the desired Job version = A.
• Uncheck “Follow the new version”.
  This pins the task to version A even if new versions are deployed.

2. Create a separate test task for the new version

• From the Job or Tasks list, create a new task (e.g., MyTask_test).
• In Task setup, choose Job version = B.
• Leave the test task unscheduled (manual runs only), or schedule it off-hours for testing.

3. Run and validate

• Launch the test task manually.
• Check logs, status, and metrics as usual.

4. Promote the new version to production

• When satisfied, go back to the scheduled task:
  • Either re-check “Follow the new version” (so it will pick up version B now and future versions),
  • or explicitly select Job version = B to stay pinned to B.

5. Clean up

• Delete the temporary test task if no longer needed.

Notes & Tips

• Why this works: unchecking “Follow the new version” freezes a task on a specific job version, so your schedule remains stable while you test elsewhere.
• Use a clear naming convention for test tasks (e.g., *_test) and keep them manual to avoid accidental runs.
• If contexts differ between versions, review Task setup › Context before testing.
• Rollback is instant: just switch the scheduled task back to version A (or keep it pinned).

L’article Test a New Job Version Without Impacting the Schedule est apparu en premier sur ETL administration console.

On master server

Install NFS service:

sudo apt update
sudo apt install nfs-kernel-server

Edit exports file:

sudo nano /etc/exports

Define rights for slave servers (imagine slave1 is is identified by 192.168.1.2 and slave 2 by 192.168.1.3):

/var/www/html/shared/var/eac 192.168.1.2(rw,sync,no_subtree_check) 192.168.1.3(rw,sync,no_subtree_check)

On slaves

Install NFS client:

sudo apt install nfs-common

Create mount point (192.168.1.1 is the main server):

sudo mkdir -p /mnt/eac
sudo mount 192.168.1.1:/var/www/html/shared/var/eac /mnt/eac

Mount on server startup:

sudo nano /etc/fstab

192.168.1.1:/var/www/html/shared/var/eac /mnt/eac nfs defaults 0 0

L’article Setup mount point between servers est apparu en premier sur ETL administration console.

Quick wins (checklist)

• Configure scheduled purges for executions and logs.
• Watch Execution Server graphs (CPU %, Memory %) and error KPIs.
• Keep scheduling enabled and tune reschedule delay.
• Run server status sampler every minute and purge old statuses daily.
• Keep ETL log tables small via retention + DB maintenance.

Execution & log retention

Each run writes an execution record and logs. Without retention, UI lists and dashboards slow down and DB/files grow indefinitely.

Strategies

• History window (recommended): keep the last 30–90 days per task.
• Fixed count: keep the last 200–1000 executions per task.
• Longest executions: keep 10–50 longest for performance analysis.

Set these via:

• Manual purge: Tasks → {Task} → Last executions → Purge the executions of the task
• Scheduled purge: Tasks → {Task} → Cron planification → Add scheduler → Purge past executions
  Modes: date, history, total, duration.

See the Purges page for details and examples.

Monitor servers

Open Monitoring to validate:

• Each node sends heartbeats (CPU/Memory sparklines not flat at 0%).
• Error Execution gauges (day/week/month) stay near zero.
• The master and any slaves show “Last status received … seconds/minutes ago”.

Tip: if “hours ago”, your sampler isn’t running—see below.

Screenshot placeholders:

• ![Servers dashboard – CPU/Memory](path/to/monitoring-servers.png)
• ![Server detail](path/to/monitoring-server-detail.png)

Scheduling hygiene

In Settings → Scheduling & System:

• Enable scheduling: true.
• Reschedule old task after X (min): start with 10; raise to 15–30 on busy systems.
• Keep Cron user set to your service account (e.g. eac) and ensure directories are writable.

Avoid enabling parallel launch unless the ETL job is idempotent and designed for concurrency.

Server status sampler & purge

EAC samples node CPU/Memory via a console command and stores recent points for graphs.

Sampler (every minute)

Run on every EAC node:

php bin/console eac:server-status --every=3 --env=prod

Cron example (bare metal/VM)

* * * * * cd /var/www/html/eac && /usr/bin/php bin/console eac:server-status --every=3 --env=prod >/dev/null 2>&1

Daily purge of old samples (required)

Removes stale status rows to keep graphs snappy:

php bin/console eac:server-status:purge --days=4 --env=prod

Cron (daily at 02:10)

10 2 * * * cd /var/www/html/eac && /usr/bin/php bin/console eac:server-status:purge --days=4 --env=prod >/dev/null 2>&1

Systemd timer (optional)

# /etc/systemd/system/eac-server-status.timer
[Timer]
OnCalendar=*:0/1
Persistent=true

# /etc/systemd/system/eac-server-status-purge.timer
[Timer]
OnCalendar=daily
Persistent=true

Keep 4–7 days of server statuses; more provides no benefit and grows storage.

ETL logging footprint

If ETL logs are enabled (APP_ENABLE_ETL_LOGS=true):

Retention. Purge old execution records on a schedule; keep only what you need for ops & audits.
Scope. Apply purges to ETL_STAT_TABLE_NAME, ETL_MET_TABLE_NAME, and ETL_LOG_TABLE_NAME.
Performance. Add indexes on the fields you filter by most (start_time, job, status, context).

— PostgreSQL (rename tables/columns to match your schema)

CREATE INDEX IF NOT EXISTS idx_stat_start  ON talend_stat(start_time);
CREATE INDEX IF NOT EXISTS idx_stat_job    ON talend_stat(job);
CREATE INDEX IF NOT EXISTS idx_stat_status ON talend_stat(status);

Log backend choice

Settings → Task logs type:

• db: fastest for searching/paging inside EAC; increases DB size.
• file: reduces DB size; ensure the log directory is on fast storage and rotated by purges.

Runtime tips

• Keep Symfony in prod mode: APP_ENV=prod, APP_DEBUG=0; warmup cache after deploy: php bin/console cache:clear --env=prod && php bin/console cache:warmup --env=prod
• Disable dev extensions (Xdebug, verbose loggers) on production nodes.
• JVM sizing for Talend jobs: keep -Xms small, -Xmx appropriate to dataset; avoid overcommitting memory.

L’article Performances est apparu en premier sur ETL administration console.

• File backend: logs written under your log directory (see Settings → Local/Upload/Temp directories).
• DB backend: logs persisted in the application database.
• ETL logs (Talend): optional, pulled from your ETL logging database when enabled.

Tabs in the execution view

There are four log tabs. The first mixes EAC runtime output and the job’s console. The last three are sourced from your Talend/log4j logging (when ETL logs are enabled).

1. Execution logs
   Mixed stream of:
   • EAC scheduler/runtime messages (enqueue, script path, PID, status, warnings)
   • Job STDOUT/STDERR (e.g., System.out.println(...), tLogRow, shell output)
     
2. ETL Logs (Talend log)
   Structured application log lines emitted by your job (log4j). Useful to trace component messages and exceptions.
   
3. ETL Statistics
   One row per run with start/end times, status, duration, context, job, etc.
   
4. ETL Metrics
   Per-component counters (rows read/written/rejected, etc.).
   

The ETL Logs / Statistics / Metrics tabs are populated only if ETL logging is enabled and your ETL studio writes to a database.

Enable ETL logs, statistics, and metrics

1. Configure your ETL studio (Talend)
   Go to: File → Project Settings → Job parameters → Stats & Logs, and select Database as the target.
2. Enable ETL logs in EAC (.env.local)

APP_ENABLE_ETL_LOGS=true

3. Provide the ETL logging DB connection

# Examples
# PostgreSQL
DATABASE_URL_ETL_LOG="postgresql://etl_user:pass@etl-db:5432/etl_logs?serverVersion=16&charset=utf8"
# MySQL/MariaDB
# DATABASE_URL_ETL_LOG="mysql://etl_user:pass@etl-db:3306/etl_logs?charset=utf8mb4"
# SQL Server
# DATABASE_URL_ETL_LOG="mssql://etl_user:pass@etl-db:1433/etl_logs?TrustServerCertificate=true"
# Oracle/sqlite also supported

4. Declare ETL table names

ETL_STAT_TABLE_NAME=talend_stat
ETL_MET_TABLE_NAME=talend_met
ETL_LOG_TABLE_NAME=talend_log

5. Correlate the ETL run with the EAC execution
   In your main job, print the root PID once (e.g., in a tJava):

System.out.println("@@eac@@root_pid=" + rootPid);

EAC uses this marker to join your console run with the ETL rows.

What goes where?

• Execution logs (tab 1):
  • Queueing + path to *_run.sh
  • PID and scheduler messages
  • All console output from your job (stdout/stderr)
  • Final SUCCESS/ERROR/WARNING marker
• ETL tabs (tabs 2–4):
  • Populated from the ETL logging DB via DATABASE_URL_ETL_LOG
  • Logs = log4j lines; Statistics = per-run aggregates; Metrics = per-component counters

Storage backend for tab 1 is defined by Task logs type (file or db). ETL tabs always read from the ETL logging database.

Best practices

• Keep console output actionable: prefer concise System.out.println(...)/tLogRow for key checkpoints.
• Use log levels in Talend (log4j) for noise control.
• Redact secrets in both console and ETL logs.
• Retention: set up Purge jobs to keep history lean (see Purges page).

L’article Execution logs est apparu en premier sur ETL administration console.

• .env — baseline defaults (versioned, read-only from the UI).
• .env.local — per-server overrides (edited from the Settings page or manually on the host).

No container/OS environment variables are used by default in this setup.

Precedence (highest → lowest)

Only two levels apply:

• If a key is present in .env.local, it overrides the value from .env on this server only.
• If the key is absent in .env.local, EAC falls back to .env.

Multi-server: each node has its own .env.local. Editing Settings on server A does not change server B.

What the Settings page actually does

• Fields labeled “Updatable (.env.local)” are written to the node’s .env.local when you Save.
• Fields labeled “.env (Present value)” show the baseline from .env and are not modified by the UI.
• Composite inputs (e.g., Mail) are assembled into a single DSN such as MAILER_DSN and written to .env.local.

To remove an override, clear the field in the UI and Save (the app will fall back to .env).
Alternatively, edit the file and delete the line.

Applying changes

After updating .env.local, reload so PHP reads the new values:

# From the app root
php bin/console cache:clear --env=prod && php bin/console cache:warmup --env=prod

# Then reload PHP-FPM (typical Debian/Ubuntu service name)
sudo systemctl reload php8.2-fpm  # adjust version if needed

If you run Symfony workers/cron wrappers under a service account, restart those units as well.

File locations & permissions (recommended)

• Location: by default EAC expects these files under the application root (e.g., /var/www/html/eac/.env and /var/www/html/eac/.env.local).
  If your build stores overrides under ./shared/.env.local, keep that convention consistent in your docs/scripts.
• Ownership: make them writable by your service user/group (e.g., eac:www-data).
• Mode: chmod 640 (or 600 if only the owner should read). Keep secrets out of world-readable files.

sudo chown eac:www-data /var/www/html/eac/.env.local
sudo chmod 640 /var/www/html/eac/.env.local

What belongs where

Common keys overridden in .env.local:

• Mail: MAILER_DSN, APP_MAILER_SENDER, APP_MAILER_DEFAULT_RECIPIENTS, APP_ENABLE_TASK_MAILER
• DB: DATABASE_URL, DATABASE_URL_ETL_LOG
• Paths: APP_CONSOLE_CMD, APP_DEPLOY_DIR, APP_TMP_DIR, APP_UPLOAD_DIR
• Feature toggles: APP_ENABLE_SCHEDULING, APP_CHECK_STATUS_WITH_GIT, APP_ENABLE_ETL_LOGS

Examples

Baseline in .env

APP_ENV=prod
APP_DEBUG=0

APP_BASE_URI="https://eac.example.com/"
APP_ENABLE_SCHEDULING=true

DATABASE_URL="postgresql://eac_usr:pass@db-host:5432/eac_db?serverVersion=16&charset=utf8"

# Mail defaults (can be empty here)
APP_MAILER_SENDER="eac@cidwe.com"
MAILER_DSN=""

Per-server override in .env.local

# Different base URI on this host
APP_BASE_URI="https://eac-paris.example.com/"

# Mail configured on this server
APP_ENABLE_TASK_MAILER=true
APP_MAILER_SENDER="no-reply@example.com"
APP_MAILER_DEFAULT_RECIPIENTS="ops@example.com,data@example.com"
MAILER_DSN="smtp://no-reply%40example.com:My%3APass@smtp.example.com:587?encryption=tls&auth_mode=login"

# ETL logs enabled on this node
APP_ENABLE_ETL_LOGS=true
DATABASE_URL_ETL_LOG="postgresql://etl_ro:pass@etl-db:5432/etl_logs?serverVersion=16&charset=utf8"

Result: all keys present in .env.local override the baseline; others fall back to .env.

L’article .env.local est apparu en premier sur ETL administration console.

Purging removes executions of the selected task and their attached logs (according to your log backend). It does not delete the “purge” executions themselves.

Why purge?

• Keeps the database small and fast (queries, dashboards, filters).
• Reduces disk usage (when log type = file). see settings( scheduling & system > task logs type)
• Improves backup/restore times.
• Enforces data retention policies.

Manual purge (per task)

From a task page:

1. Open Tasks → {Your task}.
2. Click Purge the executions of the task.
3. In the dialog, pick a Purge mode and fill its parameter(s).
4. Click Launch the purge.

Screenshots (placeholders):

the date picker input format depends on your web browser

Purge modes

• date — Purge all before a date
  Parameter: Purge before date (YYYY-MM-DD).
  Example: delete everything started before 2025-06-01.
• history — Keep only the last N days
  Parameter: Days to keep (integer).
  Example: keep 30 days of history; remove older ones.
• total — Keep only the last N executions
  Parameter: Executions to keep (integer).
  Example: keep 200 most recent runs; remove the rest.
• duration — Keep only the longest N
  Parameter: Executions to keep (integer).
  Example: keep 10 longest runs (useful for performance analysis).

The UI asks for the appropriate numeric value or date depending on the selected mode.

Scheduled purge (recommended)

Automate cleanup with the built-in scheduler.

1. Open Tasks → {Your task} → Cron planification.
2. Click Add scheduler.
3. Task type: Purge past executions.
4. Purge mode: choose one of date / history / total / duration.
5. Set Begin at / Finish at if needed.
6. Define the Cron (second, minute, hour, day, month, day-of-week).
7. Check Schedule active → Save.

Example schedules

• Nightly, keep last 30 days
  • Mode: history = 30
  • Cron: 0 0 * * * (every day at 00:00)
• Weekly, keep last 500 executions
  • Mode: total = 500
  • Cron: 0 0 * * 0 (Sundays at 00:00)
• Monthly, keep the 20 longest (capacity planning)
  • Mode: duration = 20
  • Cron: 0 3 1 * * (1st of month at 03:00)

Choosing a retention strategy

Use one or combine several purge schedules:

• Production (typical):
  • Keep 30–90 days with history, nightly.
  • Additionally keep last 200–1000 runs with total to preserve dense recent activity.
• Non-prod (dev/test):
  • Keep 7–14 days with history, nightly.
  • Or last 100–300 runs with total.
• Performance analysis windows:
  • Monthly duration keeping 10–50 longest executions.

How to size retention

• Estimate daily runs per task × average row size in log tables.
• Ensure DB growth stays well below disk thresholds (e.g., <70% usage).
• Prefer shorter history with richer metrics over unbounded growth.

Operational notes

• Multi-server: Purges operate at the data layer; results are visible on all nodes.
• Log backend: If file, purges remove the task’s related files in your log path; ensure the service user has rights to delete.
• Safety: Start with conservative settings, verify results, then tighten retention.

Related

• Acknowledgement (Ack): manage failed runs before relaunch.
• Monitoring: watch error rates and capacity trends to adjust retention.
• Settings → Task logs type: choose file vs db according to your ops policies.

L’article Good use of Execution purges est apparu en premier sur ETL administration console.

When Enable acknowledgement is turned on for a task, any failed execution must be manually acknowledged before the task can be launched again (manually or by the scheduler). This prevents blind relaunches and “error loops.”

How it works

• If the last execution ended in ERROR, the task is marked Not acked.
• While Not acked, any attempt to launch the task is blocked and a warning appears, e.g.: Unable to launch <task> because all execution tasks are not acked.
• Once you Ack the failed execution(s), the task becomes launchable again.
• Ack does not rerun the task; it only clears the block so you can run it.

Scope: Ack state is stored centrally and applies across all servers/nodes.

Enable acknowledgement for a task

1. Open the task: Tasks → {Your Task}
2. Task setup tab → check Enable acknowledgement → Save.
   • UI hint: “If activated, a task in error will not be launched until a manual action (ack) has been performed.”

Where you can Ack

You can acknowledge a single failed execution (Ack) or all failed executions of a task (Ack all):

1. Task detail page (bulk for that task)
   • Icon: Ack all (top action bar)
   • Path: Tasks → {Task}
     
2. Execution detail page (individual execution)
   • Icon: Ack
   • Path: Tasks → {Task} → Last executions → click an execution
     
   • button: Ack all
3. Task execution page
   • path:Tasks → {Task} → Last executions → Top action buttons
   • 
4. Task execution log page
   • Button: Ack (logs for that task)
   • Path:Tasks → {Task} → Last executions → Top action buttons
   • 

Same actions are available In task list and execution list

What you’ll see in the UI

• Status chip: ERROR + Not acked on failed, unacknowledged executions.
  
• Warning banner when try to launch while ack button not triggered.
  
• After Ack: the Not acked label disappears replaced by acked label if « ack » button triggered ; the task can be launched.

Typical workflow

1. A task run fails → shows ERROR and Not acked.
2. Investigate: open Execution logs (and ETL logs if enabled).
3. Fix the root cause (config, data, path, permissions, etc.).
4. Click Ack (or Ack all).
5. Relaunch the task (manually or wait for scheduler).

Troubleshooting

Related

Settings — verify paths, Java/PHP binaries, and permissions if launches fail immediately.on of the acknowledgment mechanism and its use cases

Monitoring — identify failing nodes or resource issues.

Email notifications — enable mailer to receive error alerts.

L’article Good use of Ack est apparu en premier sur ETL administration console.

When and how to use contexts efficiently

Use contexts to:

• Parameterize your jobs (paths, hosts, credentials, feature flags).
• Promote the same job from dev → test → prod without code changes.
• Centralize values that must be consistent across many tasks.
• Override a few values for a specific task only (without forking context files).

Key rule: Define multiple context groups (by project or environment), but only one effective context set applies per task run (computed by precedence below).

Sources & precedence (low → high)

When EAC prepares a run, each context key is resolved using this order:

1. Talend job defaults
   The values embedded in the job for the Default context.
2. Talend environment selected in Task setup
   The context = dev/test/prod… chosen for the task.
3. Global context file (Context Files page)
   If a context file group is linked, keys from the file override Talend values. Only keys that exist in the Talend job are applied; extra keys are ignored.
4. Task-level overrides (Task setup)
   Per-task key/value overrides take final precedence.

UI hint: Values that are actually used are highlighted with a green background in the Task setup preview.

Context files

Centralize values outside the job and reuse them across tasks.

• Where: Context files
  Group files by Project.
  
• Format: Talend-style .properties (UTF-8)
  
  • Spaces are allowed after =.
  • Unknown keys (not declared in the job) are ignored.
  • Prefer one file per environment.
• Attach to tasks: In Task setup, select the Context file group to apply.
  

Applying contexts in Task setup

Open Tasks → {Task} → Task setup to configure:

1. Talend environment (context) to run with: Default | dev | test | prod | …
2. Context file group (optional): link a centralized file to override job values.
3. Task overrides (optional): add/modify individual key/values for this task only.

You’ll see a preview of the effective context with green highlights on applied values.

Keep overrides minimal: add only what must differ for this task.

Execution details

Every run captures the context snapshot used to launch the job:

• Open Tasks → {Task} → Last executions → {Execution}.
• The header and logs remind the Talend env, any context file applied, and overrides.
  

This is essential for auditability and post-mortems.

Best practices

• Define once, reuse widely
  Put shared values in context files (per project/environment). Use task overrides sparingly.
• Name clearly
  projectA-dev.properties, projectA-prod.properties, etc.
• Separate secrets
  Avoid plain-text secrets in files. Prefer vault-backed injection or environment-level secrets; if you must store them, restrict OS permissions.
• Keep types stable
  Don’t switch a key from string to numeric between envs; Talend components may enforce types.
• Version context files
  Store under VCS (private repo) and review changes like code.
• DRY on overrides
  If many tasks share the same overrides, move them up into the context file.

Quick recipes

A) Promote dev → prod with a single job

1. Upload projectA-prod.properties.
2. In Task setup: set context = prod, link projectA (prod) file.
3. Remove any task overrides. Save. Launch.

B) One task, slightly different endpoint

1. Reuse the shared prod context file.
2. Add one task override: api.base=https://api.alt.example.com.
3. Save. Launch.

C) Audit a failed run

1. Open the execution detail.
2. Confirm Talend env, context file, and overrides used.
3. Compare with expected values; adjust in Task setup or update the context file.

L’article Good use of contexts est apparu en premier sur ETL administration console.