Root PID

The root pid of the job is used to display on the execution task ETL logs, meters and statistics.

Use the following code in a tJava component (but can be any other java componant you prefer):

System.out.println("@@eac@@root_pid="+rootPid);

Specific identifier

This value is « the main information » of the job. For example, you have 2 tasks on one job executed with to different values for a context. Use this option to display the value on the table which list execution task.

In a tJava component (but can be any other java componant you prefer):

System.out.println("@@eac@@identifier="+context.myParam);

L’article Exchange values between ETL and EAC est apparu en premier sur ETL administration console.

Server status

This service must be install on all servers. It retrives memory and cpu usage in order to determine what will be the best server to launch a task.

Copy /var/www/html/current/deployment/systemd/eac-server-status.service file into /etc/systemd/system/ with the following command

sudo cp /var/www/html/current/deployment/systemd/eac-server-status.service /etc/systemd/system/

Edit the file and change ExecStart parameter in order to specify the full path to php according to your installation.

ExecStart=/usr/bin/php /var/www/html/current/bin/console eac:server-status --every=3

You can change option « every« . This value must be set between 1 to 10 (seconds to save memory usage and cpu on database).

Assignment server

This service determines the best servers available for tasks with status « SEND TO QUEUE ». It must be installed only on the main server.

Copy /var/www/html/current/deployment/systemd/eac-execution-assign-server.service file into /etc/systemd/system/ with the following command

sudo cp /var/www/html/current/deployment/systemd/eac-execution-assign-server.service /etc/systemd/system/

Edit the file and change ExecStart parameter in order to specify the full path to php according to your installation.

ExecStart=/usr/bin/php  /var/www/html/current/bin/console eac:execution-assign-server --nextSecondsToAssign=90 --pastSecondsToAssign=60 --frequence=10

You can change options:

• nextSecondsToAssign: assign server to tasks which will executed in the next 90 seconds
• pastSecondsToAssign: assign server to tasks sent to queue and the start date is in the past of 60 seconds
• frequence: to this action all 10 seconds

Consume task

This services consumes / execute the task with status « SEND TO QUEUE ». It must be installed on all execution servers.

Copy /var/www/html/current/deployment/systemd/eac-consume.service file into /etc/systemd/system/ with the following command

sudo cp /var/www/html/current/deployment/systemd/eac-consume.service /etc/systemd/system/

Edit the file and change ExecStart parameter in order to specify the full path to php according to your installation.

ExecStart=/usr/bin/php  /var/www/html/current/bin/console eac:consume --launchForTheNextInSeconds=0 --launchForThePastInSeconds=60

You can change options:

• launchForTheNextInSeconds: execute tasks exactly on time : at 0 second. You can change to 1 if you prefer to execute task one second before
• launchForThePastInSeconds: execute tasks sent to queue and the start date is in the past of 60 seconds

L’article System services est apparu en premier sur ETL administration console.

crontab -e -u eac

1. Status Purge – on main server only

To clean up old memory and cpu usage on servers. We recommended retention of 4 days:

0 0 * * * {path_to_php} /var/www/html/bin/console eac:server-status:purge --days=1 >> /var/www/html/var/log/eac-master-server-status-purge.log 2>&1

Replace {path_to_php} by the full path to php. For example: /usr/bin/php

2. Deployment Verification – on all servers

When a job version is deployed in EAC, each execution servers must retrieve and keep a local copy of that version.
This scheduled verify that every execution server holds the correct copy; if not, EAC shows a visual alert and avoids launching an execution on any out-of-sync server.

0 1 * * * {path_to_php} /var/www/html/bin/console eac:check-jobversion-deployment --force >> /var/www/html/var/log/eac-master-check-jobversion-deployment.log 2>&1

Replace {path_to_php} by the full path to php. For example: /usr/bin/php

3. Check executions are still alive – on all servers

This schedule allow to kill all executions with status « IN PROGRESS » which no longer have any os pid running on the system.

* * * * * {path_to_php} /var/www/html/bin/console eac:check-background-task  --secondsToConsiderOutdatedAndKill=90 --secondsToConsiderOutdatedAndReaffectServer=10 >> /var/www/html/var/log/eac-master-check-background-task.log 2>&1

Replace {path_to_php} by the full path to php. For example: /usr/bin/php

You can adjust options

• secondsToConsiderOutdatedAndKill: reduce or increase the number of seconds after what an execution with the status « SENT TO QUEUE » is considering oudated because on server is available to launch it.
• secondsToConsiderOutdatedAndReaffectServer: reduce or increase the number of seconds to reaffect a task to an available server

4. Enable scheduler – on main server only

If you schedule tasks on EAC you need this to launch task on time.

* * * * * {path_to_php} /var/www/html/bin/console eac:scheduler --planForTheNextInSeconds=90 >> /var/www/html/var/log/eac-master-scheduler.log 2>&1

Replace {path_to_php} by the full path to php. For example: /usr/bin/php

You can adjust option « planForTheNextInSeconds » to provision tasks to be launched more or less than the next 90 seconds.

L’article Planned tasks est apparu en premier sur ETL administration console.

The Workflow feature allows you to automatically trigger one or more tasks after the execution of another task, based on the result status of that execution.
It enables conditional automation between tasks in the same project.

How It Works

After a task execution, you can define rules that specify:

• Status condition – The execution result that will trigger the rule.
  Possible statuses are:
  • Success
  • Warning
  • Error
  • Terminated
• Target task – The task(s) to launch automatically when the selected status condition is met.

Each rule defines a link between a source task and one or more target tasks, allowing complex workflows to be created across the project.

Rule Creation

• Select a status condition (e.g., Success).
• Choose one or several target tasks to execute when that status occurs.
• You can select any task in the project, except the current one.
• Optionally enable or disable the rule using the “Is active” switch.

• Click Add rule to save it.

You cannot create multiple rules with the same combination of status condition and target task for the same task.

If you launch a task with a workflow you’ll be prompted to launch it with the workflow or without it

Multiple Tasks per Status

You can assign several target tasks for the same status, as long as they are different tasks.
Each active rule will trigger independently when the matching status is reached.

Disabled Target Tasks

If a target task is disabled, the workflow will not execute it.
However, it will still appear in the workflow summary table and will be marked as disabled for visibility.

Linked Tasks Overview

At the bottom of the workflow configuration page, two tables are displayed:

• Workflow rules – Lists all rules defined for the current task, grouped by status (e.g., SUCCESS, WARNING, etc.).
• Tasks triggering this task – Shows all other tasks that have configured this one as a target.
  Each entry includes a direct link to the originating task.

Execution Logs and Workflow Tracking

When a task is executed as part of a workflow, the execution logs provide additional details to help you understand what triggered the task and what tasks will be triggered afterwards. This makes it easy to follow the entire chain of executions in a workflow.

Example setup

n the following example, when a task completes with a Success status, two target tasks are defined:

• wait_and_see
• EAC_testContextType_test_4

Trigger Information

At the top of the execution log of a workflow-triggered task, you will see which task caused this execution, under which status, and at what time.

For example:

The current execution was triggered following execution #6028 – EAC_testContextType_test_52, which ended with status SUCCESS at 11:38:41.

Triggered Tasks Table

At the bottom of the execution view of the initial task (the one that finished with the triggering status), a summary table lists all the tasks that have been launched as a result of its execution.

This table includes:

• Task ID and name
• Project
• Job version
• Start and end date
• Duration
• Type and server
• Current execution status

Once all target tasks are finished, you can return to the execution view of the original triggering task to see the completed summary of all triggered tasks.

Example

If task A completes with a Success status, and a workflow rule links it to task B,
then task B will be automatically triggered as soon as A finishes successfully.
If both B and C are defined for Success, both will be triggered in sequence.

Next see Retry auto.

L’article Workflow est apparu en premier sur ETL administration console.

Configuration

In the “Post process” tab, a new Retry auto section has been added.

Here, users can configure:

• Retry count → number of times the system should retry the task
• Delay between retries (in seconds) → waiting time between two retry attempts

If a task ends in ERROR or TERMINATED, it will automatically be retried according to these settings.

Execution Flow

When an execution fails:

• A new retry execution is automatically queued.
• The retry execution keeps the same Identifier but is shown as a new Execution ID.
• The system logs clearly show the relationship between the initial task and its retry executions, including the sleep duration applied based on the configured retry policy.

• On the initiating task, you can see the list of all related executions (the initial one and all retries) at the bottom of the page.

Next see Worflows.

L’article Retry auto est apparu en premier sur ETL administration console.

For paths, parameters, and response schemas, refer to the in-app documentation at the address above.

L’article API est apparu en premier sur ETL administration console.

Run a new job version safely while your scheduled task keeps using the current, stable version.

Prerequisites

• You have a scheduled task already running on version A.
• You’ve deployed job version B that you want to test.
• Role: Admin (to edit tasks and versions).

Steps

1. Pin the current scheduled task to its version

• Go to Task › Task setup of your scheduled task.
• Select the desired Job version = A.
• Uncheck “Follow the new version”.
  This pins the task to version A even if new versions are deployed.

2. Create a separate test task for the new version

• From the Job or Tasks list, create a new task (e.g., MyTask_test).
• In Task setup, choose Job version = B.
• Leave the test task unscheduled (manual runs only), or schedule it off-hours for testing.

3. Run and validate

• Launch the test task manually.
• Check logs, status, and metrics as usual.

4. Promote the new version to production

• When satisfied, go back to the scheduled task:
  • Either re-check “Follow the new version” (so it will pick up version B now and future versions),
  • or explicitly select Job version = B to stay pinned to B.

5. Clean up

• Delete the temporary test task if no longer needed.

Notes & Tips

• Why this works: unchecking “Follow the new version” freezes a task on a specific job version, so your schedule remains stable while you test elsewhere.
• Use a clear naming convention for test tasks (e.g., *_test) and keep them manual to avoid accidental runs.
• If contexts differ between versions, review Task setup › Context before testing.
• Rollback is instant: just switch the scheduled task back to version A (or keep it pinned).

L’article Test a New Job Version Without Impacting the Schedule est apparu en premier sur ETL administration console.

• Fixed batch purge.
• Fixed execution logs and status in specific edge cases.
• No longer sends error emails when a task is in WARNING.
• Excluded Log4J warnings from error reporting.

v1.10.3 — 2025-07-08

• Launched batch purges to better handle large data volumes.

v1.10.2 — 2025-07-07

• Fixed single-server launch.
• Fixed refresh of DB log streams for a task.
• Added CLI check for the PHP binary path.

v1.10.1 — 2025-07-07

• Purge tasks are launched by the best execution server.
• Purge history now uses creationDate instead of modificationDate.

v1.10.0 — 2025-07-04

• Admins can download application logs.
• Execution and purge task logs are stored in the database by default (configurable via APP_TASK_LOG_TYPE).
• Purge mode history keeps the last X days; mode total keeps a fixed number of executions.
• Purges also clean up old purge records, not only old executions.
• EAC-306 Task type labels clarified — exec: Normal execution, purge: Purge past executions.
• Fixed task kill behavior.
• Fixed verification of jobVersion deployment across all execution servers.
• Fixed log generation.
• Fixed scheduler launches when the PHP executable is not in PATH.
• EAC-368 Improved 4xx/5xx error pages.
• EAC-375 Fixed timeline filtering and grouped executions by task.
• EAC-214 If a server hasn’t updated its status for >30s, show a warning on Home, Servers list, and Server detail.
• EAC-365 Server status on the execution detail page updates live while the run is in progress.
• EAC-369 Allow killing multiple executions from the list.
• EAC-345 Added filters (status, scope, etc.) on Tasks and Executions lists.
• CPU and memory charts can be enlarged on click.
• eac:server-status considers active servers only.
• Fixed server deletion.

v1.9.2 — 2025-07-01

• Fixed second-level scheduling for purges.
• Fixed history purge mode filtering that could miss items to delete.

v1.9.1 — 2025-06-30

• EAC-363 When no server is available, redirect to the launched task page.
• EAC-360 Don’t mark EAC as failed when a non-blocking Talend subprocess errors.
• EAC-366 Hide the delete button for job versions that are used by a task.
• EAC-96 ExecutionTask now stores the jobVersion used.
• EAC-251 When deleting a jobVersion, delete related execTasks and warn users beforehand.

v1.9.0 — 2025-06-06

• EAC-96 Consolidated jobVersion at task execution time.
• EAC-360 eac:check-background-task rechecks a task after 5s before killing it.
• EAC-246: If a task/job/project is disabled, nothing related can be launched (API/Scheduler/manual).

v1.8.7 — 2025-05-28

• Fixed a display issue on task detail with uncommon contexts.
• UI performance optimizations: Tasks/Executions tables, Scheduler, Servers list.
• Home page executions list shows errors only by default.
• EAC-343 Disabled accounts can’t sign in.
• Fixed status filtering on PostgreSQL.
• Task detail: hide crontab planning panels when unused.
• Task detail: show the last failed execution.
• EAC-346 Fixed errors on the Users list (PostgreSQL).

v1.8.6 — 2025-05-26

• Optimized Tasks list rendering for large datasets.
• Introduced an internal stateless API for AJAX calls (lower resource usage).

v1.8.5 — 2025-05-20

• EAC-329 Added sidebar scroll when the screen is too small.
• EAC-312 New command to recalculate context typings for older jobs.
• EAC-311 Use context typing to customize task form fields.
• EAC-310 Fixed search on the task-executions table.
• EAC-309 Fixed sorting on the Tasks table.
• Removed stray “dump” in the search bar.

v1.8.4 — 2025-05-12

• EAC-308/309 Fixed defective sorting on the Tasks table.
• EAC-305 Added bulk delete for execution tasks.
• EAC-298 Added switch toggles for activation in tables.
• EAC-297 Reduced table font sizes.
• EAC-287/288 Fixed wrong default sort for ExecutionTask.
• EAC-270 Standardized date formats (EN/FR).

v1.8.3 — 2025-05-06

• EAC-307 Recomputed MD5 for job versions in addition to checking versions deployed on slaves.

v1.8.2 — 2025-04-25

• API: added hasExecutionInProgress.
• API: added /api/version to check API availability.
• Added a URL to open Task detail by name (not only by ID).
• Restored seconds in execution lists.
• Fixed a scheduling anomaly.

v1.8.1 — 2025-04-23

• Scheduler supports second-based ranges like 10-59/20 (not just */20).
• Scheduling expressions are validated and highlighted when invalid.
• EAC-293 Crontab checks use the user from EAC_CRON_USER.
• EAC-272 No need to comment an AppExtension line during installs.
• EAC-303 Project’s jobVersion list is visible again.

v1.8.0 — 2025-04-15

• EAC-135 Migrated execution logs UI to StimulusJS (from jQuery).
• EAC-80 Added validation on the task name field.
• EAC-122 Display context params in a table (collapsible when multiple).
• EAC-214 Show a warning on Servers page if a server has no update for >30s.
• EAC-227 Syntax highlighting for context files in comments + auto-resize textarea.
• EAC-239 Info bubble on uploaded zips list.
• EAC-240 Select/Deselect all uploads on the deployment page.
• EAC-261/258 Show screenshots of jobs/sub-jobs in JobVersion detail; full-screen on click.
• EAC-264 Option to hide “null value affected” in logs.
• EAC-262 Icon refresh.
• EAC-260 Validate context variables before launching a task.
• EAC-242 Info bubble on Project pages.
• EAC-243 Added Project column on Tasks/Jobs/Executions lists when multiple projects exist.
• EAC-248 Horizontal scroll for long contexts on the Tasks page.
• EAC-249 Show roles on the Users page.
• EAC-244 Improved breadcrumbs site-wide.
• EAC-253 Periodicity now includes seconds and never shows empty.
• EAC-268 Live refresh (logs/stats/etc.) on execution detail no longer breaks on server errors.
• EAC-216 Server statistics are more consistent and update correctly.
• EAC-156 Launch a purge from task detail.
• EAC-179 Executions list correctly filters the last X days (aligned with the home pie chart).
• Tooltips added to all actions; colors aligned with the logo.
• Show poorly deployed versions only on execution servers.

v1.7.3 — 2025-03-26

• Don’t display poorly deployed job versions with single-server license.
• EAC-215 Can disable GIT status check in eac:server-status.
• EAC-217 Scheduler doesn’t relaunch too-old tasks on resume.
• EAC-223 Purge tasks can be scheduled and no longer use the task’s contexts.
• EAC-224 Emphasize context values from the context file in task settings.

v1.7.2 — 2025-03-24

• Hide the Logs tab for purge tasks.
• Added OS PID for purge tasks.
• Added copy task action.

v1.7.1 — 2025-03-12

• Post-v1.7.0 fixes for SQL Server.
• EAC-221 Scheduler no longer controls purge tasks.

v1.7.0 — 2025-02-24

• Multi-server support.

v1.6.2 — 2025-01-22

• Fixed JSON error responses from the API.
• Fixed purge when the log file is missing.
• Fixed purge when history option wasn’t applied.

v1.6.1 — 2024-09-11

• EAC-166 Added a Makefile for development.
• EAC-163 New command eac:check-background-task — verifies background executions; otherwise stops them and sends an email.
• EAC-164 Reworked pagination, sorting & search for ETL logs, metrics and stats tables.

v1.6.0 — 2024-07-11

• EAC-24 Added API to start/stop tasks (docs at /api/doc).

v1.5.4 — 2024-07-08

• EAC-161 Prevent password managers from auto-filling context credentials on task edit.
• EAC-162 No default start date in the tasks listing.

v1.5.3 — 2024-06-19

• EAC-149 List unscheduled tasks.
• EAC-155 Check that the scheduler is present in the server crontab.
• EAC-157 After editing a schedule from the Scheduler view, return to it (not to task edit).

v1.5.2 — 2024-06-03

• EAC-142 Don’t disable the scheduler on each EAC update.
• EAC-158 Fixed scheduler end date handling.
• EAC-159 Scheduler now launches purge tasks correctly.
• EAC-160 Purge tasks are logged in EAC.

v1.5.1 — 2024-05-13

• EAC-150 Include duration-less tasks in purges; fix end date when launch aborted due to disallowed parallelism.
• EAC-154 A killed task becomes WARNING (was stuck on killed).
• EAC-152 Task execution list is no longer a dynamic table (filter/pagination/sort removed).
• EAC-153 Fixed missing task-launch dropdown for jobs with multiple tasks.

v1.5.0 — 2024-04-17

• EAC-18 Internal scheduler (php bin/console eac:scheduler) + ability to stop it from the UI.
• EAC-112 A disabled task cannot be launched.
• EAC-78 Added Allow parallel launch option on tasks.
• EAC-73 Show Talend stats/logs/metrics tables.
• EAC-114 Fixed reading JVM params at deployment.
• EAC-115 Preserve active menu after deployment.
• EAC-116 “Upload” becomes “Upload & Deploy” when requested.
• EAC-113 Remove temporary folder after unzip.
• EAC-118 New tasks default to Email on error only.
• EAC-81 Add a context file from the Project or the Context files view.
• EAC-130 Tabs update browser history.
• EAC-137 New tasks are active by default.
• EAC-119 Added /check-requirements.php page.
• EAC-138 Fixed special characters in ETL launch feedback.
• Limit ExecutionTask lists to 500 items max to avoid memory issues.
• Added Support page.

v1.4.1 — 2024-03-13

• EAC-77 Allow overriding JVM values (Xmx, Xms, -XX:MaxPermSize, -XX:PermSize); run php bin/console eac:jobversioninitjvmparam at deployment.
• EAC-101 Show job description under the task form title.
• Show Project and Job in task breadcrumbs.
• Restyled Task edit form.
• EAC-87 User form redesign + delete action.
• EAC-88 If no password is provided on user edit, keep the existing one.
• EAC-98 Show Ack button only when relevant.
• EAC-99 Minor spacing fix on deployment page.

v1.4.0 — 2024-03-11

• EAC-62 Upgrade to Symfony 7 (PHP ≥ 8.2).
• EAC-86 Better rendering of active contexts.
• EAC-85 Show only distinct flash messages.
• EAC-91 Changed ETL execution method (removed Messenger/supervisord dependency).
• EAC-95 Link to download an archive from uploads.
• EAC-97 Visual loading indicator during archive upload.

v1.3.0 — 2024-01-31

• EAC-10 Licensing.
• EAC-39 Migrated SQLite → MySQL.
• EAC-64 Two DB connections: core EAC entities vs. Talend log entities.
• EAC-34 Upload workflow includes auto-deployment.
• EAC-68 Show current product version.
• EAC-69 Show the meaning of the EAC acronym.

v1.2.0 — 2024-01-22

• EAC-59 Include execution errors in notification emails.
• EAC-60 Fixed sorting on Tasks + Executions tables.
• EAC-61 Visual improvements to execution logs (dates toned down, non-selectable; highlight command-generated logs).
• EAC-33 Allow multi-delete of uploads.
• EAC-63 In Task › Additional information, display current rather than the release name.

v1.1.0 — 2024-01

• EAC-36 After deleting a jobVersion (from Jobs view), redirect to the Job (not the jobVersion).
• EAC-17 Added executions timeline.
• Licensing system.
• EAC-35 Version tracking page.
• EAC-30 Visual distinction of environments.
• EAC-18 Human-readable crontab schedules.
• EAC-37 Date-sorting enabled on tables.
• EAC-32 Show size of uploaded ZIPs.
• EAC-21 Fixed missing logs issue.

v1.0.0 — 2023-09

• Log visualization.
• Job deployment.
• Job versioning.
• Start/stop tasks.

L’article Release notes est apparu en premier sur ETL administration console.

Admin (Role)
Full access to configure EAC: users/roles, jobs, tasks, servers, environments, purge policies, and settings. (See: API Role)

API (Role)
Programmatic access via tokens to trigger tasks and read execution data within scope; cannot alter global settings or users. (See: API Token)

API Token
Credential used by the API role to authenticate requests (e.g., start a task, query execution status). Keep tokens secret. (See: API Role)

APP_DEFAULT_URI
Base URL of the application used in links and callbacks. Configured in Settings (.env.local). (See: Settings vs .env)

APP_SERVER_NAME
Unique server name set in .env.local and mirrored in the Server definition in the UI. Used to match runtime hosts to configured servers.

APP_TASK_LOG_TYPE
Where task logs are stored: file or db. Affects visibility, retention, and purge mechanics. (See: ETL Logs)

Averaged Executions (EXEC_TASK_AVG_COUNT)
Number of recent runs used to compute average durations/progress estimates.

Context (Parameters)
Key–value pairs injected at runtime (e.g., database, dates, flags). Defined in Task Setup and/or Context Files. (See: Context File)

Context File
A versioned context configuration bundled with a JobVersion (can differ by environment). UI displays a matrix per context.

Cron / Schedule / Periodicity
Timing expression for scheduled tasks (supports seconds). Validated in UI; shown in human-readable form. (See: Scheduler)

Cron User (EAC_CRON_USER)
System user under which cron jobs are checked/executed (e.g., for eac:scheduler).

DATABASE_URL / DATABASE_URL_ETL_LOG
Connection strings for the core app DB and (optionally) a separate ETL logs DB. Credentials may require URL percent-encoding. (See: Percent-Encoding)

Deployment (JobVersion)
The process of making a packaged ETL job available to EAC servers. EAC never overwrites previous versions. (See: JobVersion)

Deployment Directory (APP_DEPLOY_DIR)
Local path on each server where job versions are stored.

Execution (Run)
A single run of a Task. Has an ID, timestamps, duration, type (Manual/Scheduled/API), server, job version, and status. (See: Task)

Execution ID
Unique identifier of a run; used to fetch logs/status and to stop/kill.

Execution Server (“Slave”)
A worker host that runs tasks. EAC schedules to a healthy server that has the correct job version locally. (See: Server Status, Rsync)

Execution Status
IN_PROGRESS, SUCCESS, WARNING, ERROR, KILLED, Acked, Not Acked. Drives colors and alerts across UI.

Follow the New Version (Task Setting)
When enabled, a task tracks the latest deployed JobVersion. When disabled, the task is pinned to a specific version. (See: Pinning)

Heartbeat / Server Status
Periodic metrics EAC collects from each execution server (CPU, memory, reachability, etc.). Used for scheduling decisions and UI health. (See: Server Status Purge)

History Purge Mode
Retention mode that keeps executions for the last X days. (See: Purge Policy)

Identifier (Execution Identifier)
Optional business key passed when starting a run (e.g., batch ID); appears in lists and helps correlation.

Job
A logical ETL artifact (e.g., Talend job) that can have multiple Tasks and multiple JobVersions. (See: Task, JobVersion)

JobVersion
A specific compiled/deployed version of a Job (e.g., 1.10.0, 0.1). Includes context files and optional screenshots of sub-tasks.

JVM Parameters (Xmx/Xms, …)
Java runtime flags stored per JobVersion and editable via commands/UI to control memory and tuning.

Kill (Execution)
Force-stop a running execution. Prefer Stop when available; kill is a last resort.

Log Types (ETL)

• Talend Logs: application logs (ETL_LOG_TABLE_NAME)
• Talend Stats: per-run stats (ETL_STAT_TABLE_NAME)
• Talend Metrics: metrics (ETL_MET_TABLE_NAME)
  Configured via Settings; require proper indexing and retention. (See: ETL Logs)

MD5 (JobVersion Hash)
Checksum used to verify deployment consistency across servers during replication checks.

Percent-Encoding (URL)
Encode special chars in usernames/passwords inside URLs: @→%40, :→%3A, /→%2F, &→%26, %→%25, +→%2B, #→%23, ?→%3F. (Encode credentials only.)

Pinning (Task → JobVersion)
Explicitly selecting a JobVersion in Task Setup and unchecking Follow the new version to prevent automatic upgrades.

Project
Top-level grouping of Jobs/Tasks. Shown in breadcrumbs and list columns when multiple projects exist.

Purge (Task)
A task that deletes old executions/logs according to a Purge Policy. Can be scheduled and monitored like any task.

Purge Policy
Retention strategy for executions/status/log tables (e.g., history X days vs total fixed count). (See: Server Status Purge)

Reschedule Old Tasks After X Minutes
When the scheduler restarts, decides if (and how far back) missed schedules should be re-queued.

Rsync (Replication Service)
File sync mechanism used to distribute JobVersions to all execution servers. Checked by eac:check-jobversion-deployment. (See: Deployment Verification)

Scheduler
Internal process (CLI command) that launches scheduled tasks per their cron expressions; can be started/stopped from the UI.

Scope
Logical partition (e.g., environment/project/tenant) used for filtering, scheduling, and API calls.

Server Code (EAC_SERVER_CODE)
Identifier for the current instance/role (e.g., master). Displayed in UI and logs.

Server Status Purge
Scheduled task that deletes old per-minute server statuses (e.g., keep 4 days). Only the last minute is required for scheduling; older data is used only for historical views.

Settings vs .env
Settings page writes to .env.local (overrides). .env contains committed defaults. Real OS env vars have the highest precedence. Never commit secrets.

Shell Command Path (PHP_PATH_COMMAND)
CLI path for PHP used by EAC to run console commands.

Stop (Execution)
Graceful request to end a run. May escalate to Kill if the process doesn’t stop.

Task
Executable wrapper around a JobVersion with its schedule, contexts, and runtime options. A task can have many Executions over time.

Task Copy
Utility to duplicate a task’s configuration (contexts, schedule, options) as a starting point.

Task Setup
Configuration screen where you select the JobVersion, contexts, schedule, parallelism, mail options, etc.

Timeline (Executions Timeline)
Visualization of executions over time (default last 24h); supports hover tooltips, grouping by task, and filters.

Total Purge Mode
Retention mode that keeps only the last N executions, regardless of age.

Upload & Deploy
UI flow to upload an archive and deploy it as a new JobVersion (with automatic post-deploy checks).

User Roles

• Admin — full configuration/control
• API — automation: trigger/read only, scoped by token

Warning (Status)
Non-fatal condition during execution; does not trigger error emails when configured accordingly.

L’article Glossary est apparu en premier sur ETL administration console.

Quick wins (checklist)

• Configure scheduled purges for executions and logs.
• Watch Execution Server graphs (CPU %, Memory %) and error KPIs.
• Keep scheduling enabled and tune reschedule delay.
• Run server status sampler every minute and purge old statuses daily.
• Keep ETL log tables small via retention + DB maintenance.

Execution & log retention

Each run writes an execution record and logs. Without retention, UI lists and dashboards slow down and DB/files grow indefinitely.

Strategies

• History window (recommended): keep the last 30–90 days per task.
• Fixed count: keep the last 200–1000 executions per task.
• Longest executions: keep 10–50 longest for performance analysis.

Set these via:

• Manual purge: Tasks → {Task} → Last executions → Purge the executions of the task
• Scheduled purge: Tasks → {Task} → Cron planification → Add scheduler → Purge past executions
  Modes: date, history, total, duration.

See the Purges page for details and examples.

Monitor servers

Open Monitoring to validate:

• Each node sends heartbeats (CPU/Memory sparklines not flat at 0%).
• Error Execution gauges (day/week/month) stay near zero.
• The master and any slaves show “Last status received … seconds/minutes ago”.

Tip: if “hours ago”, your sampler isn’t running—see below.

Screenshot placeholders:

• ![Servers dashboard – CPU/Memory](path/to/monitoring-servers.png)
• ![Server detail](path/to/monitoring-server-detail.png)

Scheduling hygiene

In Settings → Scheduling & System:

• Enable scheduling: true.
• Reschedule old task after X (min): start with 10; raise to 15–30 on busy systems.
• Keep Cron user set to your service account (e.g. eac) and ensure directories are writable.

Avoid enabling parallel launch unless the ETL job is idempotent and designed for concurrency.

Server status sampler & purge

EAC samples node CPU/Memory via a console command and stores recent points for graphs.

Sampler (every minute)

Run on every EAC node:

php bin/console eac:server-status --every=3 --env=prod

Cron example (bare metal/VM)

* * * * * cd /var/www/html/eac && /usr/bin/php bin/console eac:server-status --every=3 --env=prod >/dev/null 2>&1

Daily purge of old samples (required)

Removes stale status rows to keep graphs snappy:

php bin/console eac:server-status:purge --days=4 --env=prod

Cron (daily at 02:10)

10 2 * * * cd /var/www/html/eac && /usr/bin/php bin/console eac:server-status:purge --days=4 --env=prod >/dev/null 2>&1

Systemd timer (optional)

# /etc/systemd/system/eac-server-status.timer
[Timer]
OnCalendar=*:0/1
Persistent=true

# /etc/systemd/system/eac-server-status-purge.timer
[Timer]
OnCalendar=daily
Persistent=true

Keep 4–7 days of server statuses; more provides no benefit and grows storage.

ETL logging footprint

If ETL logs are enabled (APP_ENABLE_ETL_LOGS=true):

Retention. Purge old execution records on a schedule; keep only what you need for ops & audits.
Scope. Apply purges to ETL_STAT_TABLE_NAME, ETL_MET_TABLE_NAME, and ETL_LOG_TABLE_NAME.
Performance. Add indexes on the fields you filter by most (start_time, job, status, context).

— PostgreSQL (rename tables/columns to match your schema)

CREATE INDEX IF NOT EXISTS idx_stat_start  ON talend_stat(start_time);
CREATE INDEX IF NOT EXISTS idx_stat_job    ON talend_stat(job);
CREATE INDEX IF NOT EXISTS idx_stat_status ON talend_stat(status);

Log backend choice

Settings → Task logs type:

• db: fastest for searching/paging inside EAC; increases DB size.
• file: reduces DB size; ensure the log directory is on fast storage and rotated by purges.

Runtime tips

• Keep Symfony in prod mode: APP_ENV=prod, APP_DEBUG=0; warmup cache after deploy: php bin/console cache:clear --env=prod && php bin/console cache:warmup --env=prod
• Disable dev extensions (Xdebug, verbose loggers) on production nodes.
• JVM sizing for Talend jobs: keep -Xms small, -Xmx appropriate to dataset; avoid overcommitting memory.

L’article Performances est apparu en premier sur ETL administration console.