8.10. Pause pipeline after more than x reports per minute

[96]A sudden spike in errors might just be a temporary outage of the network, for example. In this case it is helpful to pause the pipeline after x reports per minute. If the errors persist, pause the pipeline until user action. Then an email is sent to xyz@company.de).

Some details:

  • The function can be activated for service and local dashboard separately

    (Active or ActiveInLocalDashboard)

  • Maximal number of failed report items per minute which triggers the pause can be configured via config key.

    (MaxErrorsPerMinute)

  • Duration of the pause(s) is set in minutes. Multiple pauses possible (e.g. PauseTime0=1, PauseTime1=2, ...)

  • Key 'UserResumeAsFinalPause' as option => Pipeline will be paused until a user resumes it

  • A pipeline which was resumed manually once, will not be automatically paused again

  • E-mail notification is possible (only in service, not local). E-mail notification only for pipelines which are 'paused until user resume'

    Example:

    Info mail

    Info mail

  • Dashboard shows 'automatically paused' state. Contains also tooltips with more details.

    Message: Job automatically resumed

    Message: Job automatically resumed

    Message: Job not automatically resumed

    Message: Job not automatically resumed

Configuration:

If the mail notification shall be activated than the mail server needs to be configured in $CADENAS_SETUP/reportstask.cfg.

[ReportsMailServer]
#:HELP;Hostname of mailserver. E.g.: hostname, hostname:port, smtp://<username>
:<password>@<hostname>:<port>?key1=val&key2=val, ssmtp://<username>
 :<password>@<hostname>:<port>?key1=val&key2=val
Host=
#:HELP;UserName used for authentication to the mailserver (if any)
UserName=
#:HELP;Password used for authentication to the mailserver (if any)
Password=
#:HELP;Set to true if the password is encrypted (use cnstools.helpfu to encrypt password).
 Default = false
PasswordEncrypted=

=> These settings will also be used for the ReportsTask Mail configuration. This way the mail server only needs to be configured once.

The specific settings need to be made in $CADENAS_SETUP/partwarehouse/pipeline.cfg.

[ReportWatcher]
#:Help;Activates the ReportWatcher (allows to pause pipelines if there are a lot of errors 
 in a short period of time) (default is false)
Active=
#:Help;Activates the ReportWatcher also in the LocalDashboard environment (default is false)
ActiveInLocalDashboard=
#:Help;Threshold for max errors per minute. If exceeded the pipeline is paused (default is 60)
MaxErrorsPerMinute=
#:Help;List of timeouts (in minutes). First value is used the first time a pipeline is paused, 
 the second one is used the second time the same pipeline is used, etc.
PauseTime0=
#PauseTime1=
#:Help;If true, then the pipeline needs to be resumed manual (After last PauseTime was used). 
 Only in service, not in LocalDashboard (default is false)
UserResumeAsFinalPause=
#:Help;Should an email be send when a pipeline was paused and user resume is required. Also
 see ReportWatcherMail block (default is false). If true then Mailer has to be configured.
SendMail=

Multiple timeouts: A pipeline can be paused multiple times (up to 5):

E.g.: PauseTime0=1 and PauseTime1=2: Then the first time the pipeline will be paused for one minute and the second time the pipeline will be paused for 2 minutes.

UserResumeAsFinalPause: If true then the user will have to manually resume the pipeline during the last pause and it will not be resumed automatically.

Mailer specific options for the ReportWatcher (also in $CADENAS_SETUP/partwarehouse/pipeline.cfg) ... only needs to be configured if mails should be send.

[ReportWatcherMailer]
#:HELP;The mail address which should be shown as sender of the mail
Sender=
#:HELP;Comma separated list of mail addresses which should be set as mail receivers
Receivers=
#:HELP;Sender Envelope of the mail
SenderEnvelope=
#:HELP;Define 'ReplyTo' mail address (optional)
ReplyTo=
#:HELP;Comma separated list of mail addresses which should be used as CC (optional)
CcList=
#:HELP;Comma separated list of mail addresses which should be used as BCC (optional)
BccList=
#:HELP;Subject of the outgoing mail
Subject=




[96] #132123