The overview of the Workflow Run Engine component

By Jianwu Wang & Jing Tao

1. Introduction

The Workflow Run Engine is to able to execute workflows remotely. It supports SOAP-based and RESTful invocation in a synchronous or asynchronous way. The Workflow Run Engine was implemented in two stages. This component can be downloaded from the Kepler trunk repository. Note: Stage one APIs are not included in the Kepler-Run-Engine-2.3.0 release.

Stage one has 19 interfaces and mainly handles MoML files. On this stage, all the display related actors are replaced automatically and implicitly before execution since the Workflow Run Engine executes workflows without GUI. Each display related actor will correspond with one or more output files of the Workflow Run Engine execution. For some display related actors, such as ImageDisplay and ImageJ, they may have different display contents for multiple executions. Therefore, there should be multiple corresponding ouput files. Timestamps are added to the names of output files for the same display actor so previous files are not overridden. 

Stage two has 2 interfaces and handles the Kepler kar file format. The Workflow Run Engine executes a kar file and can send the result as a run-kar file to a remote repository. Here is an overview of Workflow Run Engine in stage two:

The module is divided into two parts: the workflow-run-engine and workflow-run-engine-client. Each of them is a standard Eclipse project and has a build.xml.

The Workflow Run Engine can support user authentication/authorization by configuration(see 7.Security part for more information).

The Workflow Run Engine has been used in some projects successfully to decouple back-end workflow execution and frond-end interfaces, such as for Web UI and TPCs (Threshold for Potential Concern -- for the SANParks project).

2 Interfaces on the stage one

Interfaces 1-6 are mainly used for short-term (usually less than 1000 seconds) workflow execution; interfaces 7-17 are mainly used for long-term workflow execution; interfaces 18-19 are assistant operations.

Interface1: public KeplerWfOutput[] executeByURI(String wfURI) throws Exception;

This function takes a URL specifying the model to execute.

@param modelURL

The URL of Kepler workflow MoML file.

@return Output array

Each element is a <i>KeplerWfOutput<i> object. Each object responds to one display actor of the workflow. The content of the <i>KeplerWfOutput<i> object includes the display file name and its URL

@exception Exception

If unable to parse the URL or failed to execute the model

Interface2: public KeplerWfOutput[] executeByURIWithPara(String wfURI, KeplerWfParameter[] wfParas) throws Exception;

This function takes a URL specifying the model to execute and workflow parameters that can be configured.

@param wfURI

The URL of Kepler workflow MoML file.

@param wfParas

The array of KeplerWfParameter objects, each object has two attributes: name and value.

@return Output array

Each element is a <i>KeplerWfOutput<i> object. each object responds to one display actor of the workflow. the content of <i>KeplerWfOutput<i> object includes the display file name and its URL

@exception Exception

If unable to parse the URL or failed to execute the model.

Interface3: public KeplerWfOutput[] executeByContent(String wfContent) throws Exception;

This function executes workflow by providing workflow content, namely the string of MoML file.

@param wfContent

The XML content of Kepler workflow MoML file.

@return Output array

each element is a <i>KeplerWfOutput<i> object. Each object responds to one display actor of the workflow. The content of <i>KeplerWfOutput<i> object includes the display file name and its URL

@exception Exception

If failed to execute the model.

Interface4: public KeplerWfOutput[] executeByContentWithPara(String wfContent, KeplerWfParameter[] wfParas) throws Exception;

This function executes workflow by providing workflow content, namely the string of MoML file. The workflow parameters can be configured.

@param wfContent

The XML content of Kepler workflow MoML file.

@param wfParas

The array of KeplerWfParameter objects, each object has two attributes: name and value.

@return Output array

each element is a <i>KeplerWfOutput<i> object. Each object responds to one display actor of the workflow. The content of <i>KeplerWfOutput<i> object includes the display file name and its URL

@exception Exception

If failed to execute the model.

Interface5: public OMElement executeByAttach(OMElement inEle) throws Exception;

This function attaches the workflow Moml file for execution and attach output files back for response.

@param inEle

The message for invoking this operation, which includes workflow name (wfName) and workflow attachment ID (wfAttachID). How to get attachment ID needs special interface of AXIS2 MessageContext, see function:callKeplerWSByAttachment@KeplerExeWSClient.java

@return OMElement

The response message of invoking this operation, which includes output file name (fileName) and attachmentID for the attached output file. How to get attached files from attachment ID needs special interface of AXIS2 MessageContext, see KeplerExeWSClient.java. Notes, the response may have multiple attached files.

@exception Exception

If failed to parse input message or execute the model.

Interface6: public OMElement executeByAttachWithPara(OMElement inEle) throws Exception;

This function attaches the workflow Moml file for execution, the workflow parameters can be configured the output files will be attached back in response message.

@param inEle

The message for invoking this operation, which includes workflow name (wfName), configured parameters (wfParas) and workflow attachment ID (wfAttachID). How to get attachment ID needs special interface of AXIS2 MessageContext, see function:callKeplerWSByAttachmentWithPara@KeplerExeWSClient.java

@return OMElement

Response message of invoking this operation, which includes output file name (fileName), and attachmentID for the attached output file. How to get attached files from attachment ID needs special interface of AXIS2 MessageContext, see KeplerExeWSClient.java. Notes, the response may have multiple attached files.

@exception Exception

If failed to parse input message or execute the model.

 

**************The following interfaces are useful for long-term workflow execution.*******************

Interface7: public String startExeByURI(String wfURI) throws FileNotFoundException, Exception;

This function start the execution of a workflow by providing its URI.

@param wfURI

The URI of Kepler workflow MoML file. The following 3 formats are supported:

1) URL: e.g. http://132.249.64.206:8080/KeplerWf/KeplerWfUploaded/04-HelloWorld.xml,

file:///D:/06-WebServicesAndDataTransformation.xml

2) absolute path in the server which the WS is deployed: e.g. D:/06-WebServicesAndDataTransformation.xml

3) path relative to the system.dir in the server which the WS is deployed: e.g. ../06-WebServicesAndDataTransformation.xml

@return WfID, the unique ID of the workflow generated when it is executed.

@exception Exception

If can not parse the URI or failed to execute the model.

Interface8: public String startExeByContent(String wfContent) throws Exception ;

This function start the execution of a workflow by providing its content, namely the string of MoML file.

@param wfContent

The XML content of Kepler workflow MoML file.

@return WfID, the unique ID of the workflow generated when it is executed.

@exception Exception

If failed to execute the model.

Interfaces9: String startExeByContentWithPara(String wfContent, KeplerWfParameter[] wfParas) throws Exception;

This function start the execution of a workflow by providing its content, namely the string of MoML file. The parameters of this workflow can be configured.

@param wfContent

The XML content of Kepler workflow MoML file.

@param wfParas

The array of KeplerWfParameter objects, each object has two attributes:

name and value.

@return WfID, the unique ID of the workflow generated when it is executed.

@exception Exception

If failed to execute the model.

Interfaces10: OMElement startExeByAttach(OMElement inEle) throws Exception;

This function start execution of a workflow by attaching the workflow Moml file.

@param inEle

The message for invoking this operation, which includes

workflow name (wfName) and workflow attachment ID (wfAttachID).

how to get attachment ID needs special interface of AXIS2 MessageContext,

see function:callKeplerWSByAttachment@KeplerExeWSClient.java

@return OMElement

The response message of invoking this operation, which includes workflow ID.

The ID is generated when the workflow is executed.

@exception Exception

If failed to parse input message or execute the model.

Interfaces11: OMElement startExeByAttachWithPara(OMElement inEle) throws Exception;

This function start execution of a workflow by attaching the workflow Moml file and needed input files.

The parameters of this workflow can be configured before execution.

@param inEle

The message for invoking this operation, which includes

workflow name (wfName), input file names (inputFileName), configured parameters (wfParas),

workflow attachment ID (wfAttachID) and input file attachment IDs.

How to get attachment ID needs special interface of AXIS2 MessageContext,

see function:startExeByAttachWithPara@KeplerExeWSClient.java

@return OMElement

The response message of invoking this operation, which includes workflow ID.

The ID is generated when the workflow is executed.

@exception Exception

If failed to parse input message or execute the model.

Interfaces12: String startExeByURIWithPara(String wfURI, KeplerWfParameter[] wfParas) throws Exception;

This function start the execution of a workflow by providing its URI. The parameters of this workflow can be configured.

@param wfURI

The URI of Kepler workflow MoML file. The following 3 formats are supported:

1) URL: e.g. http://132.249.64.206:8080/KeplerWf/KeplerWfUploaded/04-HelloWorld.xml,

file:///D:/06-WebServicesAndDataTransformation.xml

2) absolute path in the server which the WS is deployed: e.g. D:/06-WebServicesAndDataTransformation.xml

3) path relative to the system.dir in the server which the WS is deployed: e.g. ../06-WebServicesAndDataTransformation.xml

@param wfParas

The array of KeplerWfParameter objects, each object has two attributes:

name and value.

@return WfID, the unique ID of the workflow generated when it is executed.

@exception Exception

If can not parse the URI or failed to execute the model.

Interfaces13: boolean stopExecution(String wfID) throws Exception;

stop execution of request workflow ID.

@param wfID

The ID of the Kepler workflow MoML file, which is generated when the workflow is sumbitted for execution.

@return boolean

return true if stop correctly, false if not correctly.

@exception FileNotFoundException, Exception

If failed to get file info of outputs.

Interfaces14: boolean pauseExecution(String wfID) throws Exception;

pause execution of request workflow ID.

@param wfID

The ID of the Kepler workflow MoML file, which is generated when the workflow is sumbitted for execution.

@return boolean

return true if pause correctly, false if not correctly.

@exception FileNotFoundException, Exception

If failed to get file info of outputs.

Interfaces15: boolean resumeExecution(String wfID) throws Exception;

resume execution of request workflow ID.

@param wfID

The ID of the Kepler workflow MoML file, which is generated when the workflow is sumbitted for execution.

@return boolean

return true if resume correctly, false if not correctly.

@exception FileNotFoundException, Exception

If failed to get file info of outputs.

Interfaces16: String getExecutionStatus(String wfID) throws Exception;

Get execution status according to the workflow ID.

@param wfID

The ID of the Kepler workflow MoML file, which is generated when the workflow is sumbitted for execution.

@return Execution status, which is one status among {corrupted, idle, initializing, executing, pausing execution, pausing execution on a breakpoint, preinitializing, resolving types, throwing a throwable, wrapping up, exiting}.

see detailed info of these status at ptolemy.actor.Manager.

If there is no record of request wfID in managerTable, will return null.

@exception Exception

If failed to get file info of outputs.

Interfaces17: KeplerWfOutput[] getResults(String wfID) throws FileNotFoundException, Exception ;

Get execution outputs (array of KeplerWfOuput object) according to the workflow ID.

This operation can also be used to check whether there has been intermediary outputs according to the workflow ID.

They are useful for long-term execution, yet the outputs may be not complete.

@param wfID

The ID of the Kepler workflow MoML file, which is generated when the workflow is sumbitted for execution.

@return KeplerWfOutput[]

array of KeplerWfOuput object (including file name and file URL)

@exception FileNotFoundException, Exception

If failed to get file info of outputs.

 

**************The following interfaces are assistant operations.*******************

Interfaces18: OMElement uploadFilesByAttach(OMElement inEle) throws Exception;

This function attaches files to AXIS2 server, which files are usually the input files of a certain workflow.

@param inEle

The message for invoking this operation, which includes

paths of the files to be uploaded.

how to get attachment ID needs special interface of AXIS2 MessageContext,

see function:uploadFilesByAttach@KeplerExeWSClient.java

@return OMElement

The response message of invoking this operation, which includes the address that store the uploaded files.

@exception Exception

If failed.

Interfaces19: boolean delLogDir(String wfID) throws FileNotFoundException, Exception;

This function delete log dir according to the workflow ID.

@param wfID

The ID of the Kepler workflow MoML file, which is generated when the workflow is sumbitted for execution.

@return boolean

if the directory exists and is deleted, retrun true; otherwise return false.

@exception FileNotFoundException, Exception

If failed to get file info of outputs.

3. Interfaces on the stage two:

Interface20: String execute(String karLSID) throws Exception;

This interface starts an execution a workflow with the given kar file LISD

@param karLSID the LISD of the kar file

@return workflow run id of this exectuion

@exception Exception

If failed.

Interface21: String getStatus(String workflowRunID);

This interface checks the status of execution for the given workflow run id. Status includes EXECUTING, SUCCEEDED, FAILED and NOTFOUND

@param workflowRunID

@return status of the workflow run

Note: this interface has not been implemented.

4. Deployment

1) Infrastructure:

Currently, the Workflow Run Engine is deployed in axis2 (http://ws.apache.org/axis2/) version 1.4 or upper which is a Workflow Run Engine container, and axis2 is deployed in Tomcat (http://tomcat.apache.org/) version 5.5.26 or upper which is a Web server. We also use Ant (http://ant.apache.org/) to build Workflow Run Engine module for the source. Ant is not necessary but recommended so that you can easily execute related tasks, e.g. service deploy and client invocation.

2) Build the workfow run engine from the source:

a) Check out the workflow run engine trunk code located in SVN at https://code.kepler-project.org/code/kepler/trunk/modules/kepler-run-engine/workflow-run-engine/

b) In the root directory, run "ant cleanAll war".

c) The workfowrunengine.war will be in the build directory.

3) Or download kepler-run-engine-bin-version.zip or kepler-run-engine-bin-version.tar.gz from https://code.kepler-project.org/code/kepler/releases/installers/:

a) Unzip kepler-run-engine-bin-version.zip or kepler-run-engine-bin-version.tar.gz.

b) workfowrunengine.war file will be in kepler-run-engine directory.

4) Deploy the Workflow Run Engine:

Copy the workflowscheduler.war file to the tomcat webapp directory on the target server

5) Restart:

Restart the tomcat instance for the workflow run engine.

5. Configuration

After completed the deployment steps, the workflow run engine will be running with the default configuration. If you want modify the default configuration, wfWSProps.properties in workflowrunengineWEB-INF/classes/ is the file needed be edited. The main variables need to be configured are:

"allowedRunningUser" sets the users who can execute the workflow run engine. The names are separated by ":". See 7.Security part for more information.