taskgraphs.executor
cloud.taskgraphs.executor
Generic interfaces for task graph executors.
Attributes
| Name | Description |
|---|---|
| GraphStructure | The structure of a task graph, as the JSON serialization or a Builder. |
Classes
| Name | Description |
|---|---|
| Executor | An interface allowing for execution and management of a task graph. |
| Node | An abstract type specifying the operations on a Node of a task graph. |
| ParentFailedError | Raised when the parent of a Node fails. |
| Status | The current status of a Node (or a graph). |
Executor
cloud.taskgraphs.executor.Executor(graph)An interface allowing for execution and management of a task graph.
This is the basic interface fulfilled by any task graph executor. While some implementations may provide more control, these operations, to the extent that they are supported, are universal across task graph implementations.
Attributes
| Name | Description |
|---|---|
| server_graph_uuid | The UUID of this execution’s log as returned by the server. |
| status | The status of the entire graph. |
Methods
| Name | Description |
|---|---|
| add_update_callback | Adds a callback that will be called when a node’s state changes. |
| cancel | If possible, cancels further execution of this graph. |
| cancelled_nodes | A snapshot of all nodes that have been cancelled. |
| execute | Starts execution of this graph with the given input values. |
| failed_nodes | A snapshot of all nodes that have failed. |
| node | Gets the node identified either by name, ID, or builder node. |
| nodes_by_name | A dictionary of all nodes, keyed by node name. |
| retry | Attempts to retry a node. |
| retry_all | Retries all retry-able nodes. |
| running_nodes | A snapshot of all nodes currently running. |
| successful_nodes | A snapshot of all nodes that have completed successfully. |
| unstarted_nodes | A snapshot of all nodes that have not yet been started. |
| visualize | Returns a visualization of this graph for a Jupyter notebook. |
| wait | Waits for the execution of this task graph to complete. |
add_update_callback
cloud.taskgraphs.executor.Executor.add_update_callback(callback)Adds a callback that will be called when a node’s state changes.
Callbacks are delivered on a best-effort basis and may be asynchronous or batched depending upon the implementation. There is no guarantee as to what thread the callback may come in on.
cancel
cloud.taskgraphs.executor.Executor.cancel()If possible, cancels further execution of this graph.
Like futures.Future.cancel, this returns True if the graph could be cancelled, and False if not.
cancelled_nodes
cloud.taskgraphs.executor.Executor.cancelled_nodes()A snapshot of all nodes that have been cancelled.
This includes both nodes that were manually cancelled, and nodes that are not executed because their parents failed.
execute
cloud.taskgraphs.executor.Executor.execute(**inputs)Starts execution of this graph with the given input values.
failed_nodes
cloud.taskgraphs.executor.Executor.failed_nodes()A snapshot of all nodes that have failed.
The returned collection of nodes is not guaranteed to be fully consistent, since in the process of iterating over nodes, the status of some nodes may have changed.
node
cloud.taskgraphs.executor.Executor.node(nid)Gets the node identified either by name, ID, or builder node.
When passed: - a str: The node with the given name. - a :class:uuid.UUID: The node with the given ID. - a :class:builder.Node: The execution node corresponding to the given node from the :class:builder.Builder.
nodes_by_name
cloud.taskgraphs.executor.Executor.nodes_by_name()A dictionary of all nodes, keyed by node name.
retry
cloud.taskgraphs.executor.Executor.retry(node)Attempts to retry a node.
Returns True if the node was retried, false if not.
retry_all
cloud.taskgraphs.executor.Executor.retry_all()Retries all retry-able nodes.
running_nodes
cloud.taskgraphs.executor.Executor.running_nodes()A snapshot of all nodes currently running.
The returned collection of nodes is not guaranteed to be fully consistent, since in the process of iterating over nodes, the status of some nodes may have changed.
successful_nodes
cloud.taskgraphs.executor.Executor.successful_nodes()A snapshot of all nodes that have completed successfully.
The returned collection of nodes is not guaranteed to be fully consistent, since in the process of iterating over nodes, the status of some nodes may have changed.
unstarted_nodes
cloud.taskgraphs.executor.Executor.unstarted_nodes()A snapshot of all nodes that have not yet been started.
The returned collection of nodes is not guaranteed to be fully consistent, since in the process of iterating over nodes, the status of some nodes may have changed.
visualize
cloud.taskgraphs.executor.Executor.visualize()Returns a visualization of this graph for a Jupyter notebook.
wait
cloud.taskgraphs.executor.Executor.wait(timeout=None)Waits for the execution of this task graph to complete.
Node
cloud.taskgraphs.executor.Node(uid, owner, name)An abstract type specifying the operations on a Node of a task graph.
Executor implementations will return instances of implementations of these Nodes when executing a task graph. If a caller uses only the methods here when manipulating task graph nodes, the actions they take will work (to the extent that they are supported) no matter the specifics of the executor itself (client-side, server-side, etc.).
The external-facing API matches that of futures.Future, with some added niceties (like status), and without the internal methods that are only “meant for use in unit tests and Executor implementations”: set_running_or_notify_cancel, set_result, and set_exception.
The generic types on this (which are really only of concern to Executor implementors) represent the Executor type and the type the Node yields::
Node[MyExecutor, int]
# A Node subtype that is executed by a MyExecutor
# and whose .result() is an int.Attributes
| Name | Description |
|---|---|
| display_name | The name for this node that should show up in UIs. |
| fallback_name | A fallback name for this node if unnamed. |
| id | The client-generated UUID of this node. |
| name | The name of the node, if present. |
| owner | The executor which this node belongs to. |
| status | The current lifecycle state of this Node. |
Methods
| Name | Description |
|---|---|
| add_done_callback | Adds a callback that will be called when this Node completes. |
| cancel | If possible, cancels execution of this node. |
| cancelled | Returns True if the Node was cancelled. |
| done | Returns True if this Node has completed or been cancelled. |
| exception | If this node failed, the exception that was raised. |
| result | The value resulting from executing this node. |
| retry | Attempts to submit this node for retry, if possible. |
| running | Returns True if the Node is currently executing. |
| task_id | The task ID that was returned from the server, if applicable. |
| wait | Waits for the given amount of time for this Node to complete. |
add_done_callback
cloud.taskgraphs.executor.Node.add_done_callback(fn)Adds a callback that will be called when this Node completes.
While the current behavior is similar to the way add_done_callback on a regular Future works, we don’t guarantee that it will remain the same (e.g. will it be called immediately, on what thread). A callback may be called back multiple times if the Node completes multiple times.
cancel
cloud.taskgraphs.executor.Node.cancel()If possible, cancels execution of this node.
Returns True if cancellation succeeded, False if we could not cancel.
cancelled
cloud.taskgraphs.executor.Node.cancelled()Returns True if the Node was cancelled.
done
cloud.taskgraphs.executor.Node.done()Returns True if this Node has completed or been cancelled.
exception
cloud.taskgraphs.executor.Node.exception(timeout=None)If this node failed, the exception that was raised.
If the Node succeeded, returns None. If the Node was cancelled, a futures.CancelledError will be raised rather than returned.
result
cloud.taskgraphs.executor.Node.result(timeout=None)The value resulting from executing this node.
Returns the result if present, or raises an exception if execution raised an exception.
retry
cloud.taskgraphs.executor.Node.retry()Attempts to submit this node for retry, if possible.
running
cloud.taskgraphs.executor.Node.running()Returns True if the Node is currently executing.
task_id
cloud.taskgraphs.executor.Node.task_id(timeout=None)The task ID that was returned from the server, if applicable.
If this was executed on the server side, this should return the UUID of the actual execution of this task. If it was purely client-side, or the server did not return a UUID, this should return None.
wait
cloud.taskgraphs.executor.Node.wait(timeout=None)Waits for the given amount of time for this Node to complete.
ParentFailedError
cloud.taskgraphs.executor.ParentFailedError(cause, node)Raised when the parent of a Node fails.
Status
cloud.taskgraphs.executor.Status()The current status of a Node (or a graph).
Attributes
| Name | Description |
|---|---|
| CANCELLED | The Node was cancelled before it could complete. |
| FAILED | The Node failed to complete. |
| PARENT_FAILED | One of the Node’s parents failed, so the Node could not run. |
| READY | All the inputs of a Node have resolved and it can run. |
| RUNNING | The Node is currently running. |
| SUCCEEDED | The Node completed successfully. |
| WAITING | A Node is waiting for input values. |