API Reference¶
Futures module¶
The following methods are part of the futures module. They can be accessed like so:
from scoop import futures
results = futures.map(func, data)
futureObject = futures.submit(func, arg)
...
More informations are available in the Usage document.
-
scoop.futures.
as_completed
(fs, timeout=None)[source]¶ Iterates over the given futures that yields each as it completes. This call is blocking.
Parameters: - fs – The sequence of Futures to wait upon.
- timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.
Returns: An iterator that yields the given Futures as they complete (finished or cancelled).
-
scoop.futures.
map
(func, *iterables)[source]¶ Equivalent to map(func, *iterables, ...) but func is executed asynchronously and several calls to func may be made concurrently. This non-blocking call returns an iterator which raises a TimeoutError if __next__() is called and the result isn’t available after timeout seconds from the original call to map(). If timeout is not specified or None then there is no limit to the wait time. If a call raises an exception then that exception will be raised when its value is retrieved from the iterator.
Parameters: - func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
- iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
- timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.
Returns: A generator of map results, each corresponding to one map iteration.
-
scoop.futures.
map_as_completed
(func, *iterables)[source]¶ Equivalent to map, but the results are returned as soon as they are made available.
Parameters: - func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
- iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
- timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.
Returns: A generator of map results, each corresponding to one map iteration.
-
scoop.futures.
shutdown
(wait=True)[source]¶ This function is here for compatibility with futures (PEP 3148) and doesn’t have any behavior.
Parameters: wait – Unapplied parameter.
-
scoop.futures.
wait
(fs, timeout=-1, return_when='ALL_COMPLETED')[source]¶ Wait for the futures in the given sequence to complete. Using this function may prevent a worker from executing.
Parameters: - fs – The sequence of Futures to wait upon.
- timeout – The maximum number of seconds to wait. If negative or not specified, then there is no limit on the wait time.
- return_when –
Indicates when this function should return. The options are:
FIRST_COMPLETED Return when any future finishes or is cancelled. FIRST_EXCEPTION Return when any future finishes by raising an exception. If no future raises an exception then it is equivalent to ALL_COMPLETED. ALL_COMPLETED Return when all futures finish or are cancelled.
Returns: A named 2-tuple of sets. The first set, named ‘done’, contains the futures that completed (is finished or cancelled) before the wait completed. The second set, named ‘not_done’, contains uncompleted futures.
Future class¶
The submit()
function returns a
Future
object.
This instance possesses the following methods.
-
class
scoop._types.
Future
(parentId, callable, *args, **kargs)[source]¶ This class encapsulates an independent future that can be executed in parallel. A future can spawn other parallel futures which themselves can recursively spawn other futures.
-
add_done_callback
(callable_, inCallbackType='standard', inCallbackGroup=None)[source]¶ Attach a callable to the future that will be called when the future is cancelled or finishes running. Callable will be called with the future as its only argument.
Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. If the callable raises an Exception then it will be logged and ignored. If the callable raises another BaseException then behavior is not defined.
If the future has already completed or been cancelled then callable will be called immediately.
-
cancel
()[source]¶ If the call is currently being executed or sent for remote execution, then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True.
-
done
()[source]¶ Returns True if the call was successfully cancelled or finished running, False otherwise. This function updates the executionQueue so it receives all the awaiting message.
-
exception
(timeout=None)[source]¶ Return the exception raised by the call. If the call hasn’t yet completed then this method will wait up to timeout seconds. More information in the Usage page. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.
If the future is cancelled before completing then CancelledError will be raised.
If the call completed without raising then None is returned.
Returns: The exception raised by the call.
-
result
(timeout=None)[source]¶ Return the value returned by the call. If the call hasn’t yet completed then this method will wait up to ‘’timeout’’ seconds. More information in the Usage page. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.
If the future is cancelled before completing then CancelledError will be raised.
If the call raised an exception then this method will raise the same exception.
Returns: The value returned by the callable object.
-
SCOOP Constants and objects¶
The following objects are available to a program that was launched using SCOOP.
Note
Please note that using these is considered as advanced usage. You should not rely on these for other purposes than debugging.
Constants | Description |
---|---|
scoop.IS_ORIGIN | Boolean value. True if current instance is the root worker. |
scoop.BROKER | broker.broker.BrokerInfo namedtuple. Address, ports and hostname of the broker. |
scoop.DEBUG | Boolean value. True if debug mode is enabled, false otherwise. |
scoop.IS_RUNNING | Boolean value. True if SCOOP is currently running, false otherwise. |
scoop.worker | 2-tuple. Unique identifier of the current instance in the pool. |
scoop.logger | Logger object. Provides log formatting and redirection facilities. See the official documentation for more information on its usage. |