View source code Display the source code in std/parallelism.d from which this page was generated on github. Improve this page Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using local clone. Page wiki View or edit the community-maintained wiki page associated with this page.

std.parallelism.task_pool.async_buf - multiple declarations

Function TaskPool.asyncBuf

Given a source range that is expensive to iterate over, returns an input range that asynchronously buffers the contents of source into a buffer of bufSize elements in a worker thread, while making previously buffered elements from a second buffer, also of size bufSize, available via the range interface of the returned object. The returned range has a length iff hasLength!S. asyncBuf is useful, for example, when performing expensive operations on the elements of ranges that represent data on a disk or network.

Prototype

auto asyncBuf(S)(
  S source,
  size_t bufSize = 100
)
if (isInputRange!S);

Examples

import std.conv, std.stdio;

void main()
{
    // Fetch lines of a file in a background thread
    // while processing previously fetched lines,
    // dealing with byLine's buffer recycling by
    // eagerly duplicating every line.
    auto lines = File("foo.txt").byLine();
    auto duped = std.algorithm.map!"a.idup"(lines);

    // Fetch more lines in the background while we
    // process the lines already read into memory
    // into a matrix of doubles.
    double[][] matrix;
    auto asyncReader = taskPool.asyncBuf(duped);

    foreach(line; asyncReader)
    {
        auto ls = line.split("\t");
        matrix ~= to!(double[])(ls);
    }
}

Exception Handling:

Any exceptions thrown while iterating over source are re-thrown on a call to popFront or, if thrown during construction, simply allowed to propagate to the caller.

Function TaskPool.asyncBuf

Given a callable object next that writes to a user-provided buffer and a second callable object empty that determines whether more data is available to write via next, returns an input range that asynchronously calls next with a set of size nBuffers of buffers and makes the results available in the order they were obtained via the input range interface of the returned object. Similarly to the input range overload of asyncBuf, the first half of the buffers are made available via the range interface while the second half are filled and vice-versa.

Prototype

auto asyncBuf(C1, C2)(
  C1 next,
  C2 empty,
  size_t initialBufSize = 0,
  size_t nBuffers = 100
)
if (is(typeof(C2.init()) : bool) && ParameterTypeTuple!C1.length == 1 && ParameterTypeTuple!C2.length == 0 && isArray!(ParameterTypeTuple!C1[0]));

Parameters

NameDescription
next A callable object that takes a single argument that must be an array with mutable elements. When called, next writes data to the array provided by the caller.
empty A callable object that takes no arguments and returns a type implicitly convertible to bool. This is used to signify that no more data is available to be obtained by calling next.
initialBufSize The initial size of each buffer. If next takes its array by reference, it may resize the buffers.
nBuffers The number of buffers to cycle through when calling next.

Examples

// Fetch lines of a file in a background
// thread while processing previously fetched
// lines, without duplicating any lines.
auto file = File("foo.txt");

void next(ref char[] buf)
{
    file.readln(buf);
}

// Fetch more lines in the background while we
// process the lines already read into memory
// into a matrix of doubles.
double[][] matrix;
auto asyncReader = taskPool.asyncBuf(&next, &file.eof);

foreach(line; asyncReader)
{
    auto ls = line.split("\t");
    matrix ~= to!(double[])(ls);
}

Exception Handling:

Any exceptions thrown while iterating over range are re-thrown on a call to popFront.

Warning

Using the range returned by this function in a parallel foreach loop will not work because buffers may be overwritten while the task that processes them is in queue. This is checked for at compile time and will result in a static assertion failure.

Authors

License

Boost License 1.0

Comments