Generator (computer science)

In computer science, a generator is a routine that can be used to control the iteration behaviour of a loop. All generators are also iterators. A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator ''looks like'' a function but ''behaves like'' an iterator.

Generators can be implemented in terms of more expressive control flow constructs, such as coroutines or first-class continuations. Generators, also known as semicoroutines, are a special case of (and weaker than) coroutines, in that they always yield control back to the caller (when passing a value back), rather than specifying a coroutine to jump to; see comparison of coroutines with generators.

Uses

Generators are usually invoked inside loops.The Icon Programming Language utilizes generators to implement its goal directed evaluation. In Icon, generators can be invoked in contexts outside of the normal looping control structures. The first time that a generator invocation is reached in a loop, an iterator object is created that encapsulates the state of the generator routine at its beginning, with arguments bound to the corresponding parameters. The generator's body is then executed in the context of that iterator until a special ''yield'' action is encountered; at that time, the value provided with the ''yield'' action is used as the value of the invocation expression. The next time the same generator invocation is reached in a subsequent iteration, the execution of the generator's body is resumed after the ''yield'' action, until yet another ''yield'' action is encountered. In addition to the ''yield'' action, execution of the generator body can also be terminated by a ''finish'' action, at which time the innermost loop enclosing the generator invocation is terminated. In more complicated situations, a generator may be used manually outside of a loop to create an iterator, which can then be used in various ways.

Because generators compute their yielded values only on demand, they are useful for representing streams, such as sequences that would be expensive or impossible to compute at once. These include e.g. infinite sequences and live data streams.

When eager evaluation is desirable (primarily when the sequence is finite, as otherwise evaluation will never terminate), one can either convert to a list, or use a parallel construction that creates a list instead of a generator. For example, in Python a generator g can be evaluated to a list l via l = list(g), while in F# the sequence expression seq evaluates lazily (a generator or sequence) but ... /code> evaluates eagerly (a list).

In the presence of generators, loop constructs of a language – such as for and while – can be reduced into a single loop ... end loop construct; all the usual loop constructs can then be comfortably simulated by using suitable generators in the right way. For example, a ranged loop like for x = 1 to 10 can be implemented as iteration through a generator, as in Python's for x in range(1, 10). Further, break can be implemented as sending ''finish'' to the generator and then using continue in the loop.


Timeline

Generators first appeared in CLU (1975), were a prominent feature in the string manipulation language Icon (1977) and are now available in Python (2001),
Python Enhancement Proposals
PEP 255: Simple GeneratorsPEP 289: Generator ExpressionsPEP 342: Coroutines via Enhanced Generators
C#, Ruby, PHP, ECMAScript (as of ES6/ES2015), and other languages. In CLU and C#, generators are called ''iterators'', and in Ruby, ''enumerators''.


Lisp

The final Common Lisp standard does not natively provide generators, yet various library implementations exist, such a
SERIES
documented in CLtL2 o
pygen



CLU

A yield statement is used to implement iterators over user-defined data abstractions.


string_chars = iter (s: string) yields (char);
index: int := 1;
limit: int := string$size (s);
while index <= limit do
yield (string$fetch(s, index));
index := index + 1;
end;
end string_chars;

for c: char in string_chars(s) do
...
end;



Icon

Every expression (including loops) is a generator. The language has many generators built-in and even implements some of the logic semantics using the generator mechanism (logical disjunction or "OR" is done this way).

Printing squares from 0 to 20 can be achieved using a co-routine by writing:

local squares, j
squares := create (seq(0) ^ 2)
every j := , @squares do
if j <= 20 then
write(j)
else
break


However, most of the time custom generators are implemented with the "suspend" keyword which functions exactly like the "yield" keyword in CLU.


C

C does not have generator functions as a language construct, but, as they are a subset of coroutines, it is simple to implement them using any framework that implements stackful coroutines, such as libdill. On POSIX platforms, when the cost of context switching per iteration is not a concern, or full parallelism rather than merely concurrency is desired, a very simple generator function framework can be implemented using pthreads and pipes.


C++

It is possible to introduce generators into C++ using pre-processor macros. The resulting code might have aspects that are very different from native C++, but the generator syntax can be very uncluttered. The set of pre-processor macros defined in this source allow generators defined with the syntax as in the following example:


$generator(descent)
;


This can then be iterated using:


int main(int argc, char* argv[])

Moreover, C++11 allows foreach loops to be applied to any class that provides the begin and end functions. It's then possible to write generator-like classes by defining both the iterable methods (begin and end) and the iterator methods (operator!=, operator++ and operator*) in the same class. For example, it is possible to write the following program:


#include
int main()

A basic range implementation would look like that:


class range
;



Perl

Perl does not natively provide generators, but support is provided by th
Coro::Generator
module which uses th
Coro
co-routine framework. Example usage:


use strict;
use warnings;
# Enable generator and yield
use Coro::Generator;
# Array reference to iterate over
my $chars = A'...'Z'

# New generator which can be called like a coderef.
my $letters = generator ;

# Call the generator 15 times.
print $letters->(), "\n" for (0..15);




Raku

Example parallel to Icon uses Raku (formerly/aka Perl 6) Range class as one of several ways to achieve generators with the language.

Printing squares from 0 to 20 can be achieved by writing:

for (0 .. *).map(* ** 2) -> $i


However, most of the time custom generators are implemented with "gather" and "take" keywords in a lazy context.


Tcl

In Tcl 8.6, the generator mechanism is founded on named coroutines.


proc generator

# Use a simple 'for' loop to do the actual generation
set count enerator
# Pull values from the generator until it is exhausted
while 1



Haskell

In Haskell, with its lazy evaluation model, everything is a generator - every datum created with a non-strict data constructor is generated on demand. For example,

countfrom n = n : countfrom (n+1)

-- Example use: printing out the integers from 10 to 20.
test1 = mapM_ print $ takeWhile (<= 20) $ countfrom 10

primes = 2 : 3 : nextprime 5 where
nextprime n , b = n : nextprime (n+2)
, otherwise = nextprime (n+2)
where b = all ((/= 0).(rem n)) $ takeWhile ((<= n).(^2)) $ tail primes

where (:) is a non-strict list constructor, ''cons'', and $ is just a ''"called-with"'' operator, used for parenthesization. This uses the standard adaptor function,

takeWhile p [] = []
takeWhile p (x:xs) , p x = x : takeWhile p xs
, otherwise = []

which re-fetches values agreeable with a predicate, and stops requesting new values as soon as a non-agreeable one is encountered. The shared storage access is used as a universal mediator in Haskell. List comprehensions can be freely used:

test2 = mapM_ print $ takeWhile (<= 20) x <- countfrom 10test3 = mapM_ print x <- takeWhile (<= 20) $ countfrom 10


Racket

Racket provides several related facilities for generators. First, its for-loop forms work with ''sequences'', which are a kind of a producer:

(for ( (in-range 10 20)
(printf "i = ~s\n" i))

and these sequences are also first-class values:

(define 10-to-20 (in-range 10 20))
(for ( 10-to-20
(printf "i = ~s\n" i))

Some sequences are implemented imperatively (with private state variables) and some are implemented as (possibly infinite) lazy lists. Also, new struct definitions can have a property that specifies how they can be used as sequences.

But more directly, Racket comes with a generator library for a more traditional generator specification. For example,

#lang racket
(require racket/generator)
(define (ints-from from)
(generator ()
(for ( (in-naturals from) ; infinite sequence of integers from 0
(yield i))))
(define g (ints-from 10))
(list (g) (g) (g)) ; -> '(10 11 12)

Note that the Racket core implements powerful continuation features, providing general (re-entrant) continuations that are composable, and also delimited continuations. Using this, the generator library is implemented in Racket.

PHP

The community of PHP implemented generators in PHP 5.5. Details can be found in the origina
Request for Comments: Generators

Infinite Fibonacci sequence:

function fibonacci()

foreach (fibonacci() as $number)

Fibonacci sequence with limit:

function fibonacci(int $limit):generator

foreach (fibonacci(10) as $number)

Any function which contains a yield statement is automatically a generator function.

Ruby

Ruby supports generators (starting from version 1.9) in the form of the built-in Enumerator class.

# Generator from an Enumerator object
chars = Enumerator.new( A', 'B', 'C', 'Z'

4.times

# Generator from a block
count = Enumerator.new do , yielder,
i = 0
loop
end

100.times

Java

Java has had a standard interface for implementing iterators since its early days, and since Java 5, the "foreach" construction makes it easy to loop over objects that provide the java.lang.Iterable interface. (The Java collections framework and other collections frameworks, typically provide iterators for all collections.)

However, Java does not have generators built into the language. This means that creating iterators is often much trickier than in languages with built-in generators, especially when the generation logic is complex. Because all state must be saved and restored every time an item is to be yielded from an iterator, it is not possible to store state in local variables or use built-in looping routines, as when generators are available; instead, all of this must be manually simulated, using object fields to hold local state and loop counters.

Even simple iterators built this way tend to be significantly bulkier than those using generators, with a lot of boilerplate code.

The original example above could be written in Java 5 as:

// Iterator implemented as anonymous class. This uses generics but doesn't need to.
for (int i: new Iterable() )

An infinite Fibonacci sequence could also be written in Java 5 as an Iterator:

Iterable fibo = new Iterable() ;
// this could then be used as...
for (int f: fibo)

Also an infinite Fibonacci sequence could also be written using Java 8 Stream interface:

Iterable myIterable = Stream
// Generates Fib sequence
.iterate(new Integer[], x -> new Integer[] )
.map(x -> x[0])::iterator;
myIterable.forEach(System.out::println);

Or get an Iterator from the Java 8 super-interface BaseStream of Stream interface.

// Save the iterator of a stream that generates fib sequence
Iterator myGenerator = Stream
// Generates Fib sequence
.iterate(new Integer[], x -> new Integer[] )
.map(x -> x[0]).iterator();

// Print the first 5 elements
for (int i = 0; i < 5; i++)

System.out.println("done with first iteration");

// Print the next 5 elements
for (int i = 0; i < 5; i++)

/* Output:
1
1
2
3
5
done with first iteration
8
13
21
34
55 */

C#

An example C# 2.0 generator (the yield is available since C# version 2.0):
Both of these examples utilize generics, but this is not required. yield keyword also helps in implementing custom stateful iterations over a collection as discussed in this discussion.

// Method that takes an iterable input (possibly an array)
// and returns all even numbers.
public static IEnumerable GetEven(IEnumerable numbers)

It is possible to use multiple yield return statements and they are applied in sequence on each iteration:

public class CityCollection : IEnumerable

XL

In XL, iterators are the basis of 'for' loops:

import IO = XL.UI.CONSOLE

iterator IntegerIterator (var out Counter : integer; Low, High : integer) written Counter in Low..High is
Counter := Low
while Counter <= High loop
yield
Counter += 1

// Note that I needs not be declared, because declared 'var out' in the iterator
// An implicit declaration of I as an integer is therefore made here
for I in 1..5 loop
IO.WriteLn "I=", I

F#

F# provides generators via '' sequence expressions,'' since version 1.9.1. These can define a sequence (lazily evaluated, sequential access) via seq , a list (eagerly evaluated, sequential access) via ... /code> or an array (eagerly evaluated, indexed access) via
seq_

forms_a_sequence_of_squares_of_numbers_from_0_to_14_by_filtering_out_numbers_from_the_range_of_numbers_from_0_to_25.


_Python

Generators_were_added_to__Python_in_version_2.2_in_2001._An_example_generator:


from_typing_import_Iterator

def_countfrom(n:_int)_->_Iterator nt
____while_True:
________yield_n
________n_+=_1

#_Example_use:_printing_out_the_integers_from_10_to_20.
#_Note_that_this_iteration_terminates_normally,_despite
#_countfrom()_being_written_as_an_infinite_loop.

for_i_in_countfrom(10):
____if_i_<=_20:
________print(i)
____else:
________break

#_Another_generator,_which_produces_prime_numbers_indefinitely_as_needed.
import_itertools

def_primes()_->_Iterator nt
____yield_2
____n_=_3
____p_=_[]
____while_True:
________#_If_dividing_n_by_all_the_numbers_in_p,_up_to_and_including_sqrt(n),
________#_produces_a_non-zero_remainder_then_n_is_prime.
________if_all(n_%_f_>_0_for_f_in_itertools.takewhile(lambda_f:_f*f_<=_n,_p)):
____________yield_n
____________p.append(n)
________n_+=_2

In_Python,_a_generator_can_be_thought_of_as_an__iterator_that_contains_a_frozen_ stack_frame._Whenever_next()_is_called_on_the_iterator,_Python_resumes_the_frozen_frame,_which_executes_normally_until_the_next_yield_statement_is_reached._The_generator's_frame_is_then_frozen_again,_and_the_yielded_value_is_returned_to_the_caller.

PEP_380_(implemented_in_Python_3.3)_adds_the_yield_from_expression,_allowing_a_generator_to_delegate_part_of_its_operations_to_another_generator_or_iterable.PEP_380_--_Syntax_for_Delegating_to_a_Subgenerator
/ref>


_Generator_expressions

Python_has_a_syntax_modeled_on_that_of_ list_comprehensions,_called_a_generator_expression_that_aids_in_the_creation_of_generators.
The_following_extends_the_first_example_above_by_using_a_generator_expression_to_compute_squares_from_the_countfrom_generator_function:

squares_=_(n_*_n_for_n_in_countfrom(2))

for_j_in_squares:
____if_j_<=_20:
________print(j)
____else:
________break

_ECMAScript

ECMAScript_6_(a.k.a._Harmony)_introduced_generator_functions.

An_infinite_Fibonacci_sequence_can_be_written_using_a_function_generator:


function*_fibonacci(limit)_

//_bounded_by_upper_limit_10
for_(const_n_of_fibonacci(10))_

//_generator_without_an_upper_bound_limit
for_(const_n_of_fibonacci())_

//_manually_iterating
let_fibGen_=_fibonacci();
console.log(fibGen.next().value);_//_1
console.log(fibGen.next().value);_//_1
console.log(fibGen.next().value);_//_2
console.log(fibGen.next().value);_//_3
console.log(fibGen.next().value);_//_5
console.log(fibGen.next().value);_//_8

//_picks_up_from_where_you_stopped
for_(const_n_of_fibGen)_

_R

The_iterators_package_can_be_used_for_this_purpose.


library(iterators)

#_Example_------------------
abc_<-_iter(c('a','b','c'))
nextElem(abc)

_Smalltalk

Example_in_ Pharo_Smalltalk:

The_Golden_ratio_generator_below_returns_to_each_invocation_'goldenRatio_next'_a_better_approximation_to_the_Golden_Ratio._


goldenRatio_:=_Generator_on:_ _, _x_y_z_r_, _
x_:=_0.
y_:=_1.
[__
z_:=_x_+_y.
r_:=_(z_/_y)_asFloat.
x_:=_y.
y_:=_z.
g_yield:_r
repeat
].

goldenRatio_next.

The_expression_below_returns_the_next_10_approximations.


Character_cr_join:_((1_to:_10)_collect:_ :dummy_, _ratio_next_.

See_more_i
A_hidden_gem_in_Pharo:_Generator



_See_also

*_ List_comprehension_for_another_construct_that_generates_a_sequence_of_values
*_ Iterator_for_the_concept_of_producing_a_list_one_element_at_a_time
*_ Iteratee_for_an_alternative
*_ Lazy_evaluation_for_producing_values_when_needed
*_ Corecursion_for_potentially_infinite_data_by_recursion_instead_of_''yield''
*_ Coroutine_for_even_more_generalization_from_subroutine
*_ Continuation_for_generalization_of_control_flow

_Notes

_References

*_Stephan_Murer,_ Stephen_Omohundro,_David_Stoutamire_and_Clemens_Szyperski:_Iteration_abstraction_in_Sather.__''ACM_Transactions_on_Programming_Languages_and_Systems'',_18(1):1-15_(1996


{{Authority_control

Programming_constructs
Articles_with_example_Python_(programming_language)_code
Articles_with_example_Haskell_code
Articles_with_example_Ruby_code
Articles_with_example_C_Sharp_code
Articles_with_example_Java_code
Articles_with_example_Perl_code
Articles_with_example_Tcl_code
Articles_with_example_Racket_code
Iteration_in_programming
Articles_with_example_R_codehtml" ;"title=" ... , ] that contain code that generates values. For example,

seq

forms a sequence of squares of numbers from 0 to 14 by filtering out numbers from the range of numbers from 0 to 25.

Python

Generators were added to Python in version 2.2 in 2001. An example generator:

from typing import Iterator

def countfrom(n: int) -> Iterator nt
while True:
yield n
n += 1

# Example use: printing out the integers from 10 to 20.
# Note that this iteration terminates normally, despite
# countfrom() being written as an infinite loop.

for i in countfrom(10):
if i <= 20:
print(i)
else:
break

# Another generator, which produces prime numbers indefinitely as needed.
import itertools

def primes() -> Iterator nt
yield 2
n = 3
p = []
while True:
# If dividing n by all the numbers in p, up to and including sqrt(n),
# produces a non-zero remainder then n is prime.
if all(n % f > 0 for f in itertools.takewhile(lambda f: f*f <= n, p)):
yield n
p.append(n)
n += 2

In Python, a generator can be thought of as an iterator that contains a frozen stack frame. Whenever next() is called on the iterator, Python resumes the frozen frame, which executes normally until the next yield statement is reached. The generator's frame is then frozen again, and the yielded value is returned to the caller.

PEP 380 (implemented in Python 3.3) adds the yield from expression, allowing a generator to delegate part of its operations to another generator or iterable.PEP 380 -- Syntax for Delegating to a Subgenerator
/ref>

Generator expressions

Python has a syntax modeled on that of list comprehensions, called a generator expression that aids in the creation of generators.
The following extends the first example above by using a generator expression to compute squares from the countfrom generator function:

squares = (n * n for n in countfrom(2))

for j in squares:
if j <= 20:
print(j)
else:
break

ECMAScript

ECMAScript 6 (a.k.a. Harmony) introduced generator functions.

An infinite Fibonacci sequence can be written using a function generator:

function* fibonacci(limit)

// bounded by upper limit 10
for (const n of fibonacci(10))

// generator without an upper bound limit
for (const n of fibonacci())

// manually iterating
let fibGen = fibonacci();
console.log(fibGen.next().value); // 1
console.log(fibGen.next().value); // 1
console.log(fibGen.next().value); // 2
console.log(fibGen.next().value); // 3
console.log(fibGen.next().value); // 5
console.log(fibGen.next().value); // 8

// picks up from where you stopped
for (const n of fibGen)

R

The iterators package can be used for this purpose.

library(iterators)

# Example ------------------
abc <- iter(c('a','b','c'))
nextElem(abc)

Smalltalk

Example in Pharo Smalltalk:

The Golden ratio generator below returns to each invocation 'goldenRatio next' a better approximation to the Golden Ratio.

goldenRatio := Generator on: , x y z r ,
x := 0.
y := 1.

z := x + y.
r := (z / y) asFloat.
x := y.
y := z.
g yield: r
repeat

goldenRatio next.

The expression below returns the next 10 approximations.

Character cr join: ((1 to: 10) collect: :dummy , ratio next .

See more i
A hidden gem in Pharo: Generator

See also

* List comprehension for another construct that generates a sequence of values
* Iterator for the concept of producing a list one element at a time
* Iteratee for an alternative
* Lazy evaluation for producing values when needed
* Corecursion for potentially infinite data by recursion instead of ''yield''
* Coroutine for even more generalization from subroutine
* Continuation for generalization of control flow

Notes

References

* Stephan Murer, Stephen Omohundro, David Stoutamire and Clemens Szyperski: Iteration abstraction in Sather. ''ACM Transactions on Programming Languages and Systems'', 18(1):1-15 (1996


{{Authority control

Programming constructs
Articles with example Python (programming language) code
Articles with example Haskell code
Articles with example Ruby code
Articles with example C Sharp code
Articles with example Java code
Articles with example Perl code
Articles with example Tcl code
Articles with example Racket code
Iteration in programming
Articles with example R code> ... , /code> that contain code that generates values. For example,

seq

forms a sequence of squares of numbers from 0 to 14 by filtering out numbers from the range of numbers from 0 to 25.

Python

Generators were added to Python in version 2.2 in 2001. An example generator:

from typing import Iterator

def countfrom(n: int) -> Iterator nt
while True:
yield n
n += 1

# Example use: printing out the integers from 10 to 20.
# Note that this iteration terminates normally, despite
# countfrom() being written as an infinite loop.

for i in countfrom(10):
if i <= 20:
print(i)
else:
break

# Another generator, which produces prime numbers indefinitely as needed.
import itertools

def primes() -> Iterator nt
yield 2
n = 3
p = []
while True:
# If dividing n by all the numbers in p, up to and including sqrt(n),
# produces a non-zero remainder then n is prime.
if all(n % f > 0 for f in itertools.takewhile(lambda f: f*f <= n, p)):
yield n
p.append(n)
n += 2

In Python, a generator can be thought of as an iterator that contains a frozen stack frame. Whenever next() is called on the iterator, Python resumes the frozen frame, which executes normally until the next yield statement is reached. The generator's frame is then frozen again, and the yielded value is returned to the caller.

PEP 380 (implemented in Python 3.3) adds the yield from expression, allowing a generator to delegate part of its operations to another generator or iterable.PEP 380 -- Syntax for Delegating to a Subgenerator
/ref>

Generator expressions

Python has a syntax modeled on that of list comprehensions, called a generator expression that aids in the creation of generators.
The following extends the first example above by using a generator expression to compute squares from the countfrom generator function:

squares = (n * n for n in countfrom(2))

for j in squares:
if j <= 20:
print(j)
else:
break

ECMAScript

ECMAScript 6 (a.k.a. Harmony) introduced generator functions.

An infinite Fibonacci sequence can be written using a function generator:

function* fibonacci(limit)

// bounded by upper limit 10
for (const n of fibonacci(10))

// generator without an upper bound limit
for (const n of fibonacci())

// manually iterating
let fibGen = fibonacci();
console.log(fibGen.next().value); // 1
console.log(fibGen.next().value); // 1
console.log(fibGen.next().value); // 2
console.log(fibGen.next().value); // 3
console.log(fibGen.next().value); // 5
console.log(fibGen.next().value); // 8

// picks up from where you stopped
for (const n of fibGen)

R

The iterators package can be used for this purpose.

library(iterators)

# Example ------------------
abc <- iter(c('a','b','c'))
nextElem(abc)

Smalltalk

Example in Pharo Smalltalk:

The Golden ratio generator below returns to each invocation 'goldenRatio next' a better approximation to the Golden Ratio.

goldenRatio := Generator on: , x y z r ,
x := 0.
y := 1.

z := x + y.
r := (z / y) asFloat.
x := y.
y := z.
g yield: r
repeat

goldenRatio next.

The expression below returns the next 10 approximations.

Character cr join: ((1 to: 10) collect: :dummy , ratio next .

See more i
A hidden gem in Pharo: Generator

See also

* List comprehension for another construct that generates a sequence of values
* Iterator for the concept of producing a list one element at a time
* Iteratee for an alternative
* Lazy evaluation for producing values when needed
* Corecursion for potentially infinite data by recursion instead of ''yield''
* Coroutine for even more generalization from subroutine
* Continuation for generalization of control flow

Notes

References

* Stephan Murer, Stephen Omohundro, David Stoutamire and Clemens Szyperski: Iteration abstraction in Sather. ''ACM Transactions on Programming Languages and Systems'', 18(1):1-15 (1996


{{Authority control

Programming constructs
Articles with example Python (programming language) code
Articles with example Haskell code
Articles with example Ruby code
Articles with example C Sharp code
Articles with example Java code
Articles with example Perl code
Articles with example Tcl code
Articles with example Racket code
Iteration in programming
Articles with example R code