Summary

Starting in MarkLogic Server version 10.0-7, XQuery FLWOR expressions that only use "let" will now stream results. Prior to 10.0-7, MarkLogic Server would have buffered results in memory. This change allows large result sets to be more easily streamed from XQuery modules.

Impact

Due to this change, code that relies on the previous behavior of buffered results from FLWOR expression with only a "let" may experience degraded performance if the results are iterated over multiple times. This is due to the fact that once a streaming result has been exhausted, the query has to be rerun in order to iterate over it again.

Best Practice

Regardless of this change, the best practice is:

• to treat all query calls as lazily-evaluated expressions and only iterate over them once.
• If the results need to be iterated multiple times, wrap the search expression in xdmp:eager() or iterate over the results once and assign the results to a new variable.

Examples

In 10.0-7 and prior versions, the following expression would be lazily-evaluated and run the search multiple times if the $results variable is iterated over multiple times.

This behavior has not changed in 10.0-7. However, prior to 10.0-7, the following expression would short-circuit the lazy evaluation and buffer all of the results in memory

In 10.0-7, this is now consistent with the other form of the expression above and returns an iterator. The search will be run multiple times if the $results variable is iterated over multiple times.

To achieve the buffering behavior in 10.0-7 or later releases, you can wrap cts:search() inside xdmp:eager() as follows

Diagnostics

The xdmp:streamable function was added in MarkLogic 10.0-7 in order to help determine if a variable will stream or not, 

Additional References

For more information about lazy evaluation in MarkLogic, see the following resources

• https://www.marklogic.com/blog/lazy-evaluation/
• https://www.marklogic.com/blog/lazy-vs-eager-evaluation/