Computing Reviews
Today's Issue Hot Topics Search Browse Recommended My Account Log In
Review Help
Efficient differentiable programming in a functional array-processing language
Shaikhha A., Fitzgibbon A., Vytiniotis D., Peyton Jones S. Proceedings of the ACM on Programming Languages3 (ICFP):1-30,2019.Type:Article
Date Reviewed: Jul 20 2021

Differentiable programming, or automatic differentiation, is a powerful technique in many fields, including dynamic systems, machine learning, and computer vision, mainly for solving nonlinear problems. Forward (versus reverse) differentiation based on chain rule is already implemented in software, both by operator overloading and source-to-source transformation. In this paper, the authors propose a functional programming framework, based on array processing and optimization rules, to generate efficient forward differentiable programs.

The reader will enjoy this well-structured, comprehensive paper’s many technicalities, some of which are explained via inspired examples. The presentation uses fundamental techniques from functional programming (mainly λ-calculus, Standard ML, F#) and functional linear algebra (similar to matrix-oriented processors like MATLAB, R, or NumPy).

In seven sections, the authors present the required background on automatic differentiation (section 2), the proposed functional array-processing language (section 3), the differentiation process (section 4), the transformation rules and code generation (section 5), experimental results (section 6), and connections to related works and possible future developments (section 7).

The most important developments relate to the differentiation process: the deriv construct; the high-level differentiation application programming interface for (diff, grad, jacob); and the automatic differentiation rules for expressions in order to support “source-to-source translation for implementing forward-mode automatic differentiation.” Many classes of transformation rules are considered for efficient differentiation (rules based on λ-calculus, rules according to the algebraic structure, fusion rules for pull-arrays, rewriting rules for conditional expressions, normalization rules for loops, partial evaluation rules for tuples, and loop fission rules). This technical part closes with C code generated for an optimized differentiated program, and a final discussion on key challenges for applying rewriting rules, heuristics, cost models, and destination-passing style techniques.

The proposed approach is compared against DiffSharp, Tapenade, Theano, TensorFlow, and Futhark. The optimized differentiated approach with improved memory management for low-level code generation outperformed competitors. However, there is no online reference to a compiler implementation demonstrating such performance.

Reviewer:  G. Albeanu Review #: CR147312 (2111-0269)
Bookmark and Share
  Featured Reviewer  
General (D.3.0 )
Automatic Differentiation (G.1.4 ... )
General (G.0 )
Would you recommend this review?
Other reviews under "General": Date
Programming languages: paradigm and practice
Appleby D., McGraw-Hill, Inc., New York, NY, 1991. Type: Book (9780075579045)
Jan 1 1992
Programming languages
Dershem H. (ed), Jipping M., Wadsworth Publ. Co., Belmont, CA, 1990. Type: Book (9780534129002)
Jan 1 1992
Comparative programming languages
Friedman L., Prentice-Hall, Inc., Upper Saddle River, NJ, 1991. Type: Book (9780131554825)
Jan 1 1992

E-Mail This Printer-Friendly
Send Your Comments
Contact Us
Reproduction in whole or in part without permission is prohibited.   Copyright 1999-2023 ThinkLoud®
Terms of Use
| Privacy Policy