AchillesGames

The LLVM Project Blog

[*]The LLVM Project Bloghttps://blog.llvm.org/Recent content on The LLVM Project BlogHugo — gohugo.iollvm.orgWed, 17 Jun 2020 22:30:32 -0700The 2020 Virtual LLVM Developers’ Meeting Programhttps://blog.llvm.org/posts/2020-08-23-llvmdevmtgprogram/Sun, 23 Aug 2020 17:00:06 -0700https://blog.llvm.org/posts/2020-08-23-llvmdevmtgprogram/

2020 Virtual LLVM Developers’ Meeting

The LLVM Foundation is excited to announce the 2020 Virtual LLVM Developers’ Meeting program! Registration will open this week.

Keynote:

  • Undef and Poison: Present and Future – J. Lee

Technical Talks:

  • Clang & Linux: Asm Goto with Outputs – B. Wendling; N. Desaulniers
  • LLVM Libc: Current Status, Challenges and Future Plans – S. Reddy; G. Chatelet; P. Asker; D. Finkelstein
  • Branch Coverage: Squeezing more out of LLVM Source-based Code Coverage – A. Phipps
  • Memory tagging in LLVM and Android – E. Stepanov; K. Serebryany; P. Collingbourne; M. Phillips
  • Towards a representation of arbitrary alias graph in LLVM IR for Fortran code – K. Li; T. Islam
  • Control-flow sensitive escape analysis in Falcon JIT – A. Pilipenko
  • Extending Clang for checking compliance with automotive coding standards – M. Vujosevic Janicic
  • An Update on Optimizing Multiple Exit Loops – P. Reames
  • Code Size Compiler Optimizations and Techniques – A. Kumar
  • Accelerate Matrix Multiplication Using the New POWER Outer Product Instructions – B. Saleil; J. Carvalho
  • CIL : Common MLIR Dialect for C/C++ and Fortran – P. NR; V. M; Ranjith; Srihari
  • Building compiler extension for LLVM 10.0.1 – S. Guelton
  • LLVM-based mutation testing for C and C++ – A. Denisov; S. Pankevich
  • Matrix Support in Clang and LLVM- F. Hahn
  • Adding CUDA® Support to Cling: JIT Compile to GPUs – S. Ehrig
  • The Present and Future of Interprocedural Optimization in LLVM – J. Doerfert; B. Homerding; S. Baziotis; S. Stipanovic; H. Ueno; K. Dinel; S. Okumura; L. Chen
  • Pushing Back Lit’s Boundaries to Test Libc++ – L. Dionne
  • Evolving “convergent”: Lessons from Control Flow in AMDGPU – N. Hähnle
  • How to update debug info in compiler transformations – A. Prantl; V. Kumar
  • Proposal for A Framework for More Effective Loop Optimizations – M. Kruse; H. Finkel
  • Changing Everything With Clang Plugins: A Story About Syntax Extensions, Clang’s AST, and Quantum Computing = H. Finkel; A. Mccaskey
  • (OpenMP) Parallelism-Aware Optimizations – J. Doerfert; S. Stipanovic; H. Mosquera; J. Chesterfield; G. Georgakoudis; J. Huber
  • Checked C: Adding memory safety support to LLVM – M. Grang; K. Kjeer

Tutorials:

  • Everything I know about debugging LLVM – N. Desaulniers
  • LLVM in a Bare Metal Environment – H. Qadeer
  • PGO: Add per-callsite counters – P. Kosov; Y. Sergey
  • Understanding Changes made by a Pass in the Opt Pipeline. – J. Schmeiser
  • Using clang-tidy for customized checkers and large scale source tree refactoring – V. Bridgers
  • Finding Your Way Around the LLVM Dependence Analysis Zoo – S. Baziotis; S. Moll
  • Using the clang static analyzer to find bugs – V. Bridgers
  • A Deep Dive into the Interprocedural Optimization Infrastructure – J. Doerfert; B. Homerding; S. Baziotis; S. Stipanovic; H. Ueno; K. Dinel; S. Okumura; L. Chen
  • MLIR Tutorial – M. Amini

Lightning Talks:

  • Finding and Outlining Similarities in LLVM IR – A. Litteken
  • A fast algorithm for global code motion of congruent instructions – A. Kumar; S. Pop
  • From Implicit Pass Dependencies to Effectiveness Prediction – H. Ueno; J. Doerfert; E. Park; G. Georgakoudis; T. Jayatilaka; S. Badruswamy
  • Using Clang as An Alternative C/C++ Frontend of The ROSE Source-to-Source Compiler – A. Wang; P. Lin; C. Liao; Y. Yan
  • OpenACC support in Flang with a MLIR dialect – V. Clement; J. Vetter
  • Fragmenting the DWARF to Enable Dead Debug Data Elimination – J. Henderson
  • Source-based Code Coverage for Embedded Use Cases – A. Phipps; C. Addison
  • pre-merge checks for LLVM – M. Goncharov; C. Kühnel
  • Getting stack size just right on XCore – J. McCrea
  • Compile Faster with the Program Repository and ccache – Y. Yi; P. Bowen-Huggett
  • GWP-TSan: Zero-Cost Detection of Data Races in Production – M. Morehouse; K. Serebryany
  • CompilerInvocation to -cc1 command line – D. Grumberg
  • Outer-Loop Vectorization Legality Analysis for RV: One Step Closer to a Powerful Vectorizer for LLVM – S. Baziotis
  • Flang Update – S. Scalpone
  • Code Feature Analysis, Tracking, and Future Usage – T. Jayatilaka; J. Doerfert; G. Georgakoudis; E. Park; H. Ueno; S. Badruswamy
  • Lowering XLA HLO using RISE – A Functional Pattern-based MLIR Dialect – M. Lücke; A. Smith; M. Steuwer
  • SYCL for CUDA: An overview of implementing PI for CUDA – A. Johnston
  • Extending LLDB to More Scripting Languages – J. Devlieghere
  • Adding a Subtarget Support to LLVM in Five Minutes – E. Yakubova

Birds of a Feather:

  • ClangBuiltLinux BoF – N. Desaulniers
  • Loop Optimization BoF – M. Kruse; K. Barton
  • LLVM Just-In-Time Compilers BoF – L. Hames
  • Code Size Optimization – S. Bartell; V. Adve

Student Research Competition

  • Enzyme: High-Performance Automatic Differentiation of LLVM – W. Moses; V. Churavy
  • SPAM: Stateless Permutation of Application Memory with LLVM – M. Ziad; M. Arroyo; S. Sethumadhavan
  • HPVM-FPGFA: Leveraging Compiler Optimizations for Hardware-Agnostic FPGA Programming – A. Ejjeh; K. Kanwar; M. Kotsifakou; V. Adve; R. Rutenbar
  • Guided Linking: shrinking and speeding up dynamically linked code – S. Bartell; V. Adve
  • ApproxTuner: A Compiler and Runtime System for Adaptive Approximations – H. Sharif; M. Kotsifakou; Y. Zhao; A. Kothari; B. Schreiber; E. Wang; Y. Sarita; N. Zhao; K. Joshi; V. Adve; S. Misailovic; S. Adve

Posters:

  • CIRCT: MLIR for Hardware Design – S. Neuendorffer; C. Lattner; A. Wilson
  • An Approach to Generate Correctly Rounded Math Libraries for New Floating Point Variants – J. Lim; M. Aanjaneya; J. Gustafson; S. Nagarakatte
  • Compiling a Higher-Order Smart Contract Language to LLVM – V. Nagaraj; J. Johannsen; A. Trunov; G. Pirlea; A. Kumar; I. Sergey
  • To -jInfinity & Beyond – W. Moses; K. Kwok; L. Sha
  • llvm-diva – Debug Information Visual Analyzer – C. Enciso
  • Quickly Finding RISC-V Code Quality Issues with Differential Analysis – L. Marques
  • Error estimates of floating-point numbers and Jacobian matrix computation in Clad – V. Vassilev; A. Penev; R. Shakhov
  • Data Dependency using MSSA: Analysis and Contrast – R. Sharma
  • Connecting Clang to The ROSE Source-to-Source Compiler – A. Wang; P. Lin; C. Liao; Y. Yan
  • Incremental Compilation Support in Clang – V. Vassilevv; D. Lange

Announcing the new LLVM Foundation Boardhttps://blog.llvm.org/posts/2020-08-21-new-llvm-foundation-board/Fri, 21 Aug 2020 17:54:06 -0700https://blog.llvm.org/posts/2020-08-21-new-llvm-foundation-board/

The LLVM Foundation is pleased to announce its new Board of Directors, which includes:

  • Kit Barton
  • Kristof Beyls
  • Mike Edwards (Treasurer)
  • Hal Finkel
  • Cyndy Ishida
  • Anton Korobeynikov
  • Tanya Lattner (President)
  • Chris Lattner
  • Tom Stellard (Secretary)

Three new members and six continuing members were elected to the nine person board. Thank you to retiring board members Chandler Carruth, Arnaud de Grandmaison, and John Regehr for all of their contributions to the board.

We were pleased to have many qualified applicants to the Board of Directors this year, which enabled us to make the board stronger than ever. Unfortunately, this also meant that we could not include everyone that we wanted to. Our goal was to create a balanced board of individuals from a wide range of backgrounds and locations, and to provide a voice to many groups within the LLVM community. This is always a challenge with such a large and vibrant community, and such a small board.

About the board of directors (listed alphabetically by last name):

Kit Barton:

Kit Barton has been contributing to LLVM since 2015. His contributions have primarily been to the PowerPC backend and loop optimizations including the loop fusion pass. He has presented multiple technical talks, and tutorials at the
LLVM Dev conferences over the last two years.

In addition to the contributions to LLVM, over the last two years Kit has driven the effort within IBM to migrate their proprietary C/C++ and Fortran compilers to leverage LLVM technology. He has also been involved in organizing the LLVM Meetups in Toronto as well as the Loop Optimization Working Group.

Kit is currently the technical lead for C/C++ and Fortran compilers on POWER and z/OS at IBM.

Kristof Beyls:

Kristof Beyls has worked on LLVM since about 2010, initially as part of tech leading the migration of Arm’s C/C++ toolchain to be based on LLVM technology. Since then, Kristof has worked on a large number of code generation projects using LLVM. He has contributed to LLVM in the areas of security mitigations, performance tuning, Arm backends, test-suite, LNT, etc.

He has been helping with the organization of EuroLLVM meetings since the start; has been organizing the FOSDEM LLVM dev rooms for the past couple of years and has organized a few socials in Belgium. He has also been on the program committee for a few of the dev meetings.

Kristof is Senior Principal Engineer at Arm.

Mike Edwards:

Mike Edwards has been involved with the LLVM Project since 2016 and has been most active behind the scenes working on infrastructure related issues. Mike joined the LLVM Foundation Board in 2018 and was elected Treasurer. Mike has used the past two years to help further the Foundations programs and support the many efforts of the Foundation to reach new users of LLVM technologies. Mike is looking forward to working with the new Board Members elected this year to help further the program development and outreach of the Foundation.

Mike is currently working as a Software Engineer at Apple, Inc. working on the Continuous Integration and Quality Engineering efforts for LLVM and Clang development.

Hal Finkel:

Hal Finkel has been an active contributor to the LLVM project since 2011. He is the code owner for the PowerPC target, the alias-analysis infrastructure, and other components.

In addition to his numerous technical contributions, Hal has chaired the LLVM in HPC workshop, which is held in conjunction with Super Computing (SC), starting in 2014. This workshop provides a venue for the presentation of peer-reviewed HPC-related researching LLVM from both industry and academia. He has also been involved in organizing an LLVM-themed BoF session at SC and LLVM socials, in addition to organizing community technical calls for Flang and aliasing analysis.

Hal is Lead for Compiler Technology and Programming Languages at Argonne National Laboratory’s Leadership Computing Facility. His team at Argonne works on several LLVM-based projects. Hal also teaches a compilers course at the University of Chicago’s Masters Program in Computer Science.

Cyndy Ishida:

Cyndy Ishida is relatively new to the LLVM community. She began her involvement by contributing Mach-O Support to TextAPI in the past year, which serves as a condensed textual representation of dynamic libraries from a static linking perspective.

In addition to open source contributions, Cyndy has participated in the program committee for the 2020 US LLVM Developers’ Meeting and in the 2019 US Women in Compilers and Tools Workshop. With a long standing admiration for the LLVM Project and Foundation, Cyndy is passionate about supporting and expanding the community. She is excited to use her position to aid in educational outreach efforts as a driver to grow the set of diverse developers that make up the open source community, and is focused on advocating for inclusivity in the LLVM project.

Cyndy is a Compiler Engineer at Apple, Inc. concentrating on enhancing library support with Clang tooling.

Anton Korobeynikov:

Anton Korobeynikov has been an active contributor to the LLVM project since 2006. Over the years, he has numerous technical contributions to areas including Windows support, ELF features, debug info, exception handling, and backends such as ARM and x86. He was the original author of the MSP430 and original System Z backend.

In addition to his technical contributions, Anton has maintained LLVM’s participation in Google Summer of Code by managing applications, deadlines, and overall organization. He also supports the LLVM infrastructure and has been on numerous program committees for the LLVM Developers’ Meetings (both US and EuroLLVM).

Anton is currently an associate professor at the Saint Petersburg State University and has served on the LLVM Foundation board of directors for the last 6 years.

Tanya Lattner:

Tanya Lattner has been involved in the LLVM project for over 16 years. She began as a graduate student who wrote her master’s thesis using LLVM, and continued on using and extending LLVM technologies at various jobs during her career as a compiler engineer.

Tanya has been organizing the US LLVM Developers’ meeting since 2008 and attended every developer meeting. She was the LLVM release manager for 3 years, moderates the LLVM mailing lists, and helps administer the LLVM infrastructure servers, mailing lists, bugzilla, etc. Tanya has also been on the program committee for the US LLVM Developers’ meeting (4 years) and the EuroLLVM Developers’ Meeting (1 year).

With the support of the initial board of directors, Tanya created the LLVM Foundation, defined its charitable and education mission, and worked to get 501(c)(3) status. She is passionate about the LLVM Community and wants to help see it thrive and grow for years to come.

Tanya is the Chief Operating Officer and has served as the President of the LLVM Foundation board for the last 6 years.

Chris Lattner:

Chris Lattner is the founder of the LLVM project and has a lengthy history of technical contributions to the project over the years. He drove much of the early implementation, architecture, and design of LLVM, Clang, and MLIR. Chris actively participates in the LLVM Developers’ meeting, served on the LLVM Board of Directors for many years, and helps drive important discussions and policy decisions related to the LLVM project.

Outside of LLVM, Chris has served in a wide range of technical leadership positions at Apple, Tesla, Google, and SiFive. These have spanned domains including compiler infrastructure, developer tools in general, machine learning infrastructure, autonomous vehicles, and microprocessor design. More details are available in his resumé page.

Tom Stellard:

Tom Stellard has been contributing to the LLVM project since 2012. He was the original author of the AMDGPU backend and was also an active contributor to libclc. He has been the LLVM project’s stable release manager since 2014.

Tom is currently a Software Engineer at Red Hat and is the technical lead for emerging toolchains including Clang/LLvm. He also maintains the LLVM packages for the Fedora
project.

The New Clang _ExtInt Feature Provides Exact Bitwidth Integer Typeshttps://blog.llvm.org/2020/04/the-new-clang-extint-feature-provides.htmlTue, 21 Apr 2020 17:13:00 +0000https://blog.llvm.org/2020/04/the-new-clang-extint-feature-provides.html
Author: Erich Keane, Compiler Frontend Engineer, Intel Corporation

Earlier this month I finally committed a patch to implement the extended-integer type class, _ExtInt after nearly two and a half years of design and implementation. These types allow developers to use custom width integers, such as a 13-bit signed integer. This patch is currently designed to track N2472, a proposal being actively considered by the ISO WG14 C Language Committee. We feel that these types are going to be extremely useful to many downstream users of Clang, and provides a language interface for LLVM’s extremely powerful integer type class.

Motivation

LLVM-IR has the ability to represent integers with a bitwidth from 1 all the way to 16,777,215((1<<24)-1), however the C language is limited to just a few power-of-two sizes. Historically, these types have been sufficient for nearly all programming architectures, since power-of-two representation of integers is convenient and practical.

Recently, Field-Programmable Gate Array (FPGA) tooling, called High Level Synthesis Compilers (HLS), has become practical and powerful enough to use a general purpose programming language for their generation. These tools take C or C++ code and produce a transistor layout to be used by the FPGA. However, once programmers gained experience in these tools, it was discovered that the standard C integer types are incredibly wasteful for two main reasons.

First, a vast majority of the time programmers are not using the full width of their integer types. It is rare for someone to use all 16, 32, or 64 bits of their integer representation. On traditional CPUs this isn’t much of a problem as the hardware is already in place, so having bits never set comes at zero cost. On the other hand, on FPGAs logic gates are an incredibly valuable resource, and HLS compilers should not be required to waste bits on large power of two integers when they only need a small subset of that! While the optimizer passes are capable of removing some of these widths, a vast majority of this hardware needs to be emitted.

Second, the C language requires that integers smaller than int are promoted to operations on the ‘int’ type. This further complicates hardware generation, as promotions to int are expensive and tend to stick with the operation for an entire statement at a time. These promotions typically have semantic meaning, so simply omitting them isn’t possible without changing the meaning of the source code. Even worse, the proliferation of auto has resulted in user code results in the larger integer size being quite viral throughout a program.

The result is massively larger FPGA/HLS programs than the programmer needed, and likely much larger than they intended. Worse, there was no way for the programmer express their intent in the cases where they do not need the full width of a standard integer type.

Using the _ExtInt Language Feature

The patch as accepted and committed into LLVM solves most of the above problems by providing the _ExtInt class of types. These types translate directly into the corresponding LLVM-IR integer types. The _ExtInt keyword is a type-specifier (like int) that accepts a required integral constant expression parameter representing the number of bits to be used. More succinctly: _ExtInt(7) is a signed integer type using 7 bits. Because it is a type-specifier, it can also be combined with signed and unsigned to change the signedness (and overflow behavior!) of the values. So “unsigned _ExtInt(9) foo;” declares a variable foo that is an unsigned integer type taking up 9 bits and represented as an i9 in LLVM-IR.

The _ExtInt types as implemented do not participate in any implicit conversions or integer promotions, so all math done on them happens at the appropriate bit-width. The WG14 paper proposes integer promotion to the largest of the types (that is, adding an _ExtInt(5) and an _ExtInt(6) would result in an _ExtInt(6)), however the implementation does not permit that and _ExtInt(5) + _ExtInt(6) would result in a compiler error. This was done so that in the event that WG14 changes the design of the paper, we will be able to implement it without breaking existing programs. In the meantime, this can be worked around with explicit casts: (_ExtInt(6))AnExtInt5 + AnExtInt6 or static_cast<ExtInt(6)>(AnExtInt5) + AnExtInt6.

Additionally, for C++, clang supports making the bitwidth parameter a dependent expression, so that the following is legal:
template<size_t WidthA, size_t WidthB>
  _ExtInt(WidthA + WidthB) lossless_mul(_ExtInt(WidthA) a, _ExtInt(WidthB) b) {
  return static_cast<
_ExtInt(WidthA + WidthB)>(a) 
       * static_cast<_ExtInt(WidthA + WidthB)>(b);

We anticipate that this ability and these types will result in some extremely useful pieces of code, including novel uses of 256 bit, 512 bit, or larger integers, plus uses of 8 and 16 bit integers for those who can’t afford promotions. For example, one can now trivially implement an extended integer type struct that does all operations provably losslessly, that is, adding two 6 bit values would result in a 7 bit value.

In order to be consistent with the C Language, expressions that include a standard type will still follow integral promotion and conversion rules. All types smaller than int will be promoted, and the operation will then happen at the largest type.  This can be surprising in the case where you add a short and an _ExtInt(15), where the result will be int. However, this ends up being the most consistent with the C language specification.

Additionally, when it comes to conversions, these types ‘lose’ to the C standard types of the same size or greater. So, an int added to a _ExtInt(32) will result in an int. However, an int and a _ExtInt(33)will be the latter. This is necessary to preserve C integer semantics.

History

As mentioned earlier, this feature has been a long time coming! In fact, this is likely the fourth implementation that was done along the way in order to get to this point. Additionally, this is far from over, we very much hope that upon acceptance of this by the WG14 Standards Committee that additional extensions and features will become available.

I was approached to implement this feature in the Fall of 2017 by my company’s FPGA group, which had the problems mentioned above. They had attempted a solution that used some clever parsing to make these look like templates, and implemented them extensively throughout the compiler. As I was concerned about the flexibility and usability of these types in the type and template system, we opted to implement these as a type-attribute under the controversially named Arbitrary Precision Int (spelled __ap_int). This spelling was heavily influenced by the vector-types implementations in GCC and Clang.

We then were able to wrap a set of typedefs (or dependent __ap_int types) in a structure that provided exactly the C and C++ interface we wished to expose. As this was a then proprietary implementation, it was kept in our downstream implementation, where it received extensive testing and usage.

Roughly a year later (and a little more than year ago from today!) I was authorized to contribute our implementation to the open source LLVM community! I decided to significantly refactor the implementation in order to better fit into the Clang type system, and uploaded it for review.This (now third!) implementation of this feature was proposed via RFC and code review at the same time.

While the usefulness was immediately acknowledged, it was rejected by the Clang code owner for two reasons: First the spelling was considered unpalatable, and Second it was a pure extension without standardization. This began the nearly year-long effort to come up with a standards proposal that would better define and describe the feature as well as come up with a spelling that was more in line with the standard language.

Thanks to the invaluable feedback and input from Richard Smith, my coworkers Melanie Blower, Tommy Hoffner, and myself were able to propose the spelling _ExtInt for standardization. Additionally, the feature again re-implemented at the beginning of this year and eventually accepted and committed!

The standardization paper (N2472) was presented at this Spring’s WG14 ISO C Language Committee Meeting where it received near unanimous support. We expect to have an updated version of the paper with wording ready for the next WG14 meeting, where we hope it will receive sufficient support to be accepted into the language.

Future Extensions

While the feautre as committed in Clang is incredibly useful, it can be taken further. There are a handful of future extensions that we wish to implement once guidance from WG14 has been given on their direction and implementation.

First, we believe the special integer promotion/conversion rules, which omit automatic promotion to  int and instead provide operations at the largest type are both incredibly useful and powerful. While we have received positive encouragement from WG14, we hope that the wording paper we provide will both clarify the mechanism and definition in a way that supports all common uses.

Secondly, we would like to choose a printf/scanf specifier that permits specifying the type for the C language. This was the topic of the WG14 discussion, and also received strong encouragement. We intend to come up with a good representation, then implement this in major implementations.

Finally, numerous people have suggested implementing a way of spelling literals of this type. This is important for two reasons: First, it allows using literals without casts in expressions in a way that doesn’t run afoul of promotion rules. Second, it provides a way of spelling integer literals larger than UINTMAX_MAX, which can be useful for initializing the larger versions of these types. While the spelling is undecided, we intend something like: 1234X would result in an integer literal with the value 1234 represented in an _ExtInt(11), which is the smallest type capable of storing this value.

However, without the integer promotion/conversion rules above, this feature isn’t nearly as useful. Additionally, we’d like to be consistent with whatever the C language committee chooses. As soon as we receive positive guidance on the spelling and syntax of this type, we look forward to providing an implementation.

Conclusion

In closing, we encourage you to try using this and provide feedback to both myself, my proposal co-authors, and the C committee itself! We feel this is a really useful feature and would love to get as much user experience as possible. Feel free to contact myself and my co-authors with any questions or concerns!

Erich Keane, Intel CorporationDeterministic builds with clang and lldhttps://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.htmlThu, 07 Nov 2019 12:34:00 +0000https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html

Deterministic builds can lower continuous integration costs and give you more confidence in your build and test process. This post outlines what it means for a build to be deterministic, the advantages of deterministic builds, and how to achieve them using LLVM tools.

What is a deterministic build, and its advantages

A build is called deterministic or reproducible if running it twice produces exactly the same build outputs.

There are several degrees of build determinism that are increasingly useful but increasingly difficult to achieve:

  1. Basic determinism: Doing a full build of the same source code in the same directory on the same machine produces exactly the same output every time, in the sense that a content hash of the final build artifacts and of all intermediate files does not change.
    • Once you have this, if all your builders are configured the same way (OS version, toolchain, build path, checkout path, …), they can share build artifacts, for example by using distcc.
    • This also allows local caching of test suite results keyed by a hash of test binary and test input files.
    • Illustrative example: ./build src out ; mv out out.old ; ./build src out ; diff -r out out.old
  2. Incremental basic determinism: Like basic determinism, but the output binaries also don’t change in partial rebuilds. In build systems that track file modification times to decide when to rebuild, this means for example that updating the modification time on a C++ source file (without doing any actual changes) and rebuilding will produce the same output as a full build.
    • This allows having build bots that don’t do full builds each time, while still allowing caching of compile artifacts and test results.
    • Illustrative example: ./build src out ; cp -r out out.old ; touch src/foo.c ; ./build src out ; diff -r out out.old
  3. Local determinism: Like incremental basic determinism, but builds are also independent of the name of the build directory. Builds of the same source code on the same machine produce exactly the same output every time, independent of the location of the source checkout directory or the build directory.
    • This allows machines to have several build directories at different locations but still share compile and test caches.
    • Illustrative example: cp -r src src2 ; ./build src out ; ./build src2 out2 ; diff -r out out2
  4. Universal determinism: Like 3, but builds are also independent of the machine the build runs on. Everybody that checks out the project at a given revision into any directory and builds it following the build instructions ends up with exactly the same bits in the build output.
    • Since exact local OS and locally installed packages no longer matter, this allows devs to share compile and test caches with bots, without having to use difficult-to-setup containers.
    • It also allows easy verification of builds done by others to make sure output binaries haven’t been tampered with.
    • Illustrative example: ./build src out ; ssh remote ./build src out && scp remote:out out2 ; diff -r out out2

Plan of attack

To make sure that a deterministic build stays deterministic, you should set up a builder that verifies that your build is deterministic. Even if your build isn’t deterministic yet, you can set up a bot that verifies that some parts of your build are deterministic and then expand the checks over time.

For example, you could have a bot that does a full build in a fixed build directory, then moves the build artifacts out of the way, and does another full build, and once your compiles have basic determinism, add a step that checks that object files between the two builds directories are the same. You could even add incremental checking for specific subdirectories or build targets while you work towards full basic determinism.

Once your links are deterministic, check that binaries are identical as well. Once all your build steps are deterministic, compare all files in the two build directories.

Once your build has incremental determinism, do an incremental build for the first build and a full build for the second build. Once your build has local determinism, do the two builds at different build paths.

Getting to basic determinism

Basic determinism needs tools (compiler, linker, etc) that are deterministic. Tools internally must not output things in hash table order, multi-threaded programs must not write output in the order threads finish, etc. All of LLVM’s tools have deterministic outputs when run with the right flags but not necessarily by default.

The C standard defines the predefined macros __TIME__ and __DATE__ that expand to the time a source file is compiled. Several compilers, including clang, also define the non-standard __TIMESTAMP__. This is inherently nondeterministic. You should not use these macros, and you can use -Wdate-time to make the compiler emit a warning when they are used.

If they are used in third-party code you don’t control, you can use -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= to make them expand to nothing.

When targeting Windows, clang and clang-cl by default also embed the current time in a timestamp field in the output .obj file, because Microsoft’s link.exe in /incremental mode silently mislinks files if that field isn’t set correctly. If you don’t use link.exe’s /incremental flag, or if you link with lld-link, you should pass /Brepro to clang-cl to make it not write the current timestamp into its output.

Both link.exe and lld-link also write the current timestamp into output .dll or .exe files. To make them instead write a hash of the binary into this field, you can pass /Brepro to the linker as well. However, some tools, such as Windows 7’s app compatibility database, try to interpret that field as an actual timestamp and can get confused if it’s set to a hash of the binary. For this case, lld-link also offers a /timestamp: flag that you can give an explicit timestamp that’s written into the output. You could use this to for example write the time of the commit the code is built at instead of the current time to make it deterministic. (But see the footnote on embedding commit hashes below.)

Visual Studio’s assemblers ml.exe and ml64.exe also insist on writing the current time into their output. In situations like this, where you can’t easily fix the tool to write the right output in the first place, you need to write wrappers that fix up the file after the fact. As an example, ml.py is the wrapper the Chromium project uses to make ml’s output deterministic.

macOS’s libtool and ld64 also insist on writing timestamps into their outputs. You can set the environment variable ZERO_AR_DATE to 1 in a wrapper to make their output deterministic, but that confuses lldb of older Xcode versions.

Gcc sometimes uses random numbers in certain symbol mangling situations. Clang does not do this, so there’s no need to pass -frandom-seed to clang.

It’s a good idea to make your build independent of environment variables as much as possible, so that accidental local changes in the environment don’t affect the build output. You should pass /X to clang-cl to make it ignore %INCLUDE% and explicitly pass system include directories via the -imsvc switch instead. Likewise, very new lld-link versions (LLVM 10 and newer, at the time of this writing still unreleased) understand the flag /lldignoreenv flag, which makes lld-link ignore the %LIB% environment variable; explicitly pass system library directories via /libpath:.

Footnote on embedding git hashes into the binary
It might be tempting to embed the git commit hash or svn revision that a binary was built at into the binary’s –version output, or use the revision as a cache key to invalidate on-disk caches when the version changes.

This doesn’t affect your build’s determinism, but it does affect the hit rate if you’re using deterministic builds to cache test run results. If your binary embeds the current commit, it is guaranteed to change on every single commit, and you won’t be able to cache test results across commits. Even commits that just fix typos in comments, add non-code documentation, or that only affect code used by some but not all of your binaries will change every binary.

For cache invalidation, consider using something finer-grained, such as only the latest commit of the directory containing the cache handling code, or the hash of all source files containing the cache handling code.

For –version output, if your build is fully deterministic, the hash of the binary itself (and its dynamic library dependencies) can serve as a stable version identifier. You can keep a map of binary hash to all commit hashes that produce that binary somewhere.

Windows only: For the same reason, just using the timestamp of the latest commit as a /timestamp: might not be the best option. Rounding the timestamp of the latest commit to 6h (or similar) granularity is a possible approach for not having the timestamp change the binary on every commit, while still keeping the timestamp close to reality. For production builds, the symbol server key for binaries is a (executable size, timestamp) pair, so here having fairly granular timestamps is important to not map binaries from consecutive commits to the same symbol server key. Depending on how often you push production binaries to your symbol servers, you might want to use the timestamp of the latest commit as /timestamp: for official builds, or you might want to round to finer granularity than you do on dev builds.

Getting to incremental determinism

Having deterministic incremental builds mostly requires having correct incremental builds, meaning that if a file is changed and the build reruns, everything that uses this file needs to be rebuilt.

This is very build system dependent, so this post can’t say much about it.

In general, every build step needs to correctly declare all the inputs it depends on.

Some tools, such as Visual Studio’s link.exe in /incremental mode, by design write a different output every time. Don’t use inherently incrementally non-deterministic tools like that if you care about build determinism.

The build should not depend on environment variables, since build systems usually don’t model dependencies on environment variables.

Getting to local determinism

Making build outputs independent of the names of the checkout or build directory means that build outputs must not contain absolute paths, or relative paths that contain the name of either directory.

A possible way to arrange for that is to put all build directories into the checkout directory. For example, if your code is at path/to/src, then you could have “out” in your .gitignore and build directories at path/to/src/out/debug, path/to/src/out/release, and so on. The relative path from each build artifact to the source is with “../../” followed by the path of the source file in the source directory, which is identical for each build directory.

The C standard defines the predefined macro __FILE__ that expands to the name of the current source file. Clang expands this to an absolute path if it is invoked with an absolute path (`clang -c /absolute/path/to/my/file.cc`), and to a relative path if it is invoked with a relative path (`clang ../../path/to/my/file.cc`). To make your build locally deterministic, pass relative paths to your .cc files to clang.

By default, clang will internally use absolute paths to refer to compiler-internal headers. Pass -no-canonical-prefixes to make clang use relative paths for these internal files.

Passing relative paths to clang makes clang expand __FILE__ to a relative path, but paths in debug information are still absolute by default. Pass -fdebug-compilation-dir . to make paths in debug information relative to the build directory. (Before LLVM 9, this is an internal clang flag that must be used as `-Xclang -fdebug-compilation-dir -Xclang .`) When using clang’s integrated assembler (the default), -Wa,-fdebug-compilation-dir,. will do the same for object files created from assembly input. (For ml.exe / ml64.exe, see the script linked to from the “Basic determinism” section above.)

Using this means that debuggers won’t automatically find the source code belonging to your binary. At the moment, there’s no way to tell debuggers to resolve relative paths relative to the location of the binary (DWARF proposal, gdb patch). See the end of this section for how to configure common debuggers to work correctly.

There are a few flags that try to make compilers produce relative paths in outputs even if the filename passed to the compiler is absolute (-fdebug-prefix-map, -ffile-prefix-map, -fmacro-prefix-map). Do not use these flags.

  • They work by adding lhs=rhs replacement patterns, and the lhs must be an absolute path to remove the absolute path from the output. That means that while they make the compile output path-independent, they make the compile command itself path-dependent, which hinders distributed compile caching. With -grecord-gcc-switches or -frecord-gcc-switches the compile command is embedded in debug info or even the object file itself, so in that case the flags even break local determinism. (Both -grecord-gcc-switches and -frecord-gcc-switches default to false in clang.)
  • They don’t affect the paths in dwo files when using fission; passing relative paths to the compiler is the only way to make these paths relative.

On Windows, it’s very unusual to have PDBs with relative paths. You can pass /pdbsourcepath:X:fakeprefix to lld-link to make it resolve all relative paths in object files against a fixed absolute path to make sure your final PDBs contain absolute paths. Since the absolute path is against a fixed prefix, this doesn’t impair determinism. With this, both binaries and PDBs created by clang-cl and lld-link will be fully deterministic and build path independent.

Also on Windows, the linker by default puts the absolute path the to the generated PDB file in the output binary. Pass /pdbaltpath:%_PDB% when you pass /debug to make the linker write a relative path to the generated PDB file instead. If you have custom build steps that extract PDB names from binaries, you have to make sure these scripts work with relative paths. Microsoft’s tools (debuggers, ETW) work fine with this set in most situations, and you can add a symbol search path in the cases where they don’t (when the binaries are copied before being run).

Getting debuggers to work well with locally deterministic builds
At the moment, no debugger offers an option to resolve relative paths in debug info against the directory the debugged binary is in.

Some debuggers (gdb, lldb) do try to resolve relative paths against the cwd, so a simple way to make debugging work is to cd into your build directory before debugging.

If you don’t want to require devs to cd into the build directory for debugging to work, you have to do debugger-specific configuration tweaks.

To make sure devs don’t miss this, you could have your custom init script set an env var and query if it’s set early during your test binary startup, and exit with a message like “Add `source /path/to/your/project/gdbinit` to your ~/.gdbinit” if the environment variable isn’t set.

gdb
`dir path/to/build/dir` tells gdb what directory to resolve relative paths against.

`show debug-file-directory` prints the list of directories gdb looks in for dwo files. Query that, append `:path/to/build/dir`, and call `set debug-file-directory` to add your build dir to that search path.

For an example, see Chromium’s gdbinit (which also does a few other unrelated things).

lldb
`settings set target.source-map ../.. /absolute/path/to/build/dir` can map the “../..” prefix that all .cc files will refer to when using the setup described above with an absolute path. This requires Xcode 10.3 or newer; the lldb shipping with Xcode 10.1 has problems with this setup.

For an example, see Chromium’s lldbinit.

Visual Studio’s debugger and windbg
If you use the setup described above,  /PDBSourcePath:X:fakeprefix will combine with the “….myfile.cc” relative paths to make your code appear at “X:myfile.cc”. To make Windows debuggers find them, you have two options:

  1. Run `subst X: C:srcrealroot` in cmd.exe before launching the debuggers to create a virtual drive that maps X: to the actual source location. Both windbg and Visual Studio will load code over X: this way.
  2. Add “C:srcrealroot” to each debugger’s source search path.
    • Windbg: Run `.srcpath+ C:srcrealroot`. You can also set this via the _NT_SOURCE_PATH  environment variable, or via  File->Source File Path (Ctrl+P). Or pass `-srcpath C:srcrealroot` when launching windbg from the command line.
    • Visual Studio: The IDE has a “Debug Source Files” property. Add C:srcrealroot to “Directories containing source code” to Project->Properties (Alt+F7)->Common Properties->Debug Source Files->Directories containing source code.

Alternatively, you could pass the absolute path to the actual build directory to /PDBSourcePath: instead of something like “X:fakeprefix”. That way, all PDBs have “correct” absolute paths in them, while your compile steps are still path-independent and can share a cache across machines. However, since executables contain a reference to the PDB hash, none of your binaries will be path-independent. This setup doesn’t require any debugger configuration, but it doesn’t allow your builds to be locally deterministic.

Getting to universal determinism

By now, your build output is deterministic as long as everyone uses the same compiler, and linker binaries, and as long as everyone uses the version of the SDK and system libraries.

Making your build independent of that requires making sure that everyone automatically uses the same compiler, linker, and SDK.

This might seem like a lot of work, but in addition to build determinism this work also gives you cross builds (where you can e.g. build the Linux version of your product on a Windows host).

It also versions the compiler, linker, and SDK used within your code, which means you will be able to update all your bots and devs to new versions automatically (and if an update causes issues, it’s easy to revert it).

You need to store the currently-used compiler, linker, and SDK versions in a file in your source control repository, and from some kind of hook that runs after pulling the newest version of the source, download compiler, linker, and SDK of the right version from some kind of cloud storage service.

You then need to modify your build files to use –sysroot (Linux), -isysroot (macOS), -imsvc (Windows) to use these hermetic SDKs for builds. They need to be somewhere below your source root to not regress build directory name invariance.

You also want to make sure your build doesn’t depend on environment variables, as already mentioned in the “Getting to incremental determinism”, since environments between different machines can be very different and difficult to control.

Build steps shouldn’t embed the hostname of the current machine or the logged-in user name in the build output, or similar.

Summary

This post explained what deterministic builds are, how build determinism spans a spectrum (local, fixed-build-dir-path-only to fully host-OS-independent) instead of just being binary, and how you can use LLVM’s tools to make your build deterministic. It also touched on techniques you can use to make your test caches more effective.

Thanks to Elly Fong-Jones for helping edit and structure this post, and to Adrian McCarthy, Bob Haarman, Bruce Dawson, Dirk Pranke, Fumitoshi Ukai, Hans Wennborg, Kai Naschinski, Reid Kleckner, Rui Ueyama, and Takuto Ikuta for reading drafts and suggesting improvements.

Closing the gap: cross-language LTO between Rust and C/C++https://blog.llvm.org/2019/09/closing-gap-cross-language-lto-between.htmlThu, 19 Sep 2019 05:15:00 +0000https://blog.llvm.org/2019/09/closing-gap-cross-language-lto-between.html
Link time optimization (LTO) is LLVM’s way of implementing whole-program optimization. Cross-language LTO is a new feature in the Rust compiler that enables LLVM’s link time optimization to be performed across a mixed C/C++/Rust codebase. It is also a feature that beautifully combines two respective strengths of the Rust programming language and the LLVM compiler platform:

  • Rust, with its lack of a language runtime and its low-level reach, has an almost unique ability to seamlessly integrate with an existing C/C++ codebase, and
  • LLVM, as a language agnostic foundation, provides a common ground where the source language a particular piece of code was written in does not matter anymore.

So, what does cross-language LTO do? There are two answers to that:

  • From a technical perspective it allows for codebases to be optimized without regard for implementation language boundaries, making it possible for important optimizations, such as function inlining, to be performed across individual compilation units even if, for example, one of the compilation units is written in Rust while the other is written in C++.
  • From a psychological perspective, which arguably is just as important, it helps to alleviate the nagging feeling of inefficiency that many performance conscious developers might have when working on a piece of software that jumps back and forth a lot between functions implemented in different source languages.

Because Firefox is a large, performance sensitive codebase with substantial parts written in Rust, cross-language LTO has been a long-time favorite wish list item among Firefox developers. As a consequence, we at Mozilla’s Low Level Tools team took it upon ourselves to implement it in the Rust compiler.

To explain how cross-language LTO works it is useful to take a step back and review how traditional compilation and “regular” link time optimization work in the LLVM world.

Background – A bird’s eye view of the LLVM compilation pipeline

Clang and the Rust compiler both follow a similar compilation workflow which, to some degree, is prescribed by LLVM:

  1. The compiler front-end generates an LLVM bitcode module (.bc) for each compilation unit. In C and C++ each source file will result in a single compilation unit. In Rust each crate is translated into at least one compilation unit.

    .c --clang--> .bc

    .c --clang--> .bc


    .rs --+
    |
    .rs --+--rustc--> .bc
    |
    .rs --+

  2. In the next step, LLVM’s optimization pipeline will optimize each LLVM module in isolation:

    .c --clang--> .bc --LLVM--> .bc (opt)

    .c --clang--> .bc --LLVM--> .bc (opt)


    .rs --+
    |
    .rs --+--rustc--> .bc --LLVM--> .bc (opt)
    |
    .rs --+

  3. LLVM then lowers each module into machine code so that we get one object file per module:

    .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o

    .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o


    .rs --+
    |
    .rs --+--rustc--> .bc --LLVM--> .bc (opt) --LLVM--> .o
    |
    .rs --+

  4. Finally, the linker will take the set of object files and link them together into a binary:

    .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o ------+
    |
    .c --clang--> .bc --LLVM--> .bc (opt) --LLVM--> .o ------+
    |
    +--ld--> bin
    .rs --+ |
    | |
    .rs --+--rustc--> .bc --LLVM--> .bc (opt) --LLVM--> .o --+
    |
    .rs --+

This is the regular compilation workflow if no kind of LTO is involved. As you can see, each compilation unit is optimized in isolation. The optimizer does not know the definition of functions inside of other compilation units and thus cannot inline them or make other kinds of decisions based on what they actually do. To enable inlining and optimizations to happen across compilation unit boundaries, LLVM supports link time optimization.

Link time optimization in LLVM

The basic principle behind LTO is that some of LLVM’s optimization passes are pushed back to the linking stage. Why the linking stage? Because that is the point in the pipeline where the entire program (i.e. the whole set of compilation units) is available at once and thus optimizations across compilation unit boundaries become possible. Performing LLVM work at the linking stage is facilitated via a plugin to the linker.

Here is how LTO is concretely implemented:

  • the compiler translates each compilation unit into LLVM bitcode (i.e. it skips lowering to machine code),
     
  • the linker, via the LLVM linker plugin, knows how to read LLVM bitcode modules like regular object files, and
     
  • the linker, again via the LLVM linker plugin, merges all bitcode modules it encounters and then runs LLVM optimization passes before doing the actual linking.

With these capabilities in place a new compilation workflow with LTO enabled for C++ code looks like this:


.c --clang--> .bc --LLVM--> .bc (opt) ------------------+ - - +
| |
.c --clang--> .bc --LLVM--> .bc (opt) ------------------+ - - +
| |
+-ld+LLVM--> bin
.rs --+ |
| |
.rs --+--rustc--> .bc --LLVM--> .bc (opt) --LLVM--> .o -+
|
.rs --+

As you can see our Rust code is still compiled to a regular object file. Therefore, the Rust code is opaque to the optimization taking place at link time. Yet, looking at the diagram it seems like that shouldn’t be too hard to change, right?

Cross-language link time optimization

Implementing cross-language LTO is conceptually simple because the feature is built on the shoulders of giants. Since the Rust compiler uses LLVM all the important building blocks are readily available. The final diagram looks very much as you would expect, with rustc emitting optimized LLVM bitcode and the LLVM linker plugin incorporating that into the LTO process with the rest of the modules:


.c --clang--> .bc --LLVM--> .bc (opt) ---------+
|
.c --clang--> .bc --LLVM--> .bc (opt) ---------+
|
+-ld+LLVM--> bin
.rs --+ |
| |
.rs --+--rustc--> .bc --LLVM--> .bc (opt) -----+
|
.rs --+

Nonetheless, achieving a production-ready implementation still turned out to be a significant time investment. After figuring out how everything fits together, the main challenge was to get the Rust compiler to produce LLVM bitcode that was compatible with both the bitcode that Clang produces and with what the linker plugin would accept. Some of the issues we ran into where:

  • The Rust compiler and Clang are both based on LLVM but they might be using different versions of LLVM. This was further complicated by the fact that Rust’s LLVM version often does not match a specific LLVM release, but can be an arbitrary revision from LLVM’s repository. We learned that all LLVM versions involved really have to be a close match in order for things to work out. The Rust compiler’s documentation now offers a compatibility table for the various versions of Rust and Clang.
     
  • The Rust compiler by default performs a special form of LTO, called ThinLTO, on all compilation units of the same crate before passing them on to the linker. We quickly learned, however, that the LLVM linker plugin crashes with a segmentation fault when trying to perform another round of ThinLTO on a module that had already gone through the process. No problem, we thought and instructed the Rust compiler to disable its own ThinLTO pass when compiling for the cross-language case and indeed everything was fine — until the segmentation faults mysteriously returned a few weeks later even though ThinLTO was still disabled.

    We noticed that the problem only occurred in a specific, presumably innocent setting: again two passes of LTO needed to happen, this time the first was a regular LTO pass within rustc and the output of that would then be fed into ThinLTO within the linker plugin. This setup, although computationally expensive, was desirable because it produced faster code and allowed for better dead-code elimination on the Rust side. And in theory it should have worked just fine. Yet somehow rustc produced symbol names that had apparently gone through ThinLTO’s mangling even though we checked time and again that ThinLTO was disabled for Rust. We were beginning to seriously question our understanding of LLVM’s inner workings as the problem persisted while we slowly ran out of ideas on how to debug this further.

    You can picture the proverbial lightbulb appearing over our heads when we figured out that Rust’s pre-compiled standard library would still have ThinLTO enabled, no matter the compiler settings we were using for our tests. The standard library, including its LLVM bitcode representation, is compiled as part of Rust’s binary distribution so it is always compiled with the settings from Rust’s build servers. Our local full LTO pass within rustc would then pull this troublesome bitcode into the output module which in turn would make the linker plugin crash again. Since then ThinLTO is turned off for libstd by default.
     

  • After the above fixes, we succeeded in compiling the entirety of Firefox with cross-language LTO enabled. Unfortunately, we discovered that no actual cross-language optimizations were happening. Both Clang and rustc were producing LLVM bitcode and LLD produced functioning Firefox binaries, but when looking at the machine code, not even trivial functions were being inlined across language boundaries. After days of debugging (and unfortunately without being aware of LLVM’s optimization remarks at the time) it turned out that Clang was emitting a target-cpu attribute on all functions while rustc didn’t, which made LLVM reject inlining opportunities.

    In order to prevent the feature from silently regressing for similar reasons in the future we put quite a bit of effort into extending the Rust compiler’s testing framework and CI. It is now able to compile and run a compatible version of Clang and uses that to perform end-to-end tests of cross-language LTO, making sure that small functions will indeed get inlined across language boundaries.

This list could still go on for a while, with each additional target platform holding new surprises to be dealt with. We had to progress carefully by putting in regression tests at every step in order to keep the many moving parts in check. At this point, however, we feel confident in the underlying implementation, with Firefox providing a large, complex, multi-platform test case where things have been working well for several months now.

Using cross-language LTO: a minimal example

The exact build tool invocations differ depending on whether it is rustc or Clang performing the final linking step, and whether Rust code is compiled via Cargo or via rustc directly. Rust’s compiler documentation describes the various cases. The simplest of them, where rustc directly produces a static library and Clang does the linking, looks as follows:


# Compile the Rust static library, called "xyz"
rustc --crate-type=staticlib -O -C linker-plugin-lto -o libxyz.a lib.rs

# Compile the C code with "-flto"
clang -flto -c -O2 main.c

# Link everything
clang -flto -O2 main.o -L . -lxyz

The -C linker-plugin-lto option instructs the Rust compiler to emit LLVM bitcode which then can be used for both “full” and “thin” LTO. Getting things set up for the first time can be quite cumbersome because, as already mentioned, all compilers and the linker involved must be compatible versions. In theory, most major linkers will work; in practice LLD seems to be the most reliable one on Linux, with Gold in second place and the BFD linker needing to be at least version 2.32. On Windows and macOS the only linkers properly tested are LLD and ld64 respectively. For ld64 Firefox uses a patched version because the LLVM bitcode that rustc produces likes to trigger a pre-existing issue this linker has with ThinLTO.

Conclusion

Cross-language LTO has been enabled for Firefox release builds on Windows, macOS, and Linux for several months at this point and we at Mozilla’s Low Level Tools team are pleased with how it turned out. While we still need to work on making the initial setup of the feature easier, it already enabled removing duplicated logic from Rust components in Firefox because now code can simply call into the equivalent C++ implementation and rely on those calls to be inlined. Having cross-language LTO in place and continuously tested will definitely lower the psychological bar for implementing new components in Rust, even if they are tightly integrated with existing C++ code.

Cross-language LTO is available in the Rust compiler since version 1.34 and works together with Clang 8. Feel free to give it a try and report any problems in the Rust bug tracker.

Acknowledgments

I’d like to thank my Low Level Tools team colleagues David Major, Eric Rahm, and Nathan Froyd for their invaluable help and encouragement, and I’d like to thank Alex Crichton for his tireless reviews on the Rust side.Announcing the program for the 2019 LLVM Developers’ Meeting – Bay Areahttps://blog.llvm.org/2019/09/announcing-program-for-2019-llvm.htmlWed, 04 Sep 2019 07:16:00 +0000https://blog.llvm.org/2019/09/announcing-program-for-2019-llvm.html
Announcing the program for the 2019 LLVM Developers’ Meeting in San Jose, CA! This program is the largest we have ever had and has over 11 tutorials, 29 technical talks, 24 lightning talks, 2 panels, 3 birds of a feather, 14 posters, and 4 SRC talks. Be sure to register to attend this event and hear some of these great talks.

Keynotes

  • Generating Optimized Code with GlobalISel – Volkan Keles, Daniel Sanders
  • Even Better C++ Performance and Productivity: Enhancing Clang to Support Just-in-Time Compilation of Templates- Hal Finkel

Technical Talks

  • Using LLVM’s portable SIMD with Zig – Shawn Landden
  • Code-Generation for the Arm M-profile Vector Extension – Sjoerd Meijer
  • Alive2: Verifying Existing Optimizations – Nuno Lopes
  • The clang constexpr interpreter – Nandor Licker
  • Souper-Charging Peepholes with Target Machine Info – Min-Yih Hsu
  • Transitioning the Networking Software Toolchain to Clang/LLVM – Ivan Baev, Jeremy Stenglein, Bharathi Seshadri
  • Link Time Optimization For Swift – Jin Lin
  • Hot Cold Splitting Optimization Pass In LLVM – Aditya Kumar
  • Making UB hurt less: security mitigations through automatic variable initialization – JF Bastien
  • Propeller: Profile Guided Large Scale Performance Enhancing Relinker – Sriraman Tallam
  • From C++ for OpenCL to C++ for accelerator devices – Anastasia Stulova
  • LLVM-Canon: Shooting for Clear Diffs – Michal Paszkowski
  • Better C++ debugging using Clang Modules in LLDB – Raphael Isemann
  • Ownership SSA and Semantic SIL – Michael Gottesman
  • arm64e: An ABI for Pointer Authentication – Ahmed Bougacha, John McCall
  • Porting by a 1000 Patches: Bringing Swift to Windows – Saleem Abdulrasool
  • The Penultimate Challange: Constructing bug reports in the Clang Static Analyzer – Kristóf Umann
  • Address Spaces in LLVM – Matt Arsenault
  • An MLIR Dialect for High-Level Optimization of Fortran – Eric Schweitz
  • Loop-transformation #pragmas in the front-end – Michael Kruse
  • Optimizing builds on Windows: some practical considerations – Alexandre Ganea
  • LLVM-Reduce for testcase reduction – Diego Treviño Ferrer
  • Memoro: Scaling an LLVM-based Heap profiler – Thierry Treyer
  • The Attributor: A Versatile Inter-procedural Fixpoint Iteration Framework – Johannes Doerfert
  • LLVM Tutorials: How to write Beginner-Friendly, Inclusive Tutorials – Meike Baumgärtner
  • Maturing an LLVM backend: Lessons learned from the RISC-V target – Alex Bradbury

Tutorials

  • Getting Started With LLVM: Basics – Jessica Paquette, Florian Hahn
  • ASTImporter: Merging Clang ASTs – Gábor Márton
  • Developing the Clang Static Analyzer – Artem Dergachev
  • Writing an LLVM Pass: 101 – Andrzej Warzynski
  • Writing Loop Optimizations in LLVM – Kit Barton, Ettore Tiotto, Hal Finkel, Michael Kruse, Johannes Doerfert
  • The Attributor: A Versatile Inter-procedural Fixpoint Iteration Framework – Johannes Doerfert
  • Getting Started with the LLVM Testing Infrastructure – Brian Homerding, Michael Kruse
  • An overview of Clang – Sven Van Haastregt, Anastasia Stulova
  • An overview of LLVM – Eric Christopher, Sanjoy Das, Johannes Doerfert
  • How to Contribute to LLVM – Chris Bieneman, Kit Barton
  • My First Clang Warning – Dmitri Gribenko, Meike Baumgartner

Student Research Competition

  • Cross-Translation Unit Optimization via Annotated Headers – William S. Moses
  • Quantifying Dataflow Analysis with Gradients in LLVM – Abhishek Shah
  • Floating Point Consistency in the Wild: A practical evaluation of how compiler optimizations affect high performance floating point code – Jack J Garzella
  • Static Analysis of OpenMP Data Mapping for Target Offloading – Prithayan Barua

Panels

  • Panel: Inter-procedural Optimization (IPO) – Teresa Johnson, Philip Reames, Chandler Carruth, Johannes Doerfert
  • The Loop Optimization Working Group – Kit Barton, Michael Kruse, TBD

Birds of a Feather

  • LLDB – Jonas Devlieghere
  • Towards Better Code Generator Design and Unification for a Stack Machine – Leonid Kholodov, Dmitry Borisenkov
  • Debug Info – Adrian Prantl

Lightning Talks

  • GWP-ASan: Zero-Cost Detection of MEmory Safety Bugs in Production – Matt Morehouse
  • When 3 Memory Models Aren’t Enough – OpenVMS on x86 – John Reagan
  • FileCheck: learning arithmetic – Thomas Preud’homme
  • -Wall Found Programming Errors and Engineering Effort to Enable Across a Large Codebase – Aditya Kumar
  • Handling 1000s of OpenCL builtin functions in Clang – Sven van Haastregt
  • NEC SX-Aurora as a Scalable Vector Playground – Kazuhisa Ishizaka
  • Implementing Machine Code Optimizations for RISC-V – Lewis Revill
  • Optimization Remarks Update – Francis Visoiu Mistrih
  • Supporting Regular and Thin LTO with a Single LTO Bitcode Format – Matthew Voss
  • Transitioning Apple’s Downstream llvm-project Repositories to the Monorepo – Alex Lorenz
  • A Unified Debug Server For Deeply Embedded Systems and LLDB – Simon Cook
  • State of LLDB and Deeply Embedded RISC-V – Simon Cook
  • Supporting a Vendor ABI Variant in Clang – Paul Robinson
  • Speculative Compilation in ORC JIT – Praveen Velliengiri
  • Optimization Remarks for Human Beings – William Bundy
  • Improving the Optimized Debugging Experience – Orlando Cazalet-Hyams
  • Improving your TableGen Descriptions – Javed Absar
  • Loom: Weaving Instrumentation for Program Analysis – Brian Kidney
  • Clang Interface Stubs: Syntax Directed Stub Library Generation. – Puyan Lotfi
  • Flang Update – Steve Scalpone
  • Lowering tale: Supporting 64 bit pointers in RISCV 32 bit LLVM backend – Reshabh Sharma
  • Virtual Function Elimination in LLVM – Oliver Stannard
  • Making a Language Cross Platform: Libraries and Tooling – Gwen Mittertreiner
  • Grafter – A use case to implement an embedded DSL in C++ and perform source to source traversal fusion transformation using Clang – Laith Sakka

Posters

  • TON Labs Backend for TON Blockchain – Dmitry Borisenkov, Dmitry Shtukenberg, Leonid Kholodov
  • LLVM Build Times Using a Program Repository – Rusell Gallop, Phil Camp
  • RISC-V Bit Manipulation Support in the Clang/LLVM Toolchain – Scott Egerton, Paolo Savini
  • Attributor, a Framework for Interprocedural Information Deduction – Johannes Doerfert, Hideto Ueno, Stefan Stipanovic
  • Overflows Be Gone: Checked C for Memory Safety – Mandeep Singh Grang
  • Cross-Translation Unit Optimization via Annotated Headers – William S. Moses
  • Quantifying Dataflow Analysis with Gradients in LLVM – Abhishek Shah
  • Floating Point Consistency in the Wild: A practical evaluation of how compiler optimizations affect high performance floating point code – Jack J Garzella
  • Static Analysis of OpenMP Data Mapping for Target Offloading – Prithayan Barua
  • NEC SX-Aurora as a Scalable Vector Playground – Kazuhisa Ishizaka
  • A Unified Debug Server For Deeply Embedded Systems and LLDB – Simon Cook
  • Speculative Compilation in ORC JIT – Praveen Velliengiri
  • Loom: Weaving Instrumentation for Program Analysis – Brian Kidney
  • Lowering tale: Supporting 64 bit pointers in RISCV 32 bit LLVM backend – Reshabh Sharma

[*]

[*]