JBake

Announcing TamboUI

Tue, 17 Feb 2026 00:00:00 +0100

Note	This blog post is co-authored by Cédric Champeau (Micronaut) and Max Rydahl Andersen (Quarkus), and cross-posted on our respective personal blogs.

Today we are excited to announce TamboUI, an open-source Terminal UI framework for the Java ecosystem!

The terminal is having a renaissance.

AI coding tools live there. Developer workflows are increasingly CLI-first again. Rust has Ratatui. Python has Rich and Textual, Go has Charm, Typescript has OpenTUI. But Java, despite its performance, maturity, and tooling, didn’t have a modern, composable, developer-friendly TUI framework.

We thought that should change.

How it started

TamboUI (pronounced like the french word “tambouille”, slang for "cooking up something" or "makeshift creation") was born a bit by accident: a couple months ago, Cédric was asking about which TUI libraries that tools like Claude Code were using. Max Andersen answered that most likely this was Ratatui, a framework written in Rust. Both thought that it was a bit sad that there was no such library for Java.

A few weeks later, Cédric did an experiment by asking AI (Claude Code) to port Ratatui to Java. The result was fairly impressive, and the beginning of a collaboration that led to the creation of TamboUI. In fact, Max gave you a hint last year that this was going to happen.

That said, TamboUI is not a Ratatui port nor is it a Textual port. We’ve put a lot of effort in going beyond the initial AI generated port. The library was designed with Java developers in mind, inspired by the good things found in other ecosystems’ approach to TUI frameworks. It offers a multi-layer API: from low-level widget primitives (like Ratatui), to a managed TUI layer with event handling, up to a declarative Toolkit DSL that handles the event loop and rendering thread for you—things that Ratatui doesn’t really cover. This brings the power of Ratatui, Textual or Rich to the Java ecosystem, with the Java touch!

Not only that, TamboUI is also GraalVM native compatible! This means that you can compile your Java TUI applications as native binaries, making Java a first class citizen in terminal applications development, with low memory footprints and fast startup!

If you want to give it a try, the easiest is to run our demos using JBang:

jbang demos@tamboui

Try it out

At this stage, the APIs are still unstable and subject to change. TamboUI is developed with the mindset of being framework-agnostic and having as few external dependencies as possible. You can choose between several backends like JLine, Aesh or the built-in Panama backend. By choosing the latter, you’ll get the best performance while not depending on any external library.

Whether you want to build:

a standalone CLI tool
an internal developer tool
a DevOps utility
add a TUI frontend to existing Java tool
an AI agent
or something entirely new

Give TamboUI a try and let us know what worked and what could be improved!

Check out the documentation at https://tamboui.dev/docs/main/, join us on Zulip, or browse the source on GitHub. We’d love your feedback and contributions!

Acknowledgments

We would like to thank the following people for their ideas, suggestions and contributions to the creation of first public release of TamboUI (in alphabetical order):

Andres Almiray
Charles Moulliard
Claus Ibsen
Graeme Rocher
Guillaume LaForge
James Cobb
Ståle Pedersen
Tako Schotanus

and of course to the Ratatui and Textual creators for their inspiration and work.

Max Rydahl Andersen & Cédric Champeau

Mesurer la rotation différentielle solaire avec JSol’Ex

Thu, 5 Feb 2026 00:00:00 +0100

Une boule de plasma en rotation

L’un des aspects fascinants du Soleil est qu’il ne tourne pas comme un corps solide. Contrairement à la Terre, qui effectue une rotation en 24 heures quelle que soit la latitude, la vitesse de rotation du Soleil varie avec la latitude : l’équateur tourne plus vite que les pôles. Ce phénomène, appelé rotation différentielle, est étudié depuis le 19ème siècle et reste un sujet de recherche important en physique solaire.

Les régions équatoriales du Soleil effectuent une rotation en environ 25 jours, tandis que près des pôles, cela prend environ 35 jours. Cette rotation différentielle joue un rôle clé dans le mécanisme de dynamo solaire qui génère le champ magnétique du Soleil et pilote le cycle solaire de 11 ans.

Dans cet article, je vais expliquer comment j’ai ajouté une fonctionnalité dans JSol’Ex 4.5.0 qui permet aux astronomes amateurs de mesurer cette rotation différentielle directement à partir d’un scan obtenu avec un spectrohéliographe. Une erreur fréquente est de penser que cette mesure n’est pas possible parce que la dispersion spectrale d’un spectrographe comme le Sol’Ex est typiquement d’environ 0.1 à 0.2 Å, c’est-à-dire plus grand que l’échelle de ce que nous essayons de mesurer (2 km/s, soit environ 0.04 Å). En pratique, nous devrons obtenir une précision sub-pixellaire et cet article explique comment y arriver.

L’effet Doppler

Le principe derrière la mesure est simple : l'effet Doppler. Quand une source lumineuse se déplace vers nous, la longueur d’onde est décalée vers le bleu du spectre (blueshift). Quand elle s’éloigne, la longueur d’onde se décale vers le rouge (redshift).

Puisque le Soleil tourne, un limbe se déplace toujours vers nous (le limbe Est) tandis que l’autre s’éloigne (le limbe Ouest). En mesurant le décalage de longueur d’onde d’une raie spectrale aux deux limbes, nous pouvons calculer la vitesse de rotation.

On pourrait se demander ce qu’il en est du mouvement propre de la Terre : nous orbitons autour du Soleil à environ 30 km/s, ce qui est bien plus grand que la vitesse de rotation solaire d’environ 2 km/s que nous essayons de mesurer. Ce mouvement orbital produit effectivement un décalage Doppler de tout le spectre solaire. Cependant, cet effet s’applique de manière égale à tous les points que nous pouvons mesurer : en d’autres termes, puisque tout le spectre est décalé de manière uniforme, tout s’annule. Il en va de même pour toute vitesse radiale du Soleil par rapport à la Terre (qui varie légèrement tout au long de l’année car l’orbite de la Terre est elliptique).

La formule fondamentale reliant le décalage Doppler à la vitesse est :

\[\frac{\Delta\lambda}{\lambda_0} = \frac{v}{c}\]

Où :

$\Delta\lambda$ est le décalage en longueur d’onde (ce qu’on cherche à mesurer)
$\lambda_0$ est la longueur d’onde au repos de la raie spectrale (typiquement la longueur d’onde H-alpha)
$v$ est la vitesse le long de la ligne de visée
$c$ est la vitesse de la lumière (299 792 km/s)

La méthodologie de mesure

Travaux antérieurs

Je ne suis pas la première personne à essayer de faire cette mesure en spectrohéliographie amateur. En 2017, Peter Zetner a expliqué sur CloudyNights comment mesurer la vitesse différentielle en utilisant un spectrohéliographe. Il a expliqué cette méthodologie en profondeur dans le livre Astronomie solaire - Observer, photographier et étudier le Soleil. Sa méthodologie repose sur des mesures utilisant la raie Na D2. mais peut aussi être appliquée sur d’autres raies brillantes incluant la raie Fe I couramment utilisée à 5250.2 Å, et implique de comparer les intensités des pixels, en les moyennant à différentes latitudes. Les résultats sont meilleurs en effectuant plusieurs scans et en moyennant les données. Bien que cela fonctionne, je voulais essayer une méthodologie différente :

Utiliser un seul scan : les images Doppler que JSol’Ex ou INTI peuvent produire sont en général très cohérentes et montrent que toutes les données dont nous avons besoin sont déjà présentes
Éviter les intensités de pixels : celles-ci sont très sensibles à l’assombrissement centre-bord, aux conditions d’observation (par exemple les nuages, même fins) ou au vignetage (assombrissement des images le long de la fente)
Rendre possible l’utilisation de scans H-alpha, non pas parce qu’ils donneraient une valeur précise de la vitesse du plasma de la photosphère, mais parce que c’est la raie la plus largement imagée dans les observations solaires amateurs

Les résultats, bien sûr, dépendront de la raie observée. La méthodologie que je décris ci-dessous est capable de retourner des résultats raisonnables sur plusieurs raies (H-alpha, Na D2. Fe I) mais échouera complètement sur certaines autres (par exemple Ca II K, typiquement parce que la raie est trop large).

Plongeons-nous dans la méthodologie.

Note

L’implémentation décrite ici a été développée de manière itérative basée sur des observations réelles. Je ne suis pas physicien solaire, juste un ingénieur essayant d’extraire des données significatives des captures de spectrohéliographe. L’algorithme fonctionne raisonnablement bien en pratique mais devrait être validé par rapport à des mesures professionnelles pour toute application scientifique. C’est pourquoi cette fonctionnalité est annoncée comme expérimentale.

Comparaison des limbes Est-Ouest

La clé pour comprendre la méthodologie est qu’elle repose sur l’analyse et l’ajustement de profils de raies spectrales. Elle nécessite d’extraire les profils de raies spectrales à différents endroits du disque solaire, de préférence près des limbes où les vitesses sont les plus élevées (plus on s’approche du méridien solaire, plus les vitesses sont faibles et donc difficiles à détecter).

La mesure de rotation différentielle extrait les vitesses de rotation solaire en comparant les décalages Doppler entre les points des limbes Est et Ouest à la même latitude héliographique.

Pour chaque latitude (par exemple, 20° Nord), l’algorithme :

Sélectionne un point sur le limbe Est
Sélectionne le point correspondant sur le limbe Ouest à la même latitude
Mesure la position de la raie spectrale à chaque point
Calcule la différence : Ouest - Est

Cette approche différentielle a un avantage crucial : elle annule les erreurs systématiques. Tout décalage instrumental, erreur de calibration en longueur d’onde, ou décalage de référence affecte les deux mesures de manière égale et disparaît quand nous prenons la différence.

Cependant, une seule mesure Est-Ouest à chaque latitude n’est pas assez fiable. Les conditions de seeing, l’activité solaire locale, ou les erreurs d’ajustement peuvent corrompre les mesures individuelles. Pour améliorer la précision, JSol’Ex échantillonne plusieurs longitudes à travers la région du limbe (typiquement 14 points de 62° à 88° de longitude) et agrège ces mesures. Cette redondance permet de rejeter les valeurs aberrantes et fournit une estimation d’erreur basée sur la cohérence des mesures.

La vitesse mesurée est alors :

\[v_{measured} = \frac{\Delta\lambda}{\lambda_0} \times \frac{c}{2}\]

Le facteur 2 apparaît parce que nous mesurons la différence entre les limbes s’approchant et s’éloignant.

De la vitesse mesurée à la vitesse équatoriale

La vitesse que nous mesurons dépend de la géométrie : quelle fraction de la vitesse de rotation est projetée le long de notre ligne de visée. À l’équateur et au limbe, nous voyons la vitesse de rotation complète. À des latitudes plus élevées ou plus près du centre du disque, nous n’en voyons qu’une fraction.

Le facteur de correction géométrique est :

\[v_{equatorial} = \frac{v_{measured}}{\cos(\phi) \times \sin(\theta)}\]

Où :

$\phi$ est la latitude héliographique
$\theta$ est la longitude héliographique (0° au centre du disque, ±90° aux limbes)

Trouver le centre de la raie spectrale

La partie la plus difficile de la mesure est de déterminer précisément le centre de la raie d’absorption. Un décalage de seulement 0.01 Å correspond à une vitesse d’environ 0.5 km/s, ce qui est significatif par rapport à la vitesse équatoriale typique d’environ 2 km/s.

JSol’Ex utilise l'ajustement de profil de Voigt pour mesurer le centre de la raie. Le profil de Voigt est la convolution d’un profil gaussien et d’un profil lorentzien, qui modélise précisément la forme des raies d’absorption solaires. La composante gaussienne représente l’élargissement Doppler thermique, tandis que la composante lorentzienne représente l’élargissement naturel et de pression.

Pour chaque point de mesure, l’algorithme :

Extrait un profil spectral du fichier SER à la position correspondante
Ajuste un profil de Voigt à la raie d’absorption
Enregistre la position du centre de raie ajustée

Le paramètre configurable "demi-largeur d’ajustement de Voigt" (par défaut : 2 Å) contrôle quelle proportion des ailes de la raie sont incluses dans l’ajustement.

Systèmes de coordonnées

L’un des aspects les plus délicats de cette implémentation est de transformer correctement les coordonnées que nous manipulons. L’algorithme commence avec les coordonnées héliographiques (où nous voulons mesurer) et les transforme en coordonnées des trames SER originales (où les données spectrales se trouvent).

1. Coordonnées héliographiques (point de départ)

Nous commençons par spécifier des points en coordonnées héliographiques :

Latitude : -90° (pôle sud) à +90° (pôle nord)
Longitude : 0° au centre du disque, ±90° aux limbes

Cela nécessite deux paramètres solaires calculés à partir de la date d’observation :

B0 : La latitude héliographique du centre du disque (varie tout au long de l’année)
P : L’angle de position de l’axe de rotation (l’inclinaison du Soleil vu depuis la Terre)

2. Coordonnées image

Les coordonnées héliographiques sont converties en positions de pixels dans l’image reconstruite. Cela implique d’inverser les corrections appliquées pendant la reconstruction :

Correction de l’angle P
Retournement/rotation
Distorsion géométrique
Angle d’inclinaison
Recadrage

3. Coordonnées du fichier SER (destination)

Finalement, les coordonnées image sont mappées vers le fichier vidéo SER brut :

Numéro de trame : Position dans la séquence de scan (dérivée de la coordonnée x)
Colonne : Position le long de la fente (dérivée de la coordonnée y)
Ligne : Direction spectrale (longueur d’onde)

Ce mapping inverse nous permet d’extraire le profil spectral exact à n’importe quelle position héliographique.

Le pipeline de traitement des données

Les mesures brutes sont bruitées. Pour produire une courbe de rotation propre, JSol’Ex utilise un pipeline de traitement en deux étapes :

Étape 1 : Agrégation en longitude

Comme décrit précédemment, à chaque latitude, plusieurs longitudes sont échantillonnées (typiquement 14 points à travers la région du limbe). Ces mesures sont combinées en utilisant l’une des trois méthodes (la médiane par défaut mais vous pouvez choisir) :

Médiane : Robuste aux valeurs aberrantes, utilise la valeur centrale
Moyenne : Simple moyenne arithmétique
Moyenne pondérée : Les points plus proches du limbe (où le signal Doppler est plus fort) ont un poids plus élevé

L’estimation d’erreur de cette étape représente la cohérence des mesures à travers les longitudes.

Étape 2 : Lissage en latitude

Même après l’agrégation en longitude, les mesures latitude par latitude restent bruitées. Les bins de latitude individuels peuvent encore être affectés par des caractéristiques localisées (taches solaires, facules) ou simplement par la dispersion des mesures.

Puisque nous nous attendons à ce que la rotation solaire varie de manière lisse avec la latitude (suivant les termes $\sin^2\phi$ et $\sin^4\phi$ de la loi de rotation différentielle), nous pouvons exploiter cette contrainte physique pour réduire davantage le bruit. JSol’Ex applique un filtre de lissage qui combine les points de latitude voisins dans une fenêtre configurable (par défaut : 5°).

De manière importante, les erreurs de l’étape 1 sont propagées plutôt que recalculées à partir de la dispersion dans la fenêtre de lissage. Cela garantit que les barres d’erreur représentent l’incertitude de mesure, pas les variations physiques de latitude.

Pour l’agrégation médiane :

\[\sigma_{smoothed} = \frac{median(\sigma_i)}{\sqrt{n}}\]

Pour la moyenne :

\[\sigma_{smoothed} = \frac{\sqrt{\sum \sigma_i^2}}{n}\]

Le résultat et la comparaison avec la théorie

La forme standard de la loi de rotation différentielle (aussi connue comme la formule de Faye, même si la formule originale n’incluait pas le 3ème terme) est :

\[\omega(\phi) = A + B \sin^2(\phi) + C \sin^4(\phi)\]

Où :

$\phi$ est la latitude héliographique
$A$ est le taux de rotation équatorial
$B$ et $C$ contrôlent la diminution de la vitesse avec la latitude

JSol’Ex compare les résultats mesurés avec les coefficients largement utilisés de Snodgrass & Ulrich (1990) :

$A = 14.713$ deg/jour (taux de rotation équatorial)
$B = -2.396$ deg/jour
$C = -1.787$ deg/jour

Cela donne une période de rotation équatoriale d’environ 24.5 jours et une période polaire d’environ 34 jours. Convertie en vitesse linéaire à la surface solaire (rayon ≈ 696 000 km), la vitesse de rotation équatoriale est d’environ 2.0 km/s.

JSol’Ex ajuste également ses propres coefficients A, B, C à partir de vos mesures, permettant une comparaison directe avec les valeurs de référence.

Résultats

Ci-dessous des exemples de profils de rotation différentielle mesurés avec JSol’Ex en utilisant deux raies spectrales différentes. Ces mesures ont été prises à des dates différentes avec des conditions de seeing différentes.

Fe I 5883 Å

H-alpha

Observations

Les profils mesurés montrent la forme générale attendue : des vitesses plus élevées à l’équateur, diminuant vers les pôles. Les courbes ajustées suivent raisonnablement bien le profil théorique de Snodgrass.

Cependant, j’observe des différences entre les raies spectrales que je ne peux pas entièrement expliquer. Les mesures Fe I montrent des vitesses absolues différentes de H-alpha, et les coefficients ajustés diffèrent entre les scans.

Plusieurs facteurs pourraient contribuer à ces différences :

Hauteur de formation : Différentes raies spectrales se forment à différentes hauteurs dans l’atmosphère solaire, et les taux de rotation peuvent varier avec l’altitude (comme suggéré par des recherches récentes de la mission CHASE)
Conditions de seeing : Les scans ont été pris à des dates différentes avec des conditions atmosphériques différentes
Différences de profil de raie : H-alpha et Fe I ont des largeurs et profondeurs de raie différentes, ce qui peut affecter la précision de l’ajustement de Voigt
Effets systématiques : Il peut y avoir des facteurs instrumentaux ou algorithmiques que je n’ai pas identifiés

Je présente ces résultats tels qu’ils sont, sans essayer de tirer des conclusions sur quelle mesure est "correcte" ou ce qui cause les différences observées. Plus de mesures sur différentes dates, raies spectrales et instruments seraient nécessaires pour comprendre ces variations.

Mesurer en pratique

La qualité des résultats dépend fortement du seeing atmosphérique. Un mauvais seeing floute les raies spectrales et rend la détermination précise du centre difficile. Les meilleurs résultats sont obtenus avec d’excellentes conditions de seeing et un instrument bien focalisé. Enfin et surtout, une haute résolution spectrale est préférée, ce qui est normalement le cas avec le Sol’Ex (HR) ou le SHG 700.

Vous pouvez aussi vous demander quelle raie spectrale vous devriez observer. J’ai testé l’algorithme avec Fe I, Na D2 et H-alpha, donnant les résultats ci-dessous. En pratique, les raies d’absorption fortes devraient en théorie produire les meilleurs résultats, car :

La profondeur de raie devrait être suffisante pour un ajustement précis
Les ailes devraient être bien définies pour l’ajustement de Voigt
Le rapport signal sur bruit devrait être élevé

Les raies très larges comme Ca II K sont trop larges pour un ajustement de Voigt précis et produiront des résultats peu fiables. Les raies trop étroites peuvent aussi représenter un défi pour l’ajustement de Voigt mais peuvent fonctionner aussi.

L’impact de la détection de la raie spectrale

Un facteur subtil mais important affectant la précision absolue des mesures est la correction polynomiale utilisée pendant la reconstruction d’image. Comprendre ceci est essentiel pour interpréter correctement vos résultats.

Pendant le traitement du spectrohéliographe, JSol’Ex calcule un polynôme qui fait correspondre chaque position de colonne le long de la fente vers la position de ligne attendue du centre de la raie spectrale. Ce polynôme est calculé à partir de la moyenne des trames SER dans le scan (une heuristique est utilisée pour inclure les trames contenant des données solaires réelles, pas le fond de ciel).

Une telle détection est cruciale car la raie spectrale n’est pas droite à travers la fente en raison de phénomènes optiques : elle constitue ce que nous appelons souvent le "smile". Par conséquent, le centre de raie peut être à la ligne 300 à la colonne 500. mais à la ligne 350 à la colonne 1000. Le polynôme capture cette courbure.

En mesurant les positions de raie relativement au polynôme, nous éliminons efficacement la distorsion optique de nos mesures. Sans cette référence, nous mesurerions la somme de la courbure optique plus le décalage Doppler, rendant impossible l’extraction des minuscules signaux de vitesse que nous recherchons.

Une question clé est : pourquoi ne pouvons-nous pas simplement mesurer la position absolue de la raie spectrale dans chaque trame et calculer les vitesses directement ?

La réponse réside dans la précision requise. Une vitesse de 2 km/s correspond à un décalage de longueur d’onde de seulement ~0.04 Å à H-alpha. Avec une dispersion spectrale typique de 0.1-0.2 Å/pixel, c’est un décalage de seulement 0.2 à 0.4 pixels.

La courbure optique à travers la fente, d’autre part, peut facilement s’étendre sur 10-30 pixels ou plus. Toute tentative de mesurer les positions absolues de raie serait complètement dominée par cette courbure, rendant la détection Doppler impossible.

Le polynôme sert de référence de base. En mesurant comment la position de raie dans chaque trame individuelle dévie du polynôme, nous isolons la composante Doppler de la composante optique. C’est l’idée clé qui rend les mesures de précision sub-pixellaire possibles, avec l’ajustement de Voigt.

Un mot sur les images Doppler

Ceci, d’ailleurs, est aussi une raison pour laquelle les images Doppler sont différentes quand on scanne en AD vs DEC. Parce que c’est une question souvent posée, et qu’il m’a fallu des mois pour comprendre la raison, je pense qu’il vaut la peine de passer un peu de temps à expliquer le phénomène. Je dois cette explication à Jean-François Pittet et Christian Buil, d’une discussion il y a quelques semaines sur la mailing list Sol’Ex.

Scan en AD vs scan en DEC : une différence critique pour les images Doppler

Les spectrohéliographes peuvent scanner le Soleil dans deux directions :

Scan en AD (Ascension Droite) : La fente est orientée Nord-Sud, et le scan procède Est-Ouest
Scan en DEC (Déclinaison) : La fente est orientée Est-Ouest, et le scan procède Nord-Sud

Ce choix a des implications profondes pour la visibilité Doppler, comme expliqué par Jean-François Pittet sur la mailing list Sol’Ex.

Scan en AD : Doppler visible dans les images

En mode scan AD, chaque trame capture une tranche verticale du Soleil du pôle Nord au pôle Sud. Au cours du scan :

Les premières trames capturent le limbe Est (s’approchant, décalé vers le bleu)
Les trames du milieu capturent le centre du disque (pas de vitesse radiale)
Les dernières trames capturent le limbe Ouest (s’éloignant, décalé vers le rouge)

Quand JSol’Ex calcule le polynôme à partir de la moyenne de toutes les trames, les contributions Est et Ouest s’équilibrent. Le blueshift du limbe Est est annulé par le redshift du limbe Ouest. Le polynôme résultant représente la référence "neutre" : approximativement la position de raie au centre du disque sans décalage Doppler.

En conséquence, quand nous reconstruisons l’image au décalage de pixel 0. les décalages Doppler deviennent visibles comme différences de contraste :

Le limbe Est apparaît plus sombre (raie décalée dans la bande passante)
Le limbe Ouest apparaît plus clair (raie décalée hors de la bande passante)

C’est pourquoi les images Doppler des scans AD montrent l’asymétrie caractéristique Est-Ouest.

Scan en DEC : Doppler "absorbé" par le polynôme

En mode scan DEC, chaque trame capture une tranche horizontale du Soleil d’Est en Ouest. Voici la différence critique : tous les pixels dans une seule trame voient approximativement le même décalage Doppler.

Par exemple, si la trame actuelle capture la moitié Est du disque :

Tous les points dans cette trame s’approchent de nous
Tous les points ont un blueshift similaire

Quand JSol’Ex calcule le polynôme à partir de la moyenne de toutes les trames, il ne capture pas seulement la courbure optique : il capture aussi le décalage Doppler moyen à travers le scan. Mais voici le problème : le décalage Doppler varie systématiquement à travers le scan (les trames Est sont décalées vers le bleu, les trames Ouest sont décalées vers le rouge).

Le polynôme "absorbe" effectivement une moyenne pondérée de ces décalages Doppler. Quand nous mesurons ensuite les positions de raie relativement à ce polynôme, une grande partie du signal Doppler a déjà été soustraite. Les images Doppler résultantes ne montreront plus la rotation du soleil (ce qui peut aussi être un avantage, pour d’autres types d’observations).

C’est pourquoi les spectrohéliographes expérimentés préfèrent souvent le scan AD pour le travail Doppler : le signal Doppler est plus visible et plus facile à détecter.

La complication de l’angle P

Il y a une subtilité supplémentaire : l’axe de rotation du Soleil n’est pas perpendiculaire à l’écliptique. L'angle P (angle de position de l’axe de rotation) varie tout au long de l’année d’environ -26° à +26°.

Deux fois par an (vers début juin et début décembre) l’angle P est proche de zéro. À ces moments :

Le scan AD est vraiment parallèle à l’équateur solaire
Le motif Doppler Est-Ouest s’aligne parfaitement avec la direction de scan

À d’autres moments, quand P est non nul :

L’axe de rotation est incliné par rapport à la direction de scan
Le motif Doppler est tourné par rapport aux axes de l’image
Une partie du signal Doppler "fuit" dans la moyenne polynomiale même en mode AD

C’est une des raisons pour lesquelles les mesures de rotation différentielle peuvent montrer des résultats légèrement différents à différentes périodes de l’année, bien que nous prenions en compte les angles P et B0.

Implications pour les mesures de rotation différentielle

Ceci explique pourquoi le scan AD est essentiel pour les mesures de vitesse différentielle :

En mode AD, les points des limbes Est et Ouest à la même latitude correspondent à la même colonne sur la fente (même position de fente, trames différentes). La valeur du polynôme à cette colonne est calculée à partir de la moyenne de toutes les trames (limbe Est, centre du disque, et limbe Ouest) donc les décalages Doppler s’annulent dans la moyenne. Quand nous calculons (Ouest - polynôme) - (Est - polynôme), les termes du polynôme sont identiques et s’annulent, nous laissant avec la vraie différence Doppler Ouest - Est.

En mode DEC, les points des limbes Est et Ouest à la même latitude correspondent aux extrémités opposées de la fente (colonnes différentes, même trame). Le polynôme à la colonne Est est calculé à partir de trames qui voient toutes le limbe Est à cette position, donc le polynôme absorbe le blueshift. De même, le polynôme à la colonne Ouest absorbe le redshift. Quand nous calculons la différence, ces décalages Doppler absorbés ne s’annulent pas : ils se soustraient de notre mesure, réduisant significativement le signal mesuré.

C’est pourquoi, en pratique, les scans DEC ne produisent pas d’images Doppler utilisables ou de mesures de vitesse différentielle fiables.

Conseils de configuration

Longitude du limbe (par défaut 75°) : Les mesures plus proches du limbe donnent des signaux Doppler plus forts mais risquent des effets d’assombrissement centre-bord
Pas de latitude (par défaut 2°) : Des valeurs plus petites donnent une résolution plus fine mais des temps de traitement plus longs
Fenêtre de lissage : Devrait être au moins 2× le pas de latitude pour un lissage efficace

Conclusion

Mesurer la rotation différentielle avec du matériel amateur aurait semblé impossible il y a quelques années seulement. Des pionniers comme Peter Zetner ou Christian Buil nous ont montré la voie. Ce que j’essaie de faire, comme toujours avec JSol’Ex, c’est de rendre ceci accessible à encore plus de personnes au prix de la "magie", c’est-à-dire que certaines personnes partageront très probablement des résultats sans comprendre la science derrière. Je ne suis pas trop inquiet par cela, parce que je suis moi-même passé par là : apprécier la reconstruction d’images SHG, puis comprendre comment ça fonctionne, puis me poser des questions comme "pourquoi est-il même possible de mesurer des vitesses Doppler si petites ?" : c’est une construction intellectuelle qui prend du temps. Nous, membres de la communauté Sol’Ex, pouvons faire mieux, je pense, pour apporter cela aux masses, en partageant nos idées et en écrivant des logiciels comme celui-ci.

Cette fonctionnalité rejoint la liste croissante des capacités scientifiques dans JSol’Ex : détection des bombes d’Ellerman, identification des régions actives, et maintenant mesure de la rotation différentielle. Chacune de celles-ci met des analyses de niveau professionnel à la portée des astronomes amateurs, mais, comme toujours, soyez prudent avec ce que je dis : je ne suis pas un scientifique, juste un ingénieur. Je n’ai aucun doute que j’ai fait des approximations, ou pris des libertés que je n’aurais probablement pas dû prendre.

Bibliographie

Articles scientifiques

Ressources amateurs

Measuring Solar Differential Rotation with JSol’Ex

Thu, 5 Feb 2026 00:00:00 +0100

A Rotating Ball of Plasma

One fascinating aspect of the Sun is that it doesn’t rotate like a solid body. Unlike the Earth, which completes one rotation in 24 hours regardless of latitude, the Sun’s rotation rate varies with latitude: the equator rotates faster than the poles. This phenomenon, called differential rotation, has been studied since the 19th century and remains an important research topic in solar physics.

The equatorial regions of the Sun complete a rotation in approximately 25 days, while near the poles, it takes about 35 days. This differential rotation is thought to play a key role in the solar dynamo mechanism that generates the Sun’s magnetic field and drives the 11-year solar cycle.

In this blog post, I will explain how I implemented a feature in JSol’Ex 4.5.0 that allows amateur astronomers to measure this differential rotation directly from their spectroheliograph data. A common mistake is to think that this measurement is not possible because the spectral dispersion of a SHG like the Sol’Ex is typically around 0.1 to 0.2Å, which larger than the scale of what we’re trying to measure (2 km/s, which is about 0.04Å). In fact, what we are trying to measure requires sub-pixel precision and we can achieve that.

The Doppler Effect

The principle behind the measurement is simple: the Doppler effect. When a light source moves towards us, the wavelength is shifted towards the blue end of the spectrum (blueshift). When it moves away, the wavelength shifts towards the red (redshift).

Since the Sun rotates, one limb is always moving towards us (let’s say the East limb) while the other is moving away (the West limb). By measuring the wavelength shift of a spectral line at both limbs, we can calculate the rotation velocity.

One might wonder about the Earth’s own motion: we orbit the Sun at approximately 30 km/s, which is much larger than the ~2 km/s solar rotation velocity we’re trying to measure. This orbital motion does produce a Doppler shift of the entire solar spectrum. However, this affects equally all points we can measure: this is a global shift which affects all measurements equally and cancels out. The same is true for any radial velocity of the Sun relative to the Earth (which varies slightly throughout the year as Earth’s orbit is elliptical).

The fundamental formula relating Doppler shift to velocity is:

\[\frac{\Delta\lambda}{\lambda_0} = \frac{v}{c}\]

Where:

$\Delta\lambda$ is the wavelength shift
$\lambda_0$ is the rest wavelength of the spectral line (typically the H-alpha wavelength)
$v$ is the velocity along the line of sight
$c$ is the speed of light (299,792 km/s)

The Measurement Methodology

Previous Work

I am not the first person to try to make this measurement in amateur spectroheliography. In 2017, Peter Zetner explained on CloudyNights how to measure differential velocity using a spectroheliograph. He explained this methodology in depth in the Solar Astronomy - Observing, imaging and studying the Sun book. His methodology relies on measurements using the Na D2 line, but can also be applied on other bright lines including the commonly used Fe I line at 5250.2 Å, and involves comparing the pixel intensities, by averaging them at different latitudes. The best results are obtained by performing several scans and averaging data. While this works, I wanted to try a different methodology:

Use a single scan: the Doppler images that JSol’Ex or INTI can produce are in general very consistent and show that all the data we need is already present
Avoid pixel intensities: these are very sensitive to limb darkening, observing conditions (e.g clouds, even if thin) or vignetting (darkening of the images along the slit)
Make it possible to use H-alpha scans, not because they would give an accurate value of the velocity of the photosphere plasma, but because that’s the most widely imaged line in amateur solar observations

The results, of course, will depend on the observed line. The methodology that I describe below is capable of returning reasonable results on several lines (H-alpha, Na D2, Fe I) but will completely fail on some others (e.g Ca II K, typically because the line is too wide).

Let’s dive into the methodology.

Note

The implementation described here was developed iteratively based on real observations. I’m not a solar physicist, just an engineer trying to extract meaningful data from spectroheliograph captures. The algorithm works reasonably well in practice but should be validated against professional measurements for any scientific application. Therefore why it is advertised as experimental.

East-West Limb Comparison

The key to understanding the methodology is that it relies on spectral line profile analysis and fitting. It requires extracting the spectral line profiles at different locations of the solar disk, preferably near the limbs where the velocities are highest.

The differential rotation measurement extracts solar rotation velocities by comparing Doppler shifts between East and West limb points at the same heliographic latitude.

For each latitude (say, 20° North), the algorithm:

Selects a point on the East limb
Selects the corresponding point on the West limb at the same latitude
Measures the spectral line position at each point
Computes the difference: West - East

This differential approach has a crucial advantage: it cancels out systematic errors. Any instrumental offset, wavelength calibration error, or baseline shift affects both measurements equally and disappears when we take the difference.

However, a single East-West measurement at each latitude is not reliable enough. Seeing conditions, local solar activity, or fitting errors can corrupt individual measurements. To improve accuracy, JSol’Ex samples multiple longitudes across the limb region (typically 14 points from 62° to 88° longitude) and aggregates these measurements. This redundancy allows outliers to be rejected and provides an error estimate based on measurement consistency.

The measured velocity is then:

\[v_{measured} = \frac{\Delta\lambda}{\lambda_0} \times \frac{c}{2}\]

The factor of 2 appears because we’re measuring the difference between approaching and receding limbs.

From Measured Velocity to Equatorial Velocity

The velocity we measure depends on the geometry: how much of the rotation velocity is projected along our line of sight. At the equator and at the limb, we see the full rotation velocity. At higher latitudes or closer to disk center, we see only a fraction.

The geometric correction factor is:

\[v_{equatorial} = \frac{v_{measured}}{\cos(\phi) \times \sin(\theta)}\]

Where:

$\phi$ is the heliographic latitude
$\theta$ is the heliographic longitude (0° at disk center, ±90° at the limbs)

Finding the Spectral Line Center

The most challenging part of the measurement is accurately determining the center of the absorption line. A shift of just 0.01 Å corresponds to a velocity of about 0.5 km/s, which is significant compared to the typical equatorial velocity of ~2 km/s.

JSol’Ex uses Voigt profile fitting to measure the line center. The Voigt profile is the convolution of a Gaussian and a Lorentzian profile, which accurately models the shape of solar absorption lines. The Gaussian component represents thermal Doppler broadening, while the Lorentzian component represents natural and pressure broadening.

For each measurement point, the algorithm:

Extracts a spectral profile from the SER file at the corresponding position
Fits a Voigt profile to the absorption line
Records the fitted line center position

The configurable "Voigt fit half-width" parameter (default: 2 Å) controls how much of the line wings are included in the fit.

Coordinate Systems

One of the trickier aspects of this implementation is correctly mapping between the different coordinate systems involved. The algorithm starts with heliographic coordinates (where we want to measure) and maps them back to the original SER frames (where the spectral data lives).

1. Heliographic Coordinates (starting point)

We begin by specifying points in heliographic coordinates:

Latitude: -90° (south pole) to +90° (north pole)
Longitude: 0° at disk center, ±90° at the limbs

This requires two solar parameters computed from the observation date:

B0: The heliographic latitude of the disk center (varies throughout the year)
P: The position angle of the rotation axis (the "tilt" of the Sun as seen from Earth)

2. Image Coordinates

The heliographic coordinates are converted to pixel positions in the reconstructed image. This involves reversing the corrections applied during reconstruction:

P-angle correction
Flip/rotation
Geometric distortion
Tilt angle
Cropping

3. SER File Coordinates (destination)

Finally, the image coordinates are mapped to the raw SER video file:

Frame number: Position in the scan sequence (derived from x-coordinate)
Column: Position along the slit (derived from y-coordinate)
Row: Spectral direction (wavelength)

This reverse mapping allows us to extract the exact spectral profile at any heliographic location.

The Data Processing Pipeline

The raw measurements are noisy. To produce a clean rotation curve, JSol’Ex uses a two-stage processing pipeline:

Stage 1: Longitude Aggregation

As described earlier, at each latitude, multiple longitudes are sampled (typically 14 points across the limb region). These measurements are combined using one of three methods (the default is median but you can choose):

Median: Robust to outliers, uses the middle value
Average: Simple arithmetic mean
Weighted Average: Points closer to the limb (where Doppler signal is stronger) get higher weight

The error estimate from this stage represents the consistency of measurements across longitudes.

Stage 2: Latitude Smoothing

Even after longitude aggregation, the latitude-by-latitude measurements remain noisy. Individual latitude bins can still be affected by localized features (sunspots, faculae) or simply measurement scatter.

Since we expect solar rotation to vary smoothly with latitude (following the $\sin^2\phi$ and $\sin^4\phi$ terms of the differential rotation law), we can exploit this physical constraint to reduce noise further. JSol’Ex applies a smoothing filter that combines nearby latitude points within a configurable window (default: 5°).

Importantly, the errors from Stage 1 are propagated rather than recomputed from the spread in the smoothing window. This ensures the error bars represent measurement uncertainty, not physical latitude variations.

For the median aggregation:

\[\sigma_{smoothed} = \frac{median(\sigma_i)}{\sqrt{n}}\]

For the average:

\[\sigma_{smoothed} = \frac{\sqrt{\sum \sigma_i^2}}{n}\]

The Outcome and Comparison with Theory

The standard form of the differential rotation law (also known as the Faye formula, even if the original formula didn’t include the 3rd term) is:

\[\omega(\phi) = A + B \sin^2(\phi) + C \sin^4(\phi)\]

Where:

$\phi$ is the heliographic latitude
$A$ is the equatorial rotation rate
$B$ and $C$ control the decrease in velocity with latitude

JSol’Ex compares measured results with the widely-used coefficients from Snodgrass & Ulrich (1990):

$A = 14.713$ deg/day (equatorial rotation rate)
$B = -2.396$ deg/day
$C = -1.787$ deg/day

This gives an equatorial rotation period of about 24.5 days and a polar period of about 34 days. Converting to linear velocity at the solar surface (radius ≈ 696,000 km), the equatorial rotation velocity is approximately 2.0 km/s.

JSol’Ex also fits its own A, B, C coefficients from your measurements, allowing direct comparison with the reference values.

Results

Below are example differential rotation profiles measured with JSol’Ex using two different spectral lines. These measurements were taken on different dates with different seeing conditions.

Fe I 5883 Å

H-alpha

Observations

The measured profiles show the expected general shape: higher velocities at the equator, decreasing toward the poles. The fitted curves follow the theoretical Snodgrass profile reasonably well.

However, I observe differences between spectral lines that I cannot fully explain. The Fe I measurements show different absolute velocities than H-alpha, and the fitted coefficients differ between scans.

Several factors could contribute to these differences:

Formation height: Different spectral lines form at different heights in the solar atmosphere, and rotation rates may vary with height (as suggested by recent research from the CHASE mission)
Seeing conditions: The scans were taken on different dates with different atmospheric conditions
Line profile differences: H-alpha and Fe I have different line widths and depths, which may affect the Voigt fitting precision
Systematic effects: There may be instrumental or algorithmic factors I haven’t identified

I present these results as they are, without attempting to draw conclusions about which measurement is "correct" or what causes the observed differences. More measurements across different dates, spectral lines, and instruments would be needed to understand these variations.

Measuring in Practice

The quality of results depends heavily on atmospheric seeing. Poor seeing blurs the spectral lines and makes accurate center determination difficult. Best results are obtained with excellent seeing conditions and a well-focused instrument. Last but not least, a high spectral resolution is preferred, which is normally the case with Sol’Ex (HR) or the SHG 700.

You may also wonder which spectral line you should observe. I tested the algorithm with Fe I, Na D2 and H-alpha, giving the results below. In practice, strong absorption lines should in theory produce the best results, because:

The line depth should be sufficient for accurate fitting
The wings should be well-defined for Voigt fitting
The signal-to-noise ratio should be high

Very broad lines like Ca II K are too wide for accurate Voigt fitting and will produce unreliable results. Too narrow lines may also represent a challenge for Voigt fitting but may work too.

The Impact of the Spectral Line Detection

One subtle but important factor affecting the absolute accuracy of the measurements is the polynomial correction used during image reconstruction. Understanding this is essential for interpreting your results correctly.

During spectroheliograph processing, JSol’Ex computes a polynomial that maps each column position along the slit to the expected row position of the spectral line center. This polynomial is computed from the average of SER frames in the scan (a heuristic is used to include frames containing actual solar data, not sky background).

Such a detection is crucial because the spectral line is not straight across the slit due to optical phenomenons: it consistutes what we often call the "smile". Therefore, the line center might be at row 300 at column 500, but at row 350 at column 1000. The polynomial captures this curvature.

By measuring line positions relative to the polynomial, we effectively remove the optical distortion from our measurements. Without this reference, we would be measuring the sum of optical curvature plus Doppler shift, making it impossible to extract the tiny velocity signals we’re after.

A key question is: why can’t we just measure the absolute position of the spectral line in each frame and compute velocities directly?

The answer lies in the precision required. A velocity of 2 km/s corresponds to a wavelength shift of only ~0.04 Å at H-alpha. With a typical spectral dispersion of 0.1-0.2 Å/pixel, this is a shift of only 0.2 to 0.4 pixels.

The optical curvature across the slit, on the other hand, can easily span 10-30 pixels or more. Any attempt to measure absolute line positions would be completely dominated by this curvature, making Doppler detection impossible.

The polynomial serves as our reference baseline. By measuring how the line position in each individual frame deviates from the polynomial, we isolate the Doppler component from the optical component. This is the key insight that makes sub-pixel precision measurements possible, alongside the Voigt fitting.

A Word About Doppler Images

This, by the way, is also a reason why the Doppler images look different when scanning in RA vs DEC. Because this is a question that is often asked, and that it took months for me to understand the reason, I think it’s worth spending a bit of time explaining the phenomenon. I owe this explanation to Jean-François Pittet and Christian Buil, from a discussion weeks ago on the Sol’Ex mailing list.

RA Scanning vs DEC Scanning: A Critical Difference for Doppler Images

Spectroheliographs can scan the Sun in two directions:

RA scanning (Right Ascension): The slit is oriented North-South, and the scan proceeds East-West
DEC scanning (Declination): The slit is oriented East-West, and the scan proceeds North-South

This choice has profound implications for Doppler visibility, as explained by Jean-François Pittet on the Sol’Ex mailing list.

RA Scanning: Doppler Visible in Images

In RA scanning mode, each frame captures a vertical slice of the Sun from North pole to South pole. Over the course of the scan:

Early frames capture the East limb (approaching, blueshifted)
Middle frames capture the disk center (no radial velocity)
Late frames capture the West limb (receding, redshifted)

When JSol’Ex computes the polynomial from the average of all frames, the East and West contributions balance out. The blueshift from the East limb is canceled by the redshift from the West limb. The resulting polynomial represents the "neutral" baseline: approximately the line position at disk center with no Doppler shift.

As a result, when we reconstruct the image at pixel shift 0, the Doppler shifts become visible as contrast differences:

East limb appears darker (line shifted into the passband)
West limb appears brighter (line shifted out of the passband)

This is why Doppler images from RA scans show the characteristic East-West asymmetry.

DEC Scanning: Doppler "Absorbed" by the Polynomial

In DEC scanning mode, each frame captures a horizontal slice of the Sun from East to West. Here’s the critical difference: all pixels within a single frame see approximately the same Doppler shift.

For example, if the current frame is capturing the Eastern half of the disk:

All points in that frame are approaching us
All points have similar blueshift

When JSol’Ex computes the polynomial from the average of all frames, it doesn’t just capture the optical curvature: it also captures the average Doppler shift across the scan. But here’s the problem: the Doppler shift varies systematically across the scan (East frames are blueshifted, West frames are redshifted).

The polynomial effectively "absorbs" a weighted average of these Doppler shifts. When we then measure line positions relative to this polynomial, much of the Doppler signal has already been subtracted out. The resulting Doppler images will no longer show the rotation of the sun (which can also be a benefit, for other kind of observations).

This is why experienced spectroheliographers often prefer RA scanning for Doppler work: the Doppler signal is more visible and easier to detect.

The P-Angle Complication

There’s an additional subtlety: the Sun’s rotation axis is not perpendicular to the ecliptic. The P-angle (position angle of the rotation axis) varies throughout the year from about -26° to +26°.

Only twice per year (around early June and early December) is the P-angle close to zero. At these times:

RA scanning is truly parallel to the solar equator
The East-West Doppler pattern aligns perfectly with the scan direction

At other times, when P is non-zero:

The rotation axis is tilted relative to the scan direction
The Doppler pattern is rotated relative to the image axes
Some Doppler signal "leaks" into the polynomial average even in RA mode

This is one reason why differential rotation measurements can show slightly different results at different times of year, despite us taking the P and B0 angles into account.

Implications for Differential Rotation Measurements

This explains why RA scanning is essential for differential velocity measurements:

In RA mode, East and West limb points at the same latitude correspond to the same column on the slit (same slit position, different frames). The polynomial value at that column is computed from the average of all frames (East limb, disk center, and West limb) so the Doppler shifts cancel out in the average. When we compute (West - polynomial) - (East - polynomial), the polynomial terms are identical and cancel, leaving us with the true Doppler difference West - East.

In DEC mode, East and West limb points at the same latitude correspond to opposite ends of the slit (different columns, same frame). The polynomial at the East column is computed from frames that all see the East limb at that position, so the polynomial absorbs the blueshift. Similarly, the polynomial at the West column absorbs the redshift. When we compute the difference, these absorbed Doppler shifts don’t cancel: they subtract from our measurement, significantly reducing the measured signal.

This is why, in practice, DEC scans do not produce usable Doppler images or reliable differential velocity measurements.

Configuration Tips

Limb longitude (default 75°): Measurements closer to the limb give stronger Doppler signals but risk limb darkening effects
Latitude step (default 2°): Smaller values give finer resolution but longer processing times
Smoothing window: Should be at least 2× the latitude step for effective smoothing

Conclusion

Measuring differential rotation with amateur equipment would have seemed impossible just a few years ago. Pioneers like Peter Zetner or Christian Buil showed us the path. What I’m trying to do, as always with JSol’Ex, is to make this accessible to even more people at the cost of "magic", which is that some people will very likely share results without understanding the science behind it. I’m not too concerned by this, as I, myself, have been going this way: enjoying SHG image reconstruction, then figuring out how it works, then asking myself questions like "why is it even possible that we measure Doppler velocities so small?": this is an intellectual construct that takes time and we can do better, I think, at brining this to the masses, sharing our insights.

This feature joins the growing list of scientific capabilities in JSol’Ex: Ellerman bomb detection, active region identification, and now differential rotation measurement. Each of these brings professional-level analysis within reach of amateur astronomers, but, as always, be cautious with what I’m saying: I’m not a scientist, just an engineer. I have no doubts that I made approximations, or took liberties that I should probably not have taken.

However, I do think that this brings value to the ecosystem.

Bibliography

Scientific Papers

Amateur Resources

JSol’Ex 4.0.0 est sorti !

Thu, 11 Sep 2025 00:00:00 +0200

Le 18 février 2023, je montais à l’Observatoire du Pic du Midi de Bigorre pour y passer une "Nuit au Sommet", une expérience mêlant observation nocturne et visite des installations scientifiques. J’y ai découvert avec fascination le coronographe Bernard-Lyot, l’expérience CLIMSO gérée par les Observateurs Associés de l’Observatoire Midi-Pyrénées.

Des étoiles plein les yeux (et une en particulier, notre soleil), ma vie d’astronome amateur centrée sur le ciel profond allait basculer vers l’observation de notre Soleil. De retour à la maison, j’ai commencé à me renseigner sur le matériel solaire, et découvrait avec stupéfaction qu’il fallait investir, beaucoup, pour pouvoir observer notre astre en toute sécurité : filtres pleine ouverture, étalons, … tout avait un coût démesuré pour un débutant comme moi. De quoi décourager, jusqu’à ce que je tombe par hasard sur le projet Sol’Ex, initié par la légende Christian Buil.

Ce projet, très didactique, permet de se construire un spectrohéliographe, par impression 3D. Un tel instrument présente de nombreux avantages : coût modeste, résolution élevée et la possibilité d’observer dans de nombreuses longueurs d’ondes tout en mettant à disposition des données exploitables scientifiquement !

L’inconvénient, si c’en est vraiment un, de cet instrument est qu’il ne produit pas une image du soleil observable directement à l’oculaire. Il est nécessaire d’utiliser un logiciel de reconstruction qui, à partir d’une vidéo enregistrant un "scan" du soleil, permet de générer une (voir plus) images du soleil. Ce processus de reconstruction, bien mystérieux pour moi à l’époque, était réalisé par un logiciel écrit par Valérie Desnoux, nommé INTI. Etant développeur (Java), je me suis alors lancé un challenge à l’époque : puisque je ne comprenais pas comment ça fonctionnait, j’allais essayer de créer moi-même un logiciel pour faire cette reconstruction. De cet appétit est né JSol’Ex.

JSol’Ex 4.0 et la collaboration scientifique

Nous voici 2 ans et demi plus tard, JSol’Ex arrive en version 4. Entre temps, le nombre d’utilisateurs a explosé et mon logiciel est exploité non seulement par les utilisateurs du Sol’Ex de Christian Buil, mais aussi pour des spectrohéliographes commerciaux comme le MLAstro SHG 700 (que j’utilise désormais). JSol’Ex a apporté de nombreuses innovations, comme la possibilité d’exécuter des scripts pour générer des animations, faire du stacking, créer des images personnalisées, ou encore la détection automatique d’éruptions, des régions actives (avec annotation), la détection de bombes d’Ellerman, la correction des bords dentelés et j’en passe.

Récemment, JSol’Ex a été utilisé pour réaliser un Atlas de Spectrohéliogrammes, et mentionné dans un article précisément sur la détection des bombes d’Ellerman. Bref, JSol’Ex était devenu suffisamment mature pour faire de la "vraie science", chose qui m’a toujours un peu effrayé pour être honnête.

Ainsi, j’ai toujours refusé d’intégrer une fonctionnalité dans JSol’Ex, qui m’a pourtant été demandée de nombreuses fois : la possibilité de l’utiliser pour soumettre des images dans la base de données BASS2000 de l’Observatoire de Paris-Meudon. Plusieurs raisons à cela : d’une, je ne faisais pas confiance à mon propre logiciel pour sa qualité "scientifique". Si je le savais capable de produire de "belles images", il est tout à fait différent de s’en servir pour une exploitation scientifique. D’autre part, il n’était pas question pour moi d’intégrer une fonctionnalité qui avait été développée par Valérie Desnoux (autrice de INTI) et l’équipe de Meudon sans son accord, par respect pour son travail.

Cependant, vous aurez compris que les temps ont changé et que la pression des utilisateurs ainsi que mes discussions avec Florence Cornu, du projet SOLAP, aux Rencontres du Ciel et de l’Espace fin 2024, puis aux JASON en Juin dernier ont eu raison de mes doutes. Avec son accord et celui de Valérie, je suis donc heureux de vous annoncer que JSol’Ex 4 est officiellement supporté pour soumettre vos images dans la base de données BASS 2000 !

Un assistant pas à pas

Ne souhaitant pas faire les choses à moitié, j’ai particulièrement peaufiné le logiciel pour simplifier cette procédure. J’ai souhaité faire les choses avec sérieux, puisqu’il s’agit d’une base de données professionnelle, exploitée par des scientifiques. Ainsi, une chose importante pour moi était de guider l’utilisateur dans ce processus et de lui faire comprendre l’importance de la qualité des données, tout en simplifiant le processus de soumission.

Figure 1. Assistant de soumission BASS2000

Ainsi par exemple, BASS2000 n’a que très peu de tolérance sur les problèmes d’orientation de l’image : il faut que le Nord solaire soit à moins de 1 degré d’erreur, sinon l’image sera refusée. L’assistant intègre donc un outil pour aider à vérifier l’orientation et corriger les erreurs minimes dues par exemple à une mise en station imparfaite.

L’assistant guidera aussi l’utilisateur dans son processus de soumission, en lui demandant de bien vérifier toutes les métadonnées associées à l’observation, et ira jusqu’à envoyer l’image sur le serveur FTP de BASS2000 pour validation par les équipes.

Je suis particulièrement reconnaissant à Florence CORNU pour son aide afin que les images JSol’Ex soient acceptées dans la base et je remercie mes beta-testeurs qui ont patiemment testé mes versions de développement pendant l’été.

Je ne vous cache pas qu’il s’agit là d’une forme de reconnaissance de mon travail, des centaines d’heures passées soirs et week-ends à développer ce logiciel qui rappelons-le est entièrement Open Source et gratuit.

Enfin, je ne peux pas m’arrêter sans vous annoncer une deuxième bonne nouvelle : non seulement vous pourrez utiliser JSol’Ex pour soumettre vos images acquises avec un Sol’Ex, mais aussi avec le MLAstro SHG 700, qui devient officiellement supporté dans la base BASS2000 !

Changements dans l’interface

Pour cette version j’ai aussi souhaité moderniser un peu l’interface graphique et la simplifier : avec le nombre de fonctionnalités croissantes arrive ce moment fatidique que tout développeur redoute : que l’interface ne devienne trop complexe pour les nouveaux utilisateurs et ne parle qu’aux anciens. J’ai essayé d’éviter cet écueil au cours du temps en refusant certaines fonctionnalités trop "de niche", mais sans pouvoir pour autant réussir à avoir une interface totalement claire.

Figure 2. Aperçu des changements d’interface

Cette nouvelle version essaie donc de regrouper les paramètres par sections plus claires, tout en ajoutant des infobulles pour guider les utilisateurs, anciens comme nouveaux, dans cet esprit qui est que le logiciel se doit d’être le plus didactique possible : j’essaie, autant que faire se peut, de vous transmettre en tant qu’utilisateurs ce que moi-même j’apprends en développant ce logiciel. Un exemple frappant de cette philosophie, c’est cette fonctionnalité que j’avais ajoutée qui permet de montrer en cliquant sur le disque solaire, à quelle image il correspond dans le fichier source (une vidéo contenant des spectres).

Détection d’ellipse assistée par l’utilisateur

Je vous parlais un peu plus haut de l’Atlas des Spectrohéliogrammes. Cet atlas, réalisé par Pál Váradi Nagy, nécessite un travail énorme d’analyse des données et est réalisé à l’aide de scripts JSol’Ex. Cependant, pour certaines longueurs d’ondes ou pour parfois des images un peu compliquées à traiter, le logiciel peut échouer à détecter correctement les contours du disque solaire. Ceci peut arriver notamment lorsque les images sont peu contrastées ou lorsqu’il y a des réflexions internes qui biaisent la détection.

Afin de résoudre ces cas complexes qui, encore une fois, nuisent à l’analyse scientifiques, j’ai ajouté la possibilité d’aider le logiciel à détecter les contours, et ainsi à obtenir un image solaire bien ronde:

Figure 3. Détection d’ellipse assistée par l’utilisateur

L’IA à la rescousse

Je suis développeur et en tant que tel, sur ce media, il me semblait pertinent d’ajouter une section sur la façon dont cette version a été développée.

Cette version est la première version qui a été développée avec l’aide de l’IA, en particulier Claude Code. En effet, les dernières innovations en matière d’IA agentique sont pour le coup la véritable révolution à venir : autant avant, avoir une IA qui n’était pas capable de comprendre le contexte, faire des refactorings ou prévoir un plan de développement ne les rendait pas particulièrement utiles, autant les IA à base d’agents, qui sont capables d’analyser votre code, appeler des outils de manière autonome et avoir de vraies interactions avec vous sont un "game changer" de mon point de vue.

Dans cette version, j’ai donc utilisé Claude Code (plan Pro) pour m’aider à faire les refactorings dont j’avais besoin et m’assister dans une tâche où je ne suis pas particulièrement à l’aise : le design d’interfaces graphiques.

De manière générale, ça fonctionne plutôt bien. Très bien même. La capacité de l’outil à définir un plan d’implémentation et comprendre les besoins est assez fascinante. Le code généré, en revanche, nécessite toujours beaucoup de review. On pourrait dire que j’ai l’impression d’avoir un (très bon) stagiaire avec moi, en permanence. En tant que développeur senior, je suis assez rapide à identifier là où l’IA complique les choses inutilement, ou utilise des design pattern dépréciés, ou ne respecte pas les conventions de code. Je serai, par exemple, terrifié si le code original produit était parti en production sans review. Mais, en lui donnant les bonnes directions, en lui expliquant ses erreurs, on arrive rapidement à ce que l’on souhaite avec la qualité que l’on souhaite.

Quelques exemples de choses qui ne fonctionnent pour le coup vraiment pas bien:

Claude vous demande de créer un fichier CLAUDE.md qui comprend des instructions sur comment compiler votre projet, comment il est organisé, etc. Bref, du contexte qui est systématiquement ajouté à chaque session. Pourtant, à l’utilisation, Claude ignore allègrement ces instructions.
En Java, les fichiers .properties sont encodés en ISO-8859-1, même dans une base où tous les sources sont en UTF-8. C’est une bizarrerie historique, mais Claude ne la comprend absolument pas. A chaque fois qu’il modifie mes fichiers properties (qui servent à l’internationalisation de l’interface), il casse systématiquement l’encodage. Pour l’éviter, le dois systématiquement, avant de lui faire faire une opération dont je sais qu’elle implique ces fichiers, lui dire "respecte les guidelines du fichier CLAUDE" où je lui ai donné une technique pour éviter les problèmes (convertir le fichier en UTF-8, puis l’éditer, puis le reconvertir en ISO-8859-1)
Les commentaires dans le code. Quand on commence à avoir un peu de bouteille comme moi, il est assez insupportable de lire des commentaires "captain obvious", ce genre de commentaires qui dit "la ligne suivante calcule 1+1". Claude en génère beaucoup. Trop. Et malgré le fait que mon fichier CLAUDE lui interdise explicitement.
La confiance en soi. Claude est bien trop optimiste et tend trop à flatter l’utilisateur. Par exemple, si je lui mens explicitement ("ton algorithme est faux parce que XXX"), il répondra systématiquement "Tu as raison !" sans "réfléchir" (j’utilise les guillemets avec intention), comme un tic de langage ! Ca devient assez frustrant à la longue, lorsqu’il ne comprend pas un problème ou complique inutilement l’implémentation.
Les limites : on arrive, sur un projet de la taille de JSol’Ex, très rapidement aux limites d’usage même sur un forfait Pro. Si je suis satisfait de ce que ça m’apporte pour le prix, je ne suis pas prêt à payer les 200€ / mois pour augmenter ces limites. N’oublions pas que je fais ça sur mon temps libre… je n’ai aucune obligation de résultat !

Quoi qu’il en soit, il s’agit là d’avancées qu’il devient difficile d’ignorer. Et ceux qui me connaissent savent à quel point je suis critique sur l’utilisation des IA, les mythes autour de ce qu’elle est capable de faire et son impact écologique. Néanmoins, une chose est certaine : les dirigeants qui pensent économiser du temps et de l’argent en virant leurs développeurs pour les remplacer par de l’IA se mettent une balle dans le pied : elle pose de sérieux problèmes de qualité de code et donc de maintenance, et, avec l’arrivée des agents, posent de graves problèmes de sécurité (un outil autonome qui décide par lui même s’il faut vous demander l’autorisation pour exécuter une commande !"). L’IA doit être vue comme une aide à la productivité, mais qui doit être cadrée par des gens d’expérience. Bref, nous sommes à un tournant et je ne suis pas encore bien à l’aise avec ce que cela implique. Nous n’avons jamais été aussi près du rêve de gosse que j’avais d’IA "autonomes", mais la maturité me force à en avoir peur.

Mais nous nous éloignons là du sujet initial et concluons donc ce billet : dites bienvenue à JSol’Ex 4, consultez la vidéo de présentation ci-dessous et n’hésitez pas à contribuer !

Ellerman Bombs Detection with JSol’Ex 3.2

Sat, 17 May 2025 00:00:00 +0200

About Ellerman Bombs

Ellerman bombs were first described by Ferdinand Ellerman back in 1917. Ellerman’s article was named -Solar Hydrogen "Bombs"- and it’s only later that these were commonly referred to as "Ellerman bombs".

I came upon this term while reading an article from Sylvain or André Rondi quite early in my solar imaging journey, and I was since then obsessed by these phenomena.

Ellerman Bombs are small, transient, and explosive events that occur in the solar atmosphere, particularly in the vicinity of sunspots. These are very short lived compared to other solar features: from several dozens of seconds to a few minutes. The most common explanation is magnetic reconnection, which occurs when two magnetic regions opposite polarity come into contact and reconnect, releasing energy in the form of heat and light. This is for example described in this article from González, Danilovic and Kneer.

Amateur observations of Ellerman bombs are quite rare, but Rondi described such observations using a spectroheliograph back in 2005. They are rare because they are mostly invisible where the amateur observations are made (the center of the H-alpha line), and are too small to notice. However, these are visible in the wings of the H-alpha line: this is where a spectroheliograph comes in handy, since the cropping window that is used to capture an image contains more than just the H-alpha line.

I had a long standing issue to do something about it in JSol’Ex, and I finally got around to it: all it took was getting some test data to entertain the ideas.

Observation of an Ellerman Bomb

I am doing many observations of the sun: I started in 2023 with a Sol’Ex, then I recently got an SHG 700, so I have accumulated quite a bit of data, which is completed by scans which are shared with me by other users of JSol’Ex.

I have been looking for Ellerman bombs in my data, but I never found any yet. This changed a couple weeks ago: I was doing some routine work on JSol’Ex and using a capture I had done in April, 29, 2025 at 08:32 UTC, when I noticed, by accident, a bright spot in the continuum image:

It may not be obvious at first, which is precisely why these are hard to spot, so here’s a hint:

JSol’Ex offers the ability to easily generate animations of the data captured at different wavelengths, so I generated a quick animation, which shows the same image at ±2.5Å from the center of the H-alpha line:

We can see the typical behavior of an Ellerman bomb: it is bright in the wings of the H-alpha line, but it vanishes when we are at the center of the line. The fine spectral dispersion of the spectroheliograph makes it possible to highlight this phenomenon very precisely.

The corresponding frame of the SER file shows the aspect of the Ellerman bomb in the spectrum:

The shape that you can see is often referred to as the "moustache". At this stage I was pretty sure I had observed my first Ellerman bomb, and that I could implement an algorithm to detect it.

JSol’Ex 3.2 auto-detection

JSol’Ex 3.2 ships with a new feature to automatically detect Ellerman bombs in the data. Currently, it is limited to H-alpha, but it should be possible to detect these in CaII as well.

The algorithm I implemented uses statistical analysis of the spectrum to match the characteristics of the "moustache" shape, in particular:

a maximum of intensity around 1Å from the center of the line
a distance which spreads up to 5Å from the center of the line
a brightening which is only visible in the wings of the line

JSol’Ex will generate, for each detection, an image showing the location of the detected bombs:

And for each bomb, it will create an image which shows the region of the spectrum which is used to detect the bomb. This is for example what is automatically generated for the bomb described above:

Algorithm details

Note	This is a description of the algorithm that I implemented in an adhoc fashion: I’m not a mathematician nor a scientist: I’m an engineer and the algorithm above was implemented using my "intuition" of what I thought would work. It is likely to change as new versions are released.

The algorithm is based on the following steps:

for each frame in the SER file, identify the "borders" of the sun
perform a Gaussian blur on the spectrum to reduce noise
within the borders, compute, for each column, the average intensity of the spectrum for the center of the line and the wings separately. The center of the line is defined as the range [-0.35Å, 0.35Å] and the wings as the range [-5Å, -0.35Å[ ∪ ]0.35Å, 5Å]
compute the maximum intensity of the wings, starting from the center of the line, and going outwards until we reach the maximum intensity (local extremum)
compute the average of each column average intensity for the wings (the "global average")

With Ellerman bomb scoring parameters defined below, the algorithm proceeds per column:

For each column index x in the spectrum image:
Build a neighborhood of up to 16 columns around x: Nₓ = { x + k | k ∈ ℤ, |k| ≤ 8 }, clamped to the image boundaries.
Compute the overall mean column intensity Ī_global = (1 / N_total) ∑_{j=1..N_total} Ī(j).
Exclude any columns in Nₓ whose average intensity falls below 90 % of Ī_global, since very dark columns (usually sunspots) would pull down our estimate of the local wing background and hide true brightening events. Call the remaining set Mₓ and let m = |Mₓ|.
If m < 1, there aren’t enough valid neighbors to form a reliable background—skip column x.
On the Gaussian-smoothed data, measure three key values at column x:
c₀ ≔ I_center(x), the mean intensity in the core region of the spectral line.
c_w ≔ I_wing(x), the average intensity across the two wing windows at ±1 Å.
c_max ≔ max_{p ∈ wing-pixels nearest ±1 Å} I(p, x), the single highest wing intensity near the expected shift.
Compute the local wing background r₀ = (1 / (m−1)) ∑_{j ∈ Mₓ, j≠x} I_wing(j). Using only nearby “bright enough” columns keeps the background estimate from being skewed by dark features.
Define a line-brightening factor B = max(1, c₀ / min(r₀, I_center,global)). Ellerman bombs boost the wings without greatly brightening the core, whereas flares brighten both.
Form an initial score S₀ = 1 + c_max / min(r₀, I_wing,global), where I_wing,global = (1/N_total) ∑_{j=1..N_total} I_wing(j). This compares the local wing peak to the typical wing level across the image.
Adjust for how many neighbors were used: S₁ = S₀ × (m / 16). Fewer valid neighbors mean less confidence, so the score is scaled down proportionally.
Compute the wing-to-background ratio rᵢ = c_max / r₀. If rᵢ ≤ 1.05, the wing peak is too close to the local background and the column is discarded. Otherwise, we boost the score further:
Raise S₁ to the power of e^{rᵢ}, giving S₂ = S₁^( e^{rᵢ} ). This makes the score grow quickly when the wing peak stands out strongly.
Multiply by √(c_max / c₀) to get S₃ = S₂ · √(c_max / c₀). That emphasizes cases where the wings are much brighter than the core.
Finally, penalize any shift away from the ideal ±1 Å wing position. If y_core and y_max are the pixel locations of line center and wing peak, compute Δλ = |y_max – y_core| × (Å / pixel), then S_final = S₃ / (1 + |1 Å – Δλ|).
If S_final > 12, mark column x as a candidate event. Use the value of B to decide:
When B < 1.5, it behaves like an Ellerman bomb (wings bright, core unchanged).
When B > 2, it matches a flare (both core and wings bright).
If 1.5 ≤ B ≤ 2, the result is ambiguous and ignored.

All thresholds (0.9× global mean, 1.05 ratio, score > 12, B cutoffs) were chosen by testing on data and visually inspecting results.

Post-filtering

There will often be cases where the same bomb is detected in multiple frames. Therefore, we need to do some merging of bombs which are spatially connected.

Eventually, we apply a limit threshold: if there are more than 5 Ellerman Bombs detected in an image, then we consider the detection to be false positives (this happens typically on saturated images, or images with too much noise). This is a bit arbitrary, but it seems to work well in practice.

Test data

I had about ~1100 scans I could reuse for detection, and it successfully discovered Ellerman bombs candidates in about 10% of them. Of course this required some tuning and several runs to get the parameters right. This doesn’t mean that you have 10% chances of finding an Ellerman bomb in your data, because the test data I have is biased (I often do 10 to 20 scans in a row, within a few minutes, to perform stacking, so if a bomb is detected in an image, it has decent chances of being detected in the next one). Also, I am using the term "Ellerman Bomb candidate", because there’s nothing better than visual confirmation to make sure that what you see is indeed an Ellerman bomb: an algorithm is not perfect, and it may fail for many reasons (noise, saturation, artifacts, etc.)

Here are a few examples of Ellerman bombs candidates detected in my data:

Conclusion

This blog post described my first visual Ellerman bomb detection. Then I described how I implemented an algorithm to automatically detect Ellerman bombs in JSol’Ex 3.2. I am very happy to release this to the wild, so that this kind of discovery is made more accessible to everyone. Of course, as I always say, you should take the detections with care, and always review the results. This is why you get both a global "map" of the detected bombs, and a detailed view of each bomb, which can be used to confirm the detection. In addition, I recommend that you create animations of the regions, which you can simply do in JSol’Ex by CTLR+clicking on the image then selecting an area around the bomb.

Finally, I’d like to thank my friends of the Astro Club de Challans, who heard me talk about Ellerman bombs detection for a while, showing them preliminary results, and who were very supportive of my work. Last but not least, thanks again to my wife for her patience, seeing me work on this (too) late at night!

Bibiography

Jagged Edges Correction with JSol’Ex 3.1

Fri, 2 May 2025 00:00:00 +0200

I’m happy to announce the release of JSol’Ex 3.1, which ships with a long awaited feature: jagged edges correction! Let’s explore in this article what this is about.

The dreaded jagged edges

Spectroheliographs like the Sol’Ex or the Sunscan are not using a traditional imaging system like, for example, in planetary imaging, where you can capture dozens to hundreds of frames per second and do the so called "lucky imaging" to get the best frames and stack them together to get a high resolution image.

In the case of a spectroheliograph, the image is built by scanning the solar disk in a series of "slices" of the sun: it takes several seconds and sometimes minutes (~3 minutes when you let the sun pass "naturally" through the slit) to get a full image of the sun.

In practice, this means that between each frame, each "slice" of the sun, the atmosphere will have slightly moved, causing some misalignment between the frames. This is also particularly visible when there is some wind, which can cause the telescope to shake a bit, and the image to be misaligned. Lastly, you may even have a mount which is not perfectly balanced, or which has some resonance at certain scan speeds.

As an illustration, let’s take this image captured using a Sunscan (courtesy of Oscar Canales):

This image shows 3 problems:

the jagged edges, which cause some unpleasant "spikes" on the edges of the sun
misalignment of features of the sun, particularly visible on filaments
a disk which isn’t perfectly round

These issues are typical of spectroheliographs, and are the main limiting factor when it comes to achieving high resolution images. Therefore, excellent seeing conditions are a must to get high quality images. Even if you do stacking, the fact that the reference image will show spikes is often a problem.

Correcting jagged edges

Starting with release 3.1.0, JSol’Ex ships with an experimental feature to correct jagged edges. It is not perfect yet, but good enough for you to provide feedback and even improve the quality of your images.

For example, here’s the same image, but with jagged edges correction applied:

And so that it’s even easier to see the difference, here’s a blinking animation of the two images:

The jagged edges are now mostly gone, the features in the sun are better aligned, and the image is much more pleasant to look at. There is still some jagging visible, the correction will never be perfect, but it is a good start.

In particular, you should be careful when applying the correction, because it could cause some artifacts in the image, in particular on prominences. As usual, with great powers comes great responsibilities!

How does it work?

To illustrate how the correction works, let’s imagine a perfect scan: a scan speed giving us a perfectly circular disk, no turbulence, no wind, etc.

In this case, what we would see during the scan is a spectrum which width slowly increases, reaches a maximum, and then decreases. The pace at which the width increases and decreases is determined by the scan speed and is predictable. In particular, the left and right borders of the spectrum will follow a circular curve.

Now, let’s get back to a "real world" scan. In that case, the left and right edges will slightly deviate from the circular curve. They will also follow the path of an ellipse: in fact, this ellipse is already required in order to perform geometric correction.

The idea is therefore quite simple in theory: we need to detect the left and right edges of the spectrum, then compare them to the ideal ellipse that we have computed. Pixels which deviate from this curve give us an information about the jagged edges. We can then compute a distortion map, which will be used to correct the image.

In practice, we also need to apply some filtering of samples: in practice, while the detection of edges is robust enough to provide us with a good geometric correction, it is not perfect. It can also be skewed by the presence of proms for example. Therefore, we are performing a sigma clipping on the detected edges, in order to remove outliers, that is to say pixels which deviate too much from the average deviation.

This is also why the correction will not work properly if the image is not focused correctly: you would combine two problems in one, and the correction would not be able to detect the edges properly.

In addition, in the image above you can see that the bottom most prominence is slightly distorted, which is caused by the fact that it’s far away from the 2 points which were used to compute the distortion. It may be possible to reduce such artifacts by using a smaller sigma factor (at the risk of undercorrecting edges).

Conclusion

In this blog post, I have described the new jagged edges correction feature in JSol’Ex 3.1. This solves one of the most common issues users are having with spectroheliographs, and I hope it will help you get better images. However, as usual, it’s a work in progress, so do not hesitate to provide feedback!

JSol’Ex 3.0.0 is out!

Mon, 14 Apr 2025 00:00:00 +0200

After dozens of hours of work, I’m happy to announce the release of JSol’Ex 3.0.0! This major release is a new milestone in the development of JSol’Ex, and it brings new features and improvements that I hope you will enjoy.

A bit of history

Since its inception as an educational project for understanding how the Sol’Ex works, JSol’Ex has grown into a powerful tool for processing and analyzing images captured with the Sol’Ex. However, it became very popular over time and started to be used outside the sole Sol’Ex community. In particular, it is now a tool of choice for many spectroheliographs owners.

I have always been keen on providing a user-friendly interface while keeping a good innovation pace. JSol’Ex was the first SHG software to offer:

automatic colorization of images
automatic detection of spectral lines
Doppler eclipse image, inverted image and orientation grid
automatic correction of the P angle
single click processing of Helium line images
embedded stacking
automatic trimming and compression of SER files
identifying what frame of a SER file matches a particular point of the solar disk
an optimal exposure calculator
automatic detection of redshifts
automatic detection and annotation of sunspots
automatic creation of animations of a single image taken at different wavelengths
a full-fledged scripting engine which allows creation of custom images, animations, etc.
support for home-made SHGs
1. and more!

All integrated into a single, easy to use, cross-platform application: no need for Gimp, ImPPG or Autostakkert! (but you can use them if you want to!).

For this new release, I wondered if I should change the name so that it better matches the new scope of the project, but eventually decided to keep it as it is, because it is already well known in the community and that changing it also implies significant amount of time spent on this that wouldn’t go into the new features.

Here comes JSol’Ex 3.0.0!

In addition to performance improvements and bugfixes, this release deserves its major version number because of many significant improvements.

Improved image quality

Better line detection

The first thing you may notice is the improved image quality. The algorithm to detect the spectral lines have been improved, which will result in a better polynomial detection and therefore a more accurate image reconstruction. This will be noticeable in images which have low signal, which is often the case in calcium.

Background removal

Next, a new background removal algorithm has been added. It is fairly common to have either internal reflections or light leaks in the optical path of a spectroheliograph. This results in images which are hard to process or not usable at all. This version of JSol’Ex is capable of removing difficult gradients. To illustrate this, here’s an image that a user with a Sunscan sent me:

The image on the left is unprocessed and shows important internal reflections. These are completely removed in the image on the right, processed automatically with JSol’Ex.

This background removal will only be applied to the "Autostretch" image, which is the default "enhanced" image that JSol’Ex is using, but it is also available as a standalone function in ImageMath scripts.

Physical flat correction

Another common issue with SHGs is the presence of vignetting, visible on the poles of the solar disk. The vignetting issue stems from the following factors, in the order of their impact:

the physical size of the SHG’s optical components — including the lens diameter, grating size, and slit length
the telescope’s focal ratio and focal length
the telescope’s own intrinsic vignetting (though this is rarely a significant factor)

For prebuilt SHGs like the MLAstro SHG 700, the size of the lens and grating is typically constrained by the housing design and cost limitations. As a result, vignetting often becomes an issue when using longer focal length telescopes,—especially when paired with a longer slit.

To fix this, JSol’Ex had until now the option to use artificial flat correction: the idea was basicaly to model the illumination of the solar disk via a polynomial and to apply a correction to the image. This works relatively well, but it can sometimes introduce some noise, or even bias the reconstruction on low-contrast images. On even longer slits, this artificial correction is not sufficient to remove the vignetting, so JSol’Ex 3 introduces the ability to use a physical flat correction.

The idea with a physical flat correction is to take a series of 10 to 20 images of the sun, using a light diffuser device at the entrance of the telescope, such as tracing paper, in order to diffuse light. The flat should be captured with the same cropping window as the one used for the solar images, but exposure will be longer, and possibly higher gain as well. The result is a SER file that JSol’Ex can use to create a model of the illumination of the disk, which can be used to correct the images.

As an illustration, here’s a series of 3 images of the Sun, taken with a prototype of a 10mm slit:

The image on the left is done without any correction and shows very strong vignetting. The image in the middle is done with the artificial flat correction, improves the situation, but still shows some vignetting. The image on the right is done with the physical flat correction, which is much better and shows no vignetting at all.

Flats can be reused between sessions, as long as you use the same cropping window and the same wavelength.

The physical flat correction can also be used on images taken with a Sol’Ex, in particular for some wavelengths like H-beta which show stronger illumination of the middle of the solar disk.

Note	Flat correction is not designed to fix transversalliums: it has to apply low pass filtering to the image to compute a good flat, which will remove the transverse lines. To correct transversalliums, use the banding correction parameters.

New Stretching Options

By default, JSol’Ex used to display images applying a linear stretch. Starting with this version, it is possible to select which stretching algorithm to use: linear, curve or no stretching at all.

Distance measurement tool

This version introduces a new tool to measure distances! This feature was suggested by Minh Nguyen from MLAstro, after seeing one of my images in Calcium H, which showed a very long filament:

This tool lets you click on waypoints to follow a path and make measurements on the disk, in which case the distances take the curvature into account, or outside the disk, for example to measure the size of prominences, in which case the distances are linear.

The measured distances are always an approximation, because it’s basically impossible to know at what height a particular feature is located, but it gives a good rough estimate.

New scripting features

Last but not least, this version significantly improves the scripting engine, aka ImageMath. While this feature is for more advanced users, it is an extremely powerful tool which lets you generate custom images, automatically stack images, create animations, etc.

In this version, the scripting engine has been rewritten to make it more enjoyable to use. It adds:

the ability to write expressions on several lines
the possibility to use named parameters
the ability to define your own functions
call an external web service to generate script snippets
the ability to import scripts into other scripts

As well as new functions. Let’s take a deeper look.

Declaring your own functions

You may have faced the situation where you wanted to apply the same operation to several images. For example, let’s imagine that you want to decorate an image with the observation details and the solar parameters.

Before, you would write something like this:

image1=draw_solar_params(draw_obs_details(some_image)
image2=draw_solar_params(draw_obs_details(some_other_image)

Now, you can define a function, let’s call it decorate, which will take an image and return the decorated image:

[fun:decorate img]
    result = draw_solar_params(draw_obs_details(img))

[outputs]
image1=decorate(some_image)
image2=decorate(some_other_image)

You can take a look at the documentation for more details.

Importing scripts

In the previous section we have seen how to define functions. It can be useful to externalize these functions in a separate file, so that they can be reused in other scripts. This is now possible with the import statement.

For example, let’s say you have a file called utils.math which contains the decorate function.

We can now import this file in our script:

[include "utils"]

[outputs]
image1=decorate(some_image)
image2=decorate(some_other_image)

This will import the utils.math file and make the decorate function available in the current script.

Named parameters

Named parameters are a new feature that allows you to pass parameters to functions by name, instead of by position. This is particularly useful for functions that take a lot of parameters, or when you want to make your code more readable.

For example, in the example above, we could have written:

[include "utils"]

[outputs]
image1=decorate(img: some_image)
image2=decorate(img: some_other_image)

The names of the parameters are documented here.

New functions

This version introduces a few new functions, which are available in the scripting engine:

bg_model: background sky modeling
a2px and px2a: conversion between pixels and Angstroms
wavelen: returns the wavelength of an image, based on its pixel shift, dispersion, and reference wavelength
remote_scriptgen: allows calling an external web service to generate a script or images
transition: creates a transition between two or more images
curve_transform: applies a transformation to the image based on a curve
equalize: equalizes the histograms of a series of images so that they look similar in brightness and contrast

And others have been improved:

find_shift: added an optional parameter for the reference wavelength
continuum: improved function reliability, enhancing Helium line extraction

The transition function, for example, is capable of generating intermediate frames in an animation, based on the actual difference of time between two images, offering the ability to have smooth, uniform transitions between images.

This is how my partial solar eclipse animation was created!

Acknowledgements

I would like to thank all the users who have contributed to this release by reporting bugs, suggesting features, and testing the software. In particular, I would like to recognize the following people:

Minh Nguyen, MLAstro’s founder for his help with the background removal and flat correction algorithms, as well as the new distance measurement tool and review of this blog post
Yves Robin for his testing and improvement ideas
my wife for her patience, while I was going to bed late every night to work on this release

Partial Eclipse of March 29, 2025

Sun, 30 Mar 2025 00:00:00 +0100

In this blog post I’m describing what is probably a world premiere (let me know if not!): capturing a partial solar eclipse using a spectroheliograph and making an animation which covers the whole event.

Edit: Turns out Olivier Aguerre did something similar, described here (french).

Preparations

On March 29, 2025, we were lucky to get a partial solar eclipse visible in France, with a maximum of about 25%. I wanted to do what was a first for me, capturing the event, so I used a TS-Optics 80mm refractor with a 560mm focal length, equipped with an MLAstro SHG 700 spectroheliograph. The Astro Club Challandais was organizing a group observation this morning, but my initial decision was not to attend. Instead, I opted to limit the risks by performing this somewhat complex setup at home, in familiar territory. Unlike the SUNSCAN, using a spectroheliograph like the SHG 700 requires more equipment: a telescope, a mount (AZ-EQ6), and in my case, a mini PC for data acquisition (running Windows) along with a laptop for remote connection to the PC.

I have conducted observations away from home before, but from experience, setting everything up—including WiFi, polar alignment, etc.—can be a bit too risky for an event like this. So, all week, I anxiously monitored the weather. Yesterday, the forecast looked grim, with thick clouds and rain. However, the gods of astronomy were merciful, blessing us with a beautiful day. The sky wasn’t entirely clear, but it was good enough for observations.

Some more context

First of all, I had a specific observation protocol in mind. As you may know, a spectroheliograph doesn’t directly produce an image—it requires software to process video scans of the Sun. In the case of the Sunscan, the software is built into the device, but for a Sol’Ex-type setup, an independent software handles this task. You are probably familiar with INTI, but I have my own software: JSol’Ex.

The advantage of developing my own software is that I was able to anticipate potential issues. One major challenge with an eclipse is that the software must "recognize" the Sun’s outline, which won’t always be perfectly round—in fact, it could be quite elliptical. The software corrects the image by detecting the edges, but when the Moon moves in front, the sampling points become completely incorrect, sometimes detecting the lunar limb instead of the Sun’s!

My strategy was to start early enough to adjust settings for minimal camera tilt and, more importantly, to ensure an X/Y ratio of 1.0. With these reference scans, I could then force all subsequent scans to use the same parameters. So, I began my first scans under a beautifully clear sky! After some adjustments, I was ready: a single scan, and we were good to go!

At the same time, I activated JSol’Ex’s integrated web server and set up a tunnel so my friends could watch my observations live! I planned to perform a scan every two minutes using a Python script in SharpCap, automating the recording, scan start, stop, and rewind. JSol’Ex’s "continuous" mode processed scans in real time. Everything was going smoothly… until panic struck—clouds!

For the past three hours, the sky had been perfectly clear. Yet, just ten minutes before the eclipse began, clouds started rolling in. What bad luck! Fortunately, by spacing out the scans, I managed to capture many of them in cloud-free moments.

The eclipse began, and the first scans featuring the Moon appeared. It worked! Forcing the X/Y ratio was effective!

As the scans piled up, I encountered a new problem. While locking the X/Y ratio helped, the software still needed to calculate an ellipse to determine the Sun’s center for cropping. But things started going wrong—the software was miscalculating everything. I had anticipated this, and I already had a workaround in mind, but the necessary code wasn’t deployed on my mini PC. So, I didn’t worry too much and simply shared the raw images, which were perfectly round—because, as you recall, I had already adjusted my X/Y ratio to 1.0!

I continued scanning, though my setup wasn’t perfectly precise. My polar alignment wasn’t flawless, and I had no millimeter-accurate return-to-start positioning. As a result, I had to manually realign between each scan. While this wasn’t a major issue, it did create some intriguing scans. For those familiar with Sol’Ex, seeing "gaps" in the spectrum due to the Moon’s presence was quite unusual, making centering more difficult.

The eclipse reaches maximum

Time passed, scans continued, and finally, we reached the maximum eclipse phase!

At one point, I wondered whether we could see lunar relief in the images. However, given the jagged edges typical of SHG imaging, it was hard to say for sure, but, I think some of the details visible below are actual surface details.

End of observation

By the end of the eclipse, I had 80 SER files, taken between 9:37 AM and 1:43 PM, totaling nearly 175 GB of data! It was time to transfer everything to my desktop PC to create an animation. This also gave me a chance to test whether my earlier workarounds for ellipse detection would function as expected. I ran batch processing, and boom—within minutes, I had this animation:

And here’s the continuum version which was weirdly compressed by Youtube:

This was just a first draft, using a beta version of my software. A few hours later I released a new version of the animation which is visible below:

Conclusion

In the end, the experiment was a success! I also took this opportunity to improve my software, which will benefit everyone. If you have eclipse scans, don’t discard them! Soon, you’ll be able to process them too.

The big question now is: Could this be done during a total solar eclipse, such as next year’s in Spain? Well, I feel lucky that this one was only 25% partial. Managing ellipse detection and mount realignment between scans is already quite tricky. During a total eclipse, there wouldn’t even be a reference point!

Unless one has flawless alignment, a mount capable of returning to position perfectly, and a steady scanning speed, this would be a real challenge. Honestly, it’s beyond my current expertise—it would require a lot more work.

P.S: For french speaking readers in west of France (or simply if you are nearby at that date), we organize the Rencontres Solaires de Vendée on June 7, where we can discuss this topic!

JSol’Ex errands: How it helped uncovering an inaccuracy of the SHG 700 specifications

Tue, 11 Mar 2025 00:00:00 +0100

For a few months, I’ve been working with Minh, from MLAstro, to improve support of the SHG 700 in JSol’Ex. While JSol’Ex was initially designed for Christian Buil’s Sol’Ex, it appears that lots of users are using it to process images acquired using a different spectroheliograph. A while back, I added the ability to declare the specifications of the spectroheliograph you are using, which makes JSol’Ex compatible with a wide variety of instruments. An example of collaboration with Minh is the experimental flat correction, which is recommended to enable with the SHG 700, and the addition of the SHG 700 to the list of instruments which are officially supported, in addition to the Sol’Ex and the Sunscan.

First light with SHG 700

As you can see, I had been working with MLAstro for quite some time already, but I didn’t own the instrument. This has recently been "fixed" and I’m now a happy user of the SHG 700!

I must say that Minh’s design is fairly impressive, it’s a truly qualitative instrument. The aluminum housing makes it extremely robust, without any flexion, and it comes pre-collimated. In addition, the micro-focusers, which are used for the collimator lens, for the camera objective and for the wavelength selection wheel, are extremely pleasant to use: anyone who has struggled with the collimation of the Sol’Ex will immediately feel the magic.

Being a regular user of the Sol’Ex, I immediately felt comfortable with the SHG 700: it only took me a few minutes to setup and get my first images! Fine tuning the focus is a breeze with the microfocusers, it’s really fantastic to use.

My first images were showcased by MLAStro, but here are a few:

A mystery

While the above images were fairly easy to produce, I was puzzled because I didn’t manage to get any decent image in Helium D3. This was surprising, because the sensitivity of the SHG700 is higher, especially because there’s no need for a ND filter or a an ERF, so it was curious that the only helium image I was able to produce was this:

To understand the problem was that, it is important to mention that unlike Minh, I was using JSol’Ex one-click, fully automated processing, a feature which was introduced a while ago (June 2024) and worked extremely well for both Sol’Ex and Sunscan files (N.B: the Sunscan app introduced the same feature a few days ago).

This feature only works if we can properly compute the dispersion of the spectrum, which is measured in Angstrom/pixel. In order to compute this, we need to know the specifications of the spectroheliograph, such as the grating density, the focal length of the collimator and objective lenses, the total angle as well as the pixel size of the camera. Once we know all this, we can determine how many pixels separate the reference line that we use, for example the Sodium D2 line, from the Helium D3 line. To illustrate this, let’s say that we have a dispersion of 0.1Å/px. The Helium line is to be found at 5875.62Å, when the reference line, the Sodium D2 line, is detected at 5889.95Å. Therefore, the Helium line can be found 14.33Å away from the D2 line, which means 14.33/0.1 = 143.3 pixels away.

Using both Sol’Ex and Sunscan, I had great experience at extracting this line automatically, so it was to say the least curious that it wouldn’t work with the SHG 700.

Investigation

So I contacted Minh, and after eliminating obvious possible candidates, like weak signal or incorrect exposition or gain, I decided to go the "old way" and searched for the helium line manually. It was with great surprise that I discovered that when JSol’Ex told me that the line should be 124.9, I was measuring something closer to 116 pixels! I brought this to Minh, and we identified 3 possible causes for this:

an error in the computation of the dispersion in JSol’Ex
a different focal length of the SHG 700
a grating which wouldn’t have the expected number of lines/mm

It was also possible, but unlikely, that a combination of 2 and 3 would happen. The first thing I’ve done is double checking my formula to compute the dispersion in JSol’Ex. I was doubtful that it could be wrong, given that it was used with success on different SHGs. In addition, it gave exactly the same result as the Ken Harrison' SimSpec SHG spreadsheet.

So we moved to the 2d option: the SHG 700 was advertised with a focal length of 75mm. However, the pixel shift I was manually measuring was closer to 70mm. I brought this again to Minh, who contacted the supplier of the lens, and here’s what he told me:

The lens from the first production run had a focal length of 72mm, a discrepancy I was unaware of at the time. I had sourced this lens from a supplier in China and provided them with the Zemax file for my self-designed 6-element Double Gauss 75mm lens. The first prototype was disappointing—its near-UV performance and coating were poor, and contrast was low across the field. This was largely due to my inexperience in optical design, as I had only begun learning Zemax a month prior, making this my first optical project.

I raised these concerns with the supplier, who was very accommodating and offered to help optimize the design while ensuring key parameters such as focal length, aperture, and exit pupil remained unchanged. With each revision, the lens improved—by the second and third prototypes, contrast was significantly better, sharpness in the blue end of the spectrum improved, and the field was much flatter across the FOV. The field of view closely matched the earlier lens, and because I had already tested focal length in previous iterations, I didn’t think to recheck it. Visually, the lenses appeared identical, except for a slight shift in coating hue.

The third prototype was approved for production and became the MLAstro "75mm" compound lens used in all MLAstro SHGs. However, it was only later confirmed by the supplier that the final production lens actually had a focal length of 72mm instead of 75mm. The optimizations for blue performance and field flattening had slightly shortened the focal length."

— Mihn Truong Nguyen
MLAstro

So I changed the focal length to 72mm and got this image instead:

and a stack of 5 images processed entirely in JSol’Ex:

That’s quite a difference! So it appeared that the software was correct, and that it allowed identifying a problem in the spectrograph specifications, because a spectral line wasn’t found where it should have been! While going from 75mm to 72mm won’t make much of a difference, the fact of not using the right numbers makes a huge difference in JSol’Ex: computations are all off, which includes the pixel shifts like in this exercise, but also the measured redshifts. In addition, this would make it impossible to perform more complicated tasks like finding the ionized Fe lines when imaging the corona E. The image is stil less contrasted than it should be, which may indicate that the computation is slightly off, or it’s just due to the weather conditions that day, I didn’t have the opportunity to retry.

Lastly, we can actually see fairly easily that the new focal length is a better fit, by using JSol’Ex "profile" tab. In that tab, we compare the profile of the spectrum that you captured with a reference spectrum from the BASS2000 database: this is also how the software automatically determines what spectral line is observed, by comparing the profiles together.

With a 75mm length and an H-alpha profile, here’s what we got:

You can see that while the H-alpha profile is found, as soon as we move towards the wings, there are slight shifts between the local minimas. When we switch to a 72mm length, these are perfectly aligned:

Conclusion

The SHG 700 is a fairly impressive instrument: it’s robust, it’s a pleasure to use with its microfocusers, and Minh is always super responsive and very patient. Its use doesn’t come without drawbacks, though. It’s weight, for example, restricts it to refractors which have a good focuser. The dimension of the sun is also smaller than with the Sol’Ex at equivalent focal length (see this post for an explanation). However, it produces stunning images, sometimes rivalizing these produced with an etalon.

While testing it, I faced this problem that the helium line images were significantly worse than with the Sol’Ex, which didn’t quite make sense. After investigation, it turned out we had highlighted a difference in the specifications, a lens had been changed from 75mm to 72mm by the supplier, without letting MLAstro know. That’s a pity, but, in the end, the problem is very easy to fix in JSol’Ex. Be sure to upgrade to JSol’Ex 2.11.2 which includes a fix to update the focal length.

Automatic active region detection with JSol’Ex 2.9

Sat, 1 Feb 2025 00:00:00 +0100

The latest release of JSol’Ex as of writing this blog post, JSol’Ex 2.9, ships with a new ability: automatically detecting active regions. In this blog post, I will explain the principles and the algorithm I used, but also show its limits.

About active regions

First of all, a bit of vocabulary. An active region is a region of the Sun atmosphere where special activity occurs, such as flares or coronal mass ejections. They don’t have to be associated with sunspots, but in general, they are. In JSol’Ex, we’re essentially detecting sunspots, and the terminology "active regions" essentially comes from the fact that we can use the https://www.swpc.noaa.gov/NOAA database] to label these active regions.

Spectroheliographs

An instrument like Christian Buil’s Sol’Ex, called a spectroheliograph (or SHG in short) doesn’t offer any kind of "live view" like typical solar instruments like Coronado or Lunt instruments. Instead, what we get is a video file, where each frames consists of a "slice" of the sun, observed as a portion of the light spectrum:

The video above is an excerpt from a so-called "scan": the principle is to have a slice of the sun passing through a slit, and a grating is used to spread the light into a spectrum. Here, we are observing a curved dark line, which is the H-alpha line, and the "wobbling" we see around that line is an illustration of the Doppler effect. The idea of software like Valérie Desnoux’s INTI and JSol’Ex is to extract pixels from the studied line (here H-alpha) in order to reconstruct an image of the sun.

However, as you can see in that animation, there’s a lot more information available in each frame. While we are mostly interested in the central line (here H-alpha), it’s also interesting to look "above" or "below" the line, which is often referred to "pixel shifting", in which case we’re not studying H-alpha, but a different wavelength: that’s one of the strenghts of Sol’Ex, which makes it possible to do "science" at home!

In particular, this video shows a couple interesting features:

sometimes, you see some vertical dark lines appearing: these are features of the solar atmosphere. The darker lines which spread around multiple columns are these we are mostly interested in for this blog post: they correspond to sunspots!
near the end of the video, on the right, you will see a white flash apparearing inside of of these regions: it’s a solar flare!

This video is actually an excerpt from one of the many captures I’ve done with Sol’Ex, and I captured it during an eruption, on May 5th, 2024. What you are seeing here is the massive AR3664 region which was suject to multiple X-flares and resulted in beautiful auroras on Earth!

Automatic active region detection

I’ve always considered that it would be a cool feature to be able to detect these lines automatically in JSol’Ex, and provide an overlay of the detected sunspots. Here we go, it is finally implemented in JSol’Ex 2.9.0! If you select the "full processing mode" or that you check the "active regions" checkbox in a custom process, then JSol’Ex will automatically create an image of the sun with the detected active regions as an overlay:

Unfortunately that day I didn’t capture a full disk, but you can see that JSol’Ex annotated the disk with the detected active regions, but it also added the labels for these regions automatically.

Let’s take another example with an image captured in Calcium K on January 17, 2025:

You will notice 2 different colors for labels:

blue labels are the ones detected by JSol’Ex
red labels are the ones coming from the NOAA database, which were not detected

Sometimes, you will see like in the image above regions which are not detected, and others which are detected but not in the NOAA database. The reason why not all of them are detected is that I had to choose "reasonable" detection thresholds, which work for most use cases. I plan to improve the algorithm over time to make it more robust, based on your feedback.

How does it work?

The algorithm is based on a simple analysis of each frame. If you open your SER file in JSol’Ex video analyzer, you will see something similar to this:

On the top, you see the original frame, with the curved H-alpha line. On the bottom, you see the corrected frame, where the H-alpha line is straightened, and the active regions are highlighted in purple.

The algorithm is based on the following steps:

For each frame:

Detect the borders of the sun
For each column, compute the average intensity of the column, as well as its standard deviation
Compute a 3rd order polynomial fit of the average intensity and standard deviation
now, for each column, we compare its average intensity and standard deviation to the polynomial fits: if both of them are below a particular threshold, we consider that we have detected a candidate

The reason to only keep candidates which have both average intensity and standard deviation below a threshold is because:

sunspots characterize by a clear vertical line, which means that the standard deviation is low (most pixels have a similar value on the column)
sunspots are darker than the surrounding, which means that the average intensity is lower

At the moment, I’m using the following thresholds (note that they may change in future releases, as I improve the accuracy):

average intensity threshold: 0.95 times the predicted value
standard deviation threshold: 0.85 times the predicted value

Once we have the results for all frames, we aggregate active regions by collecting adjacent candidates.

Finally, we filter out regions which are too small, then perform clustering of regions which are close to each other.

The whole algorithm can be found here (note that it also includes redshift detection, which was discussed in a different blog post).

Limitations

This algorithm proves to work relatively well in many different wavelengths (H-alpha, calcium, magnesium …). However, there are sometimes false positives. This is for example the case when I’m scanning in H-beta, due to the long focal length of my telescope and astigmatism, and the fact that H-beta provides a lot of details.

In this case, the algorithm detects areas which are actually just noise:

Note that these are close to the north and south poles. I have also noticed that the algorithm tends to detect noise as active regions when we’re close to the limb, which is also why I’m currently filtering these out.

At this stage, I have chosen not to make the detection thresholds configurable, because I consider these internal implementation details, which may change in the future, and that I don’t want to expose to the user.

Annotation of active regions

Once piece of work that we didn’t explain yet is how JSol’Ex puts labels around active regions. For this, we are using the NOAA Solar Region Summary database.

This provides us, at a particular date, the list of active regions with their positions on the solar disk. JSol’Ex will compare these with the detected regions, and put the labels in different colors based on whether they were detected or not.

However, the position of active regions will only be correct if:

you have properly oriented your image (north at the top, east on the left): to help you with this, use the "GONG" tab on the right of the interface to download a reference image and compare with yours
you are using an equatorial mount. If you are not, then make sure to check the new "alt-az" mode and enter your GPS coordinates in the settings, so that JSol’Ex can compute the parallactic angle of the sun at the moment of the observation and automatically correct the orientation of the image.

The data from NOAA is cached in your local filesystem, so that we don’t have to download it every time you open a video file.

Frequently Asked Questions

Despite only having released this a day ago, I already have received the same question multiple times: can JSol’Ex be used to annotate an existing image of the sun, that is to say, take a JPG or PNG image and annotate it?

If you have read carefully this blog post and that I explained things correctly, you will have understood that the answer is no: because my algorithm is based on the analysis of each frame, there’s no such information available in a single image.

Conclusion

I hope you will enjoy this new feature in JSol’Ex 2.9.0. This blog post is a tentative explanation of the algorithm, and I will be happy to answer any questions you may have about it. As always, feel free to contribute, JSol’Ex is open source!

Extraction automatique d’images Hélium avec JSol’Ex

Fri, 14 Jun 2024 00:00:00 +0200

Dans ce billet, je vais prendre exemple d’un cas d’utilisation particulier, l’extraction d’images Hélium, pour expliquer comment j’envisage le développement de JSol’Ex, dans l’optique de simplifier, d’une version à une autre, son exploitation par les utilisateurs.

Je rappelle à toutes fins utiles que le logiciel officiel n’est pas JSol’Ex mais bien INTI de Valérie Desnoux.

Principe du Sol’Ex

Rappelons tout d’abord le principe de base du Sol’Ex de Christian Buil. Le concept requiert de reconstituer une image du soleil, ligne par ligne, en exploitant une vidéo dont chaque trame montre une partie du spectre solaire, centré sur une raie particulière que l’on étudie. La raie la plus communément étudiée est la raie H-alpha, qui, sur le spectre solaire, apparaît en absorption, c’est à dire qu’elle sera une ligne sombre sur une image du spectre :

Figure 1. Image du spectre solaire, raie H-alpha sombre détectée

En haut, l’image telle qu’on la trouve dans la vidéo, la ligne est "déformée". En bas, l’image "corrigée". Les points bleus correspondent à la détection de l’effet Doppler, les lignes sombres verticales correspondent à des taches solaires.

Les logiciels comme INTI et JSol’Ex exploitent le fait que la ligne étudiée est la plus sombre de l’image pour se repérer. Un des aspects fondamentaux de la reconstruction que je n’avais honnêtement pas compris lorsque je me suis lancé dans ce projet, c’est que tout le signal permettant de recomposer l’image H-alpha se trouve dans cette ligne sombre. En effet, même si elle paraît "noire", il y a en fait bien d’infimes variations dans cette ligne, et c’est celà qui nous sert à reconstituer une image !

Tip

8 bits vs 16 bits

Cet aspect explique l’importance d’utiliser une acquisition en 16 bits et non en 8-bits. En effet, si l’acquisition est faite en 8 bits, on ne dispose que de 256 valeurs possibles pour encoder l’ensemble de la dynamique de l’image. Si le signal ne se trouve qu’au centre de la raie, on comprend vite qu’on ne dispose alors que d’un nombre très limité de valeurs pour ce qui se passe dans cette simple région. En utilisant 16 bits, on augmente sensiblement notre capacité à encoder des valeurs de niveaux différents dans le centre de la raie, puisqu’on dispose cette fois de 65536 valeurs possibles pour l’ensemble de l’image !

Le cas des raies Hélium

La détection de raies sombres permet, si la fenêtre de cropping du spectre est relativement petite, de se concentrer sur une raie à étudier. Ca fonctionne très bien pour les raies H-alpha, Magnésium, Calcium K ou H, etc, qui sont des raies en absorption.

En revanche, la raie Hélium est une raie en émission : elle est "claire" au lieu d’être sombre, mais elle est aussi impossible à distinguer dans la plupart des trames parce que noyée dans le flux. Aussi les logiciels ne peuvent pas la trouver.

Alors, comment fait JSol’Ex, dans ses dernières versions, pour être capable de produire en 1 clic une image Hélium comme celle ci-dessous ? La question est intéressante non seulement d’un point de vue technique, mais aussi pour comprendre la façon dont j’aborde le développement de JSol’Ex.

Le site de Christian Buil est une nouvelle mine d’information pour nous aider : pour pouvoir produire une telle image, il est nécessaire de prendre une raie sombre comme référence, puis d’appliquer un "décalage de pixels" pour "trouver" la raie Hélium. Enfin, il est nécessaire de procéder à une soustraction du continuum (une 2ème image).

Cette notion de décalage de pixels est bien connue des utilisateurs chevronnés du Sol’Ex. Le principe est relativement simple : au lieu de reconstituer une image en utilisant le centre de la raie détectée (la ligne rouge dans l’image spectrale ci-dessus), on va reconstituer une image en se décalant vers le haut ou vers le bas de quelques pixels. C’est ce même principe qui permet de créer des images Doppler du soleil, qui montrent en bleu les régions qui s’approchent de nous et en rouge celles qui s’éloignent. On peut même produire des timelapses assez spectaculaires comme cette animation que j’ai réalisée le 5 juin 2024 et qui est générée par le logiciel en quelques minutes :

Comme Christian l’explique dans cette vidéo, trouver ce décalage de pixels n’est pas forcément chose simple. Tout d’abord, il est nécessaire de trouver la raie "sombre" de référence : on utilisera traditionellement le doublet du sodium pour se répérer, puis on utilisera la raie Sodium D2 ou encore la raie Fe I présente à côté comme référence.

Ensuite, il faut trouver ce fameux décalage de pixels pour "tomber" sur la raie Hélium. La vidéo de Christian ne montre pas les dernières améliorations que Valérie Desnoux a intégré à INTI pour rendre les choses plus simples, mais, grossièrement, il faut :

faire une capture surexposée qui va nous permettre de "voir" la raie hélium
faire une capture "normale" pour disposer de l’image à réellement traiter
ouvrir INTI et traiter la première vidéo pour disposer de l’image moyenne montrant la raie Hélium en bord de champ
traiter la 2ème vidéo en mode "raie libre", utiliser "polynôme automatique" et ouvrir l’image moyenne calculée lors du premier traitement pour trouver le décalage
enfin utiliser un autre logiciel comme i-Spec pour faire la soustraction

Cette procédure, bien que documentée, est très intimidante pour les utilisateurs et explique que peu se soient lancés dans ce défi.

Dans JSol’Ex, la procédure est simplifiée au maximum :

on fait une capture "classique", intégrant la raie de référence (ex SoD2) et en utilisant un cropping assez large pour intégrer aussi la raie proche Hélium
on ouvre le fichier dans JSol’Ex : les images sont générées automatiquement, il n’y a rien à calculer, pas de soustraction de continuum, rien, juste un clic !

Ca ne prend que quelques secondes !

Le processus de simplification

ImageMath

Avant d’en arriver là, il a fallu de nombreuses améliorations, qui ont été livrées au fur et à mesure des versions. La première, c’est ce système de script que j’ai appelé "ImageMath" (le nom est inspiré de PixelMath dans PixInsight) : il permet d’utiliser des scripts pour produire des images que JSol’Ex ne produit pas tout seul. Cette avancée a permis d’écrire un script qui produit une image Hélium à partir d’un seul fichier SER :

Un exemple de script permettant de générer une image Hélium

[params]
# Entrer la valeur du décalage de raie
RaieHelium = -85
# Limites hautes et basses pour le continuum
ContinuumLo=-80
ContinuumHi=-70
# Stretch de l'image
Stretch=10

## Variables temporaires
[tmp]
continuum = max(range(ContinuumLo,ContinuumHi))
helium_raw = autocrop(img(RaieHelium) - continuum)

## Maintenant les images !
[outputs]
helium = asinh_stretch(helium_raw, blackPoint, Stretch)
helium_fixed = asinh_stretch(fix_banding(helium_raw;BandWidth;BandIterations),blackPoint, Stretch)
helium_color = colorize(helium_fixed, "Helium (D3)")

Néanmoins, vous noterez que ce script nécessite toujours de déterminer le décalage de la raie Hélium, ainsi que la position du continuum. Ceci pouvait cependant être fait simplement avec le même fichier SER en l’ouvrant dans le "débogueur de raie", une procédure que je décrivais à l’époque dans une vidéo.

Cette procédure permettait ainsi d’obtenir une image Hélium en moins de 5 minutes, ce qui était déja une amélioration sensible par rapport à avant : plus besoin de faire 2 vidéos distinctes et un décalage de pixels calculé par le logiciel avec l’assistance de l’utilisateur.

Bien que ce soit une amélioration notable, on peut faire encore mieux.

Profils spectraux

JSol’Ex affiche depuis longtemps dans un onglet le "profil spectral" de l’image étudiée et peut aussi calculer la dispersion, mesurée en Angrstöms par pixel, d’une image. Ce profil correspond à l’intensité des raies pour un décalage de pixels donné. Depuis la version 2.3.0 cependant, j’ai intégré une fonctionnalité qui permet de détecter automatiquement la raie étudiée grâce à la comparaison de ce profil à un profil de référence (les données sont issues de la base de données BASS2000). Grâce à celà, il est désormais possible de savoir comment une image "s’aligne" entre le profil de référence et celui qu’on étudie. Puisque l’on connait à la fois la position de la raie de référence (Sodium D2 par exemple) et la dispersion, il est alors possible de calculer de combien de pixels on doit se déplacer pour trouver la raie Hélium !

Cette fonctionnalité est d’ailleurs disponible dans les scripts sous le nom de 'find_shift`.

Soustraction du continuum

A ce stade, nous disposons donc d’une image dont on sait qu’elle contient une raie de référence (Sodium D2 ou Fer Fe I) mais aussi le décalage en pixels de la raie Hélium. Il nous manque cependant la soustraction du continuum. Là encore, le script ci-dessus montre qu’il fallait entrer une valeur "à la main" pour trouver ce qu’il fallait soustraire. Une façon simple de procéder était encore une fois d’ouvrir le débogueur et de regarder les lignes plus claires dans le spectre : l’oeil étant assez sensible aux changements de contraste, il n’était pas trop compliqué de trouver un intervalle raisonnable pour le continuum.

Néanmoins, si on souhaite arriver à un traitement complètement automatique, on ne peut plus se baser sur une valeur "pifométrique". Une façon naïve de régler le problème aurait été d’utiliser un décalage fixe (par exemple 15 pixels). Cependant, ça ne fonctionne pas, pour plusieurs raisons :

le décalage dépend de la résolution (taille des pixels, dispersion)
on peut tomber sur une raie trop sombre, cette région du spectre étant assez contrastée
le résultat est très sensible à l’exposition

Pour cette raison, j’ai ajouté une fonctionnalité qui calcule un "continuum synthétique".

Il faut noter que contrairement à l’image "continuum" qui utilise un décalage fixe (et configurable) dans les traitements standards, ce continuum synthétique est utilisé uniquement dans le contexte du traitement Hélium, ou lors de l’utilisation de la fonction continuum() dans un script.

L’idée de cette fonction est de calculer une image qui représente au mieux le continuum à soustraire. Au lieu d’utiliser un seul décalage de pixels, on va effectuer un calcul à partir de plusieurs décalages (cette fonction nécessite donc plus de ressources lors du traitement). Ainsi la première chose à comprendre c’est qu’il ne s’agira pas d’une image à une longueur d’onde précise, mais bel et bien d’une image synthétique basée sur des propriétés statistiques.

En premier lieu, on effectue un échantillonnage des images à différents décalages de pixels, du minimum possible au maximum possible par rapport à la fenêtre de cropping du spectre et de la distorsion. Par exemple, si chaque trame fait 2000x200 pixels, on dispose potentiellement de 100 décalages entiers (la hauteur de l’image). Cependant, à cause de la distorsion, seuls un sous-ensemble nous permet d’avoir des lignes complètes lors de la reconstruction (disons, 180 lignes, qui donneront 180 images). Sur ces 180 décalages possible, nous n’allons, pour des raisons de performance, uniquement en retenir un échantillon (environ 1/3), ce qui nous donnera donc 60 images à étudier. Sur ces 60 images, on élimine celles qui correspondent à un décalage de pixel trop proche de la raie de référence puisque de manière évidente elles ne correspondent pas au continuum.

Dès lors commence l’analyse statistique : pour chacune de ces images, on calcule leur valeur moyenne. Ensuite, nous calculons la moyenne de ces moyennes et nous ne conservons que les images dont la moyenne est supérieure à cette moyenne.

A ce stade, il nous reste donc quelques candidates, dont la luminosité est suffisante pour être considérée comme le continuum, mais on dispose bien d’une liste d’images. La dernière étape consiste donc à calculer la médiane de toutes ces images, pour obtenir un et un seul "continuum synthétique".

Si on reprend les étapes, celà nous donne par exemple:

on dispose de 60 images à des décalages de pixels (entiers)
on retire les 8 qui sont au centre de la raie étudiée, il en reste 52
on calcule la valeur moyenne de chacune de ces images : 5000, 8000, 5200, 6400, …
on calcule la moyenne de toutes ces moyennes, par ex: 6000
on ne retient que les images dont la moyenne est supérieure à cette moyenne, disons qu’il en reste 30
on calcule la médiane de ces 30 images

Il ne nous reste donc plus qu’à faire la soustraction entre l’image au décalage de pixels de la raie Hélium avec le continuum synthétique calculé et on obtient l’image Hélium ci-dessus, en quelques secondes seulement !

Conclusion

Dans ce billet, j’ai décris la façon dont je procède pour améliorer JSol’Ex. En premier lieu, il s’agit d’identifier un besoin particulier, par exemple, ici, produire une image Hélium. A partir de ce besoin, il s’agit de chercher comment simplifier un tel traitement. La simplification ne se fait alors qu’à partir de ce dont je dispose à un instant t. Par exemple, dans un premier temps, il a s’agit d’ajouter un système de scripts, qui, outre le traitement des images Hélium, permet d’exploiter la richesse des données disponibles dans un scan. Les personnes intéressées peuvent en apprendre plus sur les scripts dans ce tutoriel.

Ensuite, il a s’agit d’ajouter des fonctionnalités permettant de simplifier la configuration pour les utilisateurs, en l’occurrence la détection automatique de raie, puis d’exploiter ces nouvelles fonctionnalités pour simplifier encore plus le traitement des images.

Ainsi, il s’agit d’une méthode très itérative, mais toujours dans le but de livrer des incréments fournissant une certaine valeur ajoutée aux utilisateurs. La bonne nouvelle, c’est que j’ai beaucoup d’autres idées en tête !

Ressources

JSol’Ex (téléchargement et documentation)
INTI (logiciel officiel)
Sol’Ex (site officiel Sol’Ex)

JSol’Ex 2.2.0 et amélioration du contraste

Thu, 25 Apr 2024 00:00:00 +0200

JSol’Ex 2.2.0 est tout juste sorti ! Dans cette version, j’ai particulièrement travaillé sur l’amélioration des contrastes. En effet, celles et ceux qui m’ont entendu parler de ce logiciel savent que je râle depuis longtemps sur mon algo d’amélioration de contraste. Dans JSol’Ex, il y a plusieurs algorithmes d’implémentés. Historiquement, le premier était une transformation sinus hyperbolique inverse, qui fonctionne relativement bien pour faire ressortir les zones plus sombres, mais éclaircit trop le reste. L’autre algorithme, utilisé aussi par INTI, c’est la normalisation locale d’histogramme (CLAHE).

Les problèmes avec CLAHE

CLAHE est un algorithme qui améliore sensiblement le contraste. Cependant, je n’ai jamais été très content du résultat. Je ne sais pas comment Valérie Desnoux fait dans INTI, qui a l’air de s’en sortir mieux, mais dans JSol’Ex les résultats obtenus sont assez moyens. En particulier, je note plusieurs défauts :

la création d’artéfacts, liés au fait que cet algorithme travaille sur des "tuiles" d’une taille donnée. Plus la taille de la tuile est élevée, moins l’amélioration de contraste est aggressive, mais plus les artéfacts sont visibles
sur l’image du soleil, les limbes sont toujours éclaircis, ce qui me mène au point suivant
les images solaires paraissent plus "plates" : le relief, en particulier l’obscurcissement lié à la sphéricité du soleil, disparaît. C’est aussi un effet mécanique de l’algorithme lui-même : plus la taille des tuiles est faible, moins on peut utiliser de valeurs distinctes. Ainsi pour une taille de tuile à 16 pixels, on aura au maximum 256 valeurs possibles (pour une image 16 bits avec 65536 valeurs à l’origine). Or, les meilleurs résultats, sur Sol’Ex, sont obtenus avec une taille de tuile de 8 pixels, soit une réduction à 64 valeurs possibles maximum.
il y a trop de paramètres possibles : la taille des tuiles, la dynamique et enfin le "facteur de coupe" pour l’égalisation. Sans entrer dans les détails, trouver une combinaison qui fonctionne bien indépendamment de la taille des images en entrée, de la dynamique de l’image et enfin du type d’image (H-alpha, calcium) est presque impossible.

Pour illustrer mon propos, voici par exemple un disque solaire (le fichier SER est celui disponible sur le site de Christian Buil) traité par JSol’Ex 2.1 (version précédente donc), avec les paramètres par défaut, une tuile de taille 16 pixels (à gauche) ou 64 pixels (à droite) :

On note déja clairement éclaircissement des limbes, et des artéfacts "carrés" qui apparaissent dès que la taille des tuiles augmente. Trouver les bons paramètres qui conviennent à tous étant compliqué, JSol’Ex offrait déja la possibilité de changer ces paramètres (taille de tuile, intensité, …) mais je pense que peu d’utilisateurs le faisaient.

Aussi ais-je décidé d’ajouter un algorithme dans JSol’Ex 2.2 dédié aux images solaires. Pour vous donner un peu envie de lire la suite, voici ce qu’on peut obtenir avec JSol’Ex 2.2 sur la même image :

Nouvel algorithme

Le nouvel algorithme est disponible dans JSol’Ex 2.2 et est l’algorithme utilisé par défaut. Il combine plusieurs techniques, dont une correction du gamma avec masques, CLAHE et un stretching dynamique de l’image, pour produire une image plus digne de ce que Sol’Ex peut faire. Je ne prétends pas que cet algorithme est parfait, il a lui aussi des défauts, mais il semble être un bon compromis.

Afin de vérifier mes assertions, j’ai procédé méthodiquement à une comparaison des images produites avec les paramètres par défaut de INTI, JSol’Ex 2.1 et JSol’Ex 2.2 (la seule modification est l’autocrop activé et un retournement vertical pour que les images soient orientées de la même façon).

Commençons par la vidéo H-alpha de démonstration utilisée par Christian Buil sur son site.

Les images pouvant être difficiles à comparer en taille réduite, je vous invite à faire un clic doit et ouvrir l’image dans un nouvel onglet pour comparer.

On note clairement que sur JSol’Ex 2.1, il y avait certes une amélioration de contraste, mais elle se faisait au prix d’artéfacts et d’une perte de relief. En revanche, JSol’Ex 2.2 offre une image plus nette que celle d’INTI, sans avoir les défauts de CLAHE.

Continuons avec une autre image solaire en H-alpha, avec plus de dynamique :

Là encore on remarque que JSol’Ex 2.1 produisait une image raisonnable mais assez saturée et surtout "plate". La version 2.2.0, quant à elle, offre une meilleure dynamique tout en préservant les détails et cette impression de profondeur.

Nous pouvons aussi comparer les résultats sur une image obtenue avec une plus longue focale et un disque partiel :

Cette fois-ci on note que les 3 logiciels s’en tirent honorablement. Cependant, JSol’Ex 2.1.3 affiche une image moins contrastée que la v2.2.0, alors qu’INTI v6 affiche une image légèrement plus floue.

Malheureusement suite à une fausse manipulation, j’ai perdu la plupart de mes fichiers SER en raie calcium. J’ai cependant pu faire des comparaisons, sur des fichiers loins d’être idéaux. Les résultats sont cependant intéressants :

Cette fois-ci, on constate que JSol’Ex 2.1 s’en sortait plutôt bien. INTI là encore fait un travail remarquable, mais JSol’Ex 2.2 sature trop l’image. C’est un défaut que j’espère arriver à corriger, qui est lié au fait que les images calcium on un histogramme avec une gaussienne à base très large. Il est néanmoins possible d’atténuer la saturation en choisissant un facteur de correction moins fort (par exemple 1.1 au lieu de la valeur par défaut 1.5), ce que je vous encourage à faire pour les images Calcium (conserver le 1.5 pour le h-alpha, baisser pour le calcium) :

Le résultat dépendra cependant beaucoup de l’exposition initiale de votre vidéo. Voici un autre exemple en raie calcium K:

Il est utile de noter que si vous n’êtes pas satisfait du nouvel algorithme, il est tout à fait possible de repasser à CLAHE :

Enfin, il vous est tout à fait possible d’être plus ou moins "aggressif" sur l’amélioration de constraste à effectuer. Ansi, dans les paramètres de traitement, vous pouvez changer le facteur gamma qui permet d’assombrir l’image. Pour l’exemple, si on pousse les curseurs un peu loin (par exemple à 4 dans l’image suivante), vous constaterez que l’image reste exploitable, et surtout que les défauts de CLAHE qui a tendance à aplatir les images est complètement disparu :

Pour aller encore plus loin

Tout d’abord, JSol’Ex offre un langage de script qui permet d’aller bien plus loin dans les traitements, de générer automatiquement des animations, etc. Bien sûr, les améliorations de JSol’Ex 2.2 sont disponibles dans les scripts par l’intermédiaire de 2 nouvelles fonctions : adjust_gamma qui permet de réaliser une simple correction de gamma sur l’image, et auto_constrast qui correspond à la correction décrite dans ce billet de blog.

Enfin, nous n’avons pas encore parlé des fonctionnalités désactivées par défaut, mais qui étaient déja disponibles dans JSol’Ex : la déconvolution et l’amélioration de la netteté. Ces deux options sont activables dans les paramètres de traitement :

Je ne recommande pas nécessairement de cocher la case "aiguiser les détails" si vous avez des images trop bruitées. Cette dernière ligne compare donc la même image, traitée avec INTI v6, JSol’Ex 2.2 (dernière version donc), mais sans déconvolution, avec déconvolution et finalement avec déconvolution et amélioration des détails :

En ce qui me concerne, je trouve la version déconvoluée particulièrement plaisante à l’oeil, et j’active donc systématiquement la déconvolution :

Conclusion

En conclusion, dans ce billet je vous ai présenté le nouvel algorithme de correction de contraste, activé par défaut dans JSol’Ex 2.2. Je pense que les résultats sont assez probants, et que JSol’Ex n’a plus à rougir de INTI en ce qui concerne les images avec amélioration de contraste. Cependant, je rappelle encore une fois que le logiciel officiel est INTI, que c’est le seul validé avec le sérieux de Christian Buil et Valérie Desnoux. Si vous voulez déposer des images sur BASS2000, vous devez utiliser INTI. Par ailleurs, je vous conseille toujours de comparer les résultats, ne considérez pas que JSol’Ex fait un meilleur travail, c’est probablement faux.

Par exemple, il reste des améliorations à faire sur les images calcium. Mais il est cependant tout à fait possible de modifier les paramètres par défaut, voire de changer d’algorithme.

N’oubliez pas non plus que JSo’Ex offre d’autres fonctionnalités comme la colorisation automatique des images, le stacking, la création de mosaïques solaires ou encore un langage de script particulièrement puissant, qui vous permet par exemple de générer des animations automatiquement. Je vais donc conclure ce billet avec un exemple de script qui utilise cette nouvelle amélioration de contraste pour produire une animation qui nous fait plonger dans l’atmosphère solaire en jouant sur le décalage de pixels :

# Décalage de pixels qu'on applique, de -15 à +15, pas de 0.5
images=range(-15,15,.5)
# On calcule une image corrigée de ses transversalliums
corrigee=fix_banding(images,32,40)
# Déconvolution
decon=rl_decon(corrigee)
# On redimensionne
redim=rescale_rel(autocrop2(decon,1.1),.5,.5)
# On applique la correction de contraste décrite dans ce billet
cst=auto_contrast(redim,1.5)

[outputs]
# Enfin on produit l'animation (75ms entre chaque frame)
animation=anim(cst,75)

Your browser does not support the video tag.

Bonus

J’en profite enfin pour partager deux vidéos, à destination des débutants sur Sol’Ex, que j’ai faites récemment :

Ressources

JSol’Ex (téléchargement)
JSol’Ex (documentation)
INTI (logiciel officiel)
Sol’Ex (site officiel Sol’Ex)
CLAHE (algorithme)
Gamma correction (algorithme)

Vers un effondrement civilisationnel ?

Thu, 22 Feb 2024 00:00:00 +0100

Coucher des mots sur le papier, aussi numérique soit-il, est une forme de thérapie : faire sortir ce qui doit sortir, organiser ses idées, prendre le temps de la réflexion, se rassurer parfois. Prédire le futur est bien évidemment impossible, mais il y a des ressentis, des signaux plus ou moins faibles, qui peuvent nous donner une idée de ce qui nous attend. Ce billet n’a aucune prétention scientifique, il s’agit plutôt d’une synthèse des nombreuses sources que j’ai lues ou visionnées sur le sujet au cours des dernières années. Au centre de ce texte, une interrogation : est-on en train d’assister à la fin de notre civilisation ? Après tout, l’Histoire est faite de disparitions de civilisations, pourquoi la nôtre survivrait plus qu’une autre ? En trame de fond, la question qui me préoccupe le plus : a-t-on déjà scellé le sort de nos enfants ? A qui s’adresse ce billet ? A ma famille et mes amis, probablement. À mes enfants, sûrement, pour exprimer mes regrets, mes craintes, mais aussi mon espoir envers eux. Enfin à tous celles et ceux qui devraient le lire mais qui ne le feront jamais, certainement.

La genèse

J’ai une sensibilité à la fois sur les sujets écologiques et scientifiques depuis ma jeunesse. Je me souviens encore des conversations avec mon copain Erwan, au collège. Nous arrivions tôt le matin, bien avant le début des cours, et avions entre autres de longues conversations sur la Terre et comment la sauver. A l’époque, nous ne parlions pas de réchauffement climatique : les sujets tournaient essentiellement autour de la pollution, du trou dans la couche d’ozone, de la disparition des papillons de nuit ou des ours polaires.

Dans ce terreau fertile aux idées dites écologistes, j’étais biberonné aux séries SF, et en particulier par Star Trek, série humaniste qui m’a profondément marqué. Il en est ressorti une double culture, à la fois la prise en compte de la sauvegarde de l’environnement, mais aussi un profond respect pour la science et ce qu’elle permet d’expliquer du monde. J’aime la rigueur de la démarche scientifique, ainsi que la capacité à se remettre en question, en opposition aux points de vue dogmatiques que l’on trouve par exemple dans les religions. Agnostique, j’ai cherché mes réponses dans la science et continue de le faire aujourd’hui, notamment en appliquant le principe du rasoir d’Okham.

J’avais déja sur ce blog abordé le sujet du rationnel et des reproches qu’on m’en a fait. Cette rationalité est aussi celle qui me pousse à écrire et partager mon angoisse : lorsqu’on a un "esprit scientifique" et que l’on a les informations dont on dispose aujourd’hui, il est difficile, voire impossible d’être optimiste. Pire, il est difficile d’imaginer autre chose qu’un effondrement pur et simple de notre société. Cette prise de conscience, ce pessimisme, doit se convertir en énergie, en action.

Précisons cependant que lorsque je parle d’effondrement civilisationnel, je me moque éperdument des fantasmes de l’extrême-droite sur le grand remplacement (culturel), de leur racisme, de leur xénophobie : je suis à l’opposé de tout cela, pour moi la "tradition chrétienne de la France" n’est qu’une vaste blague à l’échelle de l’Humanité et je suis de ceux qui pensent que l’égalité des droits n’est pas négociable, quelle que soit son origine, son orientation sexuelle ou politique.

Non, je parle ici d’un effondrement de notre mode de vie basé sur la consommation, la propriété privée et la mondialisation : je parle de la disparition possible de la démocratie et du retour de la guerre et des horreurs qui vont avec.

De quoi parle-t-on ?

Une fois que vous savez ce qui se passe au niveau climatique, il n’y a que trois options possibles : le renoncement, le cynisme ou l’action. Pour mes enfants, j’ai choisi d’agir : c’est notamment pour cela que je me suis lancé en politique, certes à un niveau local puisque limité aux municipales, il y a quelques années. C’est aussi pour celà que je communique beaucoup sur les sujets écologique et politique, quitte à en agacer quelques uns.

Une des premières choses dont il faut prendre conscience, c’est l’urgence climatique. Je discute souvent du réchauffement avec des personnes sensibilisées au sujet, mais paradoxalement, rares sont ceux qui ont vraiment conscience de l’urgence de la situation. Au contraire, on entend dire que les écolos sont pénibles, le RN va même jusqu’à affirmer que le GIEC exagère. Rappelons que le GIEC est un organe international qui synthétise les conclusions de centaines d’études et que de part la nature extrêmement politique de ses rapports, c’est précisément plutôt l’inverse qui se produit, avec des rapports qui proposent différents scénarios, les politiques préférant regarder les plus optimistes.

Récemment, Jean-Marc Jancovici, spécialiste mondial du sujet, inventeur du concept de "bilan carbone" était auditionné au Sénat devant des élus parfois médusés par ses réponses.

1.5 degrés, c’est mort. 2 degrés, sauf chute de comète ou effondrement économique, c’est parti pour être mort.

— Jean-Marc Jancovici
Audition au Sénat du 12 Février 2024

Pourtant, aucune des affirmations de Jancovici, dont je me sens proche idéologiquement au moins sur ces sujets, ne devraient être surprenantes : on le sait depuis longtemps, les rapports s’enchaînent, mais l’inaction persiste.

Une grosse partie du problème est qu’il est difficile pour un être humain, de quantifier par nos simples sens une augmentation de 2 degrés. Aujourd’hui encore, une canicule est systématiquement illustrée par des vacanciers à la plage, là où on devrait montrer des ruisseaux asséchés, des forêts qui partent en fumée, des personnes âgées en détresse, des agriculteurs qui ne peuvent plus arroser leurs cultures.

Il est important de comprendre que 2 degrés, c’est une moyenne mondiale : le réchauffement est plus important dans certaines zones de la planète (les pôles par exemple) et moins dans d’autres. Une différence de 2 degrés à l’échelle planétaire, c’est 3 ou 4 degrés en plus en France (en 2022, nous avons atteint +2.7⁰) Entre -1.8 et -18 degrés, c’est précisément ce qui séparait l’ère glacière de l’ère pré-industrielle. A l’époque, nous avions des glaciers de plusieurs centaines de mètres de haut, l’Angleterre était reliée au continent (les océans étaient 120 mètres plus bas). Si une si petite différence de température pouvait provoquer une telle conséquence, imaginez ce que 2 ou 3 degrés en plus pourraient donner. Précisément, on a du mal à s’imaginer, et c’est pour celà que le GIEC établit des scénarios, des prédictions. Ainsi, il est plutôt rassurant que la France se prépare à un scénario à +4⁰.

Pourtant, lorsque la France est touchée par une "vague de froid" (je le mets entre guillemets parce que ces vagues sont moins fréquentes et moins fortes qu’il y a 20 ans), nous retrouvons ces réflexes climatosceptiques, en confondant météo (temps court) et climat (temps long).

Aujourd’hui nous sommes, mondialement, au niveau de la barre des 1.5 degrés et les conséquences au niveau local se font déja sentir :

en Espagne, des sécheresses à répétition ont annihilé les capacités agricoles du pays. Le verger de l’Europe va disparaître. Le sud de la France commence à souffrir des mêmes déréglements.
en Catalogne, cette même sécheresse menace l’approvisionnement en eau potable de millions d’habitants
en France, des événements climatiques de plus en plus extrêmes menacent populations et industries

A 1.5 degrés, nous avons déja scellé le destin de pays entiers. Les îles Tuvalu, par exemple, vont disparaître. Ce pays est le premier du monde à avoir signé un accord d’asile climatique avec l’Australie. Imaginez-vous votre pays entier disparaître ? Ceci nous semble lointain, pourtant c’est la réalité que sont en train de vivre des millions de personnes dans le monde : leur existence même est menacée. Que feriez-vous si vous ne pouviez, physiquement, plus vivre en France ? Laissez-moi penser que les obsessions migratoires de politiciens et autres consultants du bar du coin sont d’un tel ridicule comparé à ce qui nous attend…

Nous sommes en avance sur les 1.5 degrés prévus par le GIEC: pour la première fois, les +1.5⁰ ont été dépassés pendant 12 mois consécutifs.

A 2 degrés, nous savons déjà que la sécurité alimentaire ne sera plus assurée, que des conflits liés aux migrations climatiques vont éclater, pour l’accès aux ressources et pour justement empêcher les migrations de personnes qui cherchent simplement à survivre. Fantasme ? Regardez par exemple ce qui se passe entre l’Iran et l’Afghanistan pour l’accès à l’eau potable.

A 2.5 degrés, certaines régions du monde, en particulier en zone tropicale, deviennent inhabitables. Les rassuristes avancent que l’homme a toujours su s’adapter, mais il existe cependant des limites physiologiques, la réalité scientifique s’impose à tous. Au-delà d’un certain seuil, la combinaison de la température et de l’humidité ambiante rend inefficace le mécanisme de transpiration. Nous mourrons, il ne s’agit pas d’une opinion, mais d’une réalité devenue tristement célèbre l’été dernier lors d’un concert de Taylor Swift.

De la difficulté des ordres de grandeur

Un des problèmes majeurs du réchauffement climatique est qu’il est, même en étant sensible au sujet, difficile de se rendre compte des ordres de grandeur. J’étais par exemple déjà sensibilisé au sujet, mais, malgré tout, je continuais à prendre l’avion en particulier pour mon travail. Je savais que l’avion était mauvais pour le climat, mais je n’avais aucune idée d’à quel point : on sait que c’est mauvais, mais rares sont ceux qui nous donnent les ordres de grandeur. On se retrouve ainsi avec des politiques qui nous incitent à éteindre nos appareils en veille mais nous proposent de faire un aller-retour à Rome pour 100€ ou moins par avion ! Des non-sens écologiques, mais pas économiques : un aller-retour Paris-Dubaï par exemple, c’est l’équivalent de votre budget carbone annuel selon les accords de Paris ! Des outils existent désormais pour évaluer votre impact et des activistes comme Bon Pote font un excellent travail pour décrire les problèmes inhérents à l’industrie aéronautique.

Habitué des conférences à l’international pour mon travail, ma prise de conscience sur ce sujet particulier fut aussi tardive que brutale. J’en ai ai encore honte. J’ai cependant pris la décision, voici un peu plus de 4 ans, de ne plus prendre l’avion et de n’assister qu’aux conférences auxquelles je peux me rendre par train. C’est pourtant un vœux pieux, puisque je reste un être humain : que ferais-je lorsque mon entreprise me donnera le choix entre prendre la porte ou donner une conférence à San Francisco, ou une réunion de travail au Maroc ? Ce problème est aussi sujet de tensions familiales, entre les vœux légitimes de "voyager", d’explorer le monde et ses merveilles, et mon obstination à dire non. Combien de temps résisterais-je à cette pression ? Une bonne partie de ma carrière s’est construite sur le fait de rencontrer des experts de mon domaine à l’étranger, mais aussi à partager mes connaissances : je suis le fruit du réchauffement climatique. Quelle légitimité ai-je à vous "faire la morale" et vous demander, à vous, d’y renoncer ? Qui suis-je pour refuser ce que j’ai moi-même fait ?

Je n’ai pas de bonne réponse à cette question, si ce n’est de faire appel à votre sensibilité. D’autre part, je ne serai pas de ceux qui vous jugent pour vos choix : je suis le premier plein de contradictions et nous faisons, chacun, nos choix en fonction de ce que nous savons à un instant T, de nos contraintes, et finalement de nos convictions. Pour certaines décisions, nous avons le choix et certaines décisions seront plus faciles à prendre pour vous que pour moi, et inversement !

Il est bien plus facile pour un citadin de se séparer de son SUV que pour quelqu’un qui n’a pas accès aux transports au commun en zone rurale. Inversement, manger local et faire travailler producteurs locaux est plus simple pour un rural qu’un citadin. Chacun de nous, à notre échelle, faisons des choix qui nous semblent opportuns compte-tenu de nos situations personnelles.

Prenons donc ces ordres de grandeur en exemple. Pour respecter les accords de Paris et limiter le réchauffement climatique à 2 degrés, notre empreinte carbone, par personne, doit tomber à 2 tonnes par an. A ce jour, les estimations varient, mais l’empreinte carbone d’un français est de l’ordre de 10 tonnes par an.

En France, 31% des émissions sont liées au transport, donc à la voiture individuelle et aux transports de marchandises, c’est dire si le sujet de la voiture est important. Pourtant, que constate-t-on ? L’automobile est le 2d plus gros contributeur à la publicité en France, avec environ 1500€ de budget communication par voiture. Notre Président aime la bagnole et envoie un message désastreux. Bruno Le Maire, devant trouver 10 milliards d’économies, choisit de le faire sur l’environnement et les mobilités. En clair, on nous vend des voitures au lieu de nous vendre des vélos électriques, dont l’empreinte carbone est 20 fois inférieure.

Ce site montre l’évolution de l’empreinte carbone par tête et par pays, depuis le début de l’ère industrielle. Il y a 2 choses importantes à comprendre :

l’empreinte carbone est directement corrélée au niveau de vie des habitants. Plus on est riche, plus on consomme. Plus on consomme, plus notre empreinte est forte. L’empreinte carbone est 10 à 50 fois plus forte dans les pays industrialisés qu’en Afrique. Ainsi, il est complètement faux d’affirmer, comme Nicolas Sarkozy, que la crise démographique est responsable de la crise climatique. Ça n’est pas le nombre de personnes qui compte, mais leur capacité à consommer. Diminuer la population serait donc une solution… à condition de le faire dans les pays développés !
l’empreinte carbone est essentiellement impactée par la consommation d’énergies fossiles (pétrole et charbon). C’est ce qui explique que l’empreinte carbone d’un français est plus faible que celle d’un allemand ou d’un polonais : là où ils utilisent des centrales à charbon pour se chauffer et faire tourner leurs industries, nous avons des centrales nucléaires.

Sur ce sujet du nucléaire, soyons clairs : je suis écolo et pour. Lorsqu’on connaît ces ordres de grandeurs, lorsqu’on sait qu’environ 240 000 personnes meurent en Europe tous les ans à cause de la pollution atmosphérique principalement liée à la combustion de charbon, la position écologique anti-nucléaire traditionnelle est difficilement tenable. Cette position anti-nucléaire, je l’avais de part de mes lectures lorsque j’étais gamin (Pif’Gadget, Greenpeace, des positions essentiellement liées à la peur du nucléaire, des accidents et de leurs terribles conséquences. Ça, c’était avant d’avoir pris le temps d’étudier le sujet. J’ai depuis largement révisé ma position et suis convaincu qu’on ne pourra pas s’en sortir sans nucléaire. Il d’ailleurs assez ironique de constater que la biodiversité a fortement augmenté à Tchernobyl, depuis que la zone n’est plus occupée, de quoi être optimiste sur la résilience des écosystèmes si nous devions disparaître !

Ceci ne veut pas dire qu’on ne peut pas développer les énergies renouvelables, bien au contraire, mais il faut encore une fois avoir les ordres de grandeur en tête. Vous trouverez des chiffres différents selon les estimations plus ou moins pessimistes (notamment le facteur de charge), mais il faut comprendre que pour remplacer 1 seul réacteur nucléaire, il faut environ 1000 éoliennes ou 55 km² de panneaux solaires. Nous avons 56 réacteurs en France… Sans réduire notre consommation électrique, ceci signifie que nous devons exacerber encore plus la concurrence pour l’occupation des sols : doit-on s’en servir pour produire de la nourriture, de l’énergie, du logement ou la laisser libre pour la biodiversité ?

Est-il dès lors si surprenant que l’Allemagne ait dù raser des villages entiers pour exploiter le charbon nécessaire à la remise en marche de ses centrales ? Une bombe climatique ! Doit-on ensuite s’étonner que ces exemples soient repris par les réactionnaires pour décrédibiliser toute action de protection de l’environnement ?

Voilà le noeud du problème : notre société moderne est bâtie sur la consommation massive d’énergie, en particulier du pétrole, et à moins de réduire drastiquement cette consommation, nous ne pourrons tenir nos engagements. A l’heure actuelle, il faudrait réduire de 5% par an notre consommation d’énergies fossiles pour y arriver. Cette consommation étant directement corrélée à la sacro-sainte croissance, il s’agit ni plus ni moins que d’avoir l’équivalent d’un COVID, tous les ans ! Nos politiques ont déja renoncé, personne n’étant prêt aux sacrifices que celà implique.

Note

On m’a posé la question de la chaîne d’approvisionnement d’uranium, qui peut être problématique. C’est vrai, mais uniquement si nous restons à cette génération de réacteurs. La 4ème génération, les surgénérateurs sont capables d’utiliser les produits dérivés de la fission et nous donnent suffisamment de ressources pour des siècles de consommation. Pourtant, ce sont tous les expérimentateurs qui ont été mis à l’arrêt (Superphénix ou Astrid). Un gâchis dénoncé par Yves Bréchet lors de son audition au Sénat.

De sacrifices, pourtant, c’est de cela qu’il s’agit. Sacrifier nos générations futures, ou sacrifier notre mode de vie. Nous vivons toujours selon le mythe d’une croissance infinie : toute notre société est bâtie sur ce seul concept économique qui n’a aucun sens physique. La réalité physique des choses est pourtant implacable : il n’existe pas de croissance infinie dans un monde fini. Là encore, il est indispensable de parler d’ordres de grandeur : une croissance à 2% signifie un doublement de la valeur tous les 36 ans. Qui dit valeur dit production, dit consommation d’énergie, dit impact sur l’environnement.

Dit autrement, la croissance est une courbe exponentielle : il s’agit du genre de courbes que l’on ne souhaite pas voir.

N’est-il pas inquiétant de constater que dans ce graphique, des crises majeures comme les chocs pétroliers des années 70 ou le COVID n’ont eu que peu ou pas d’influence sur notre consommation ?

Une autre visualisation de la notion d’exponentielle, c’est celle des anomalies de température, produite par la NASA. Une animation que je trouve particulièrement efficace pour comprendre l’effet d’emballement :

Your browser does not support the video tag.

Cette exponentielle explique pourquoi le développement des énergies renouvelables ne s’est pas faite au détriment des énergies fossiles : elle s’est faite essentiellement en complément, parce que nous avons toujours besoin de plus d’énergie pour maintenir cette croissance.

Ainsi, si la consommation de charbon dans la production énergétique mondiale baisse en pourcentage, le volume n’a jamais été aussi élevé ! Une mauvaise nouvelle à cependant relativiser, puisque la production de charbon baisse : en effet, dans des pays tels que l’Allemagne, les renouvelables sont destinés à remplacer les centrales à charbon.

Le problème des courbes comme celles-ci est que les prévisions deviennent presque impossible à formuler, on entre dans le domaine de l’inconnu, et les esprits scientifiques comme moi n’aiment pas cela. Serge Zaka, docteur en agroclimatologie, décrit des phénomènes de réchauffement des océans statistiquement impossibles (autrement dit, les marges dépassent ce que la statistique classique considère comme possible).

Cette question de la croissance, pourtant, est au cœur de notre survie. Attention, je ne parle pas de la survie de l’espèce, ni même de "sauver la planète". En effet, même en étant pessimiste, je pense que l’espèce humaine survivra. En revanche, ne comptez pas sur moi pour vous dire combien survivront : mon intuition, compte-tenu des paramètres dont nous parlons ici, est que nous risquons de voir une nouvelle crise majeure. Et quand bien même l’espèce humaine ne survivrait pas, la vie, elle, continuera sans nous : elle existait avant, elle existera après.

C’est l’histoire d’un skieur qui se répétait "jusqu’ici tout va bien"

Pour prendre une analogie, nous sommes un peu dans la situation d’un skieur du dimanche : il descend sa piste, il à se sent à l’aise. Il accélère, prend plaisir, jusqu’à se rendre compte qu’il a pris trop de vitesse, que la pente est bien trop raide et qu’il ne peut plus maîtriser sa course. En bas, un croisement, d’autres skieurs, il est trop tard pour s’arrêter et il n’y a que 2 solutions :

virer, se laisser tomber, quitte à se casser une jambe, pour éviter le groupe
continuer et entrer en collision, en entraînant de nombreuses autres personnes dans la chute et potentiellement de nombreuses victimes

M’est avis que beaucoup de personnes choisissent instinctivement la 2ème solution, parce que la première n’est pas confortable et que notre instinct de préservation nous joue des tours : incapable d’anticiper des conséquences encore pires à long terme, notre espèce se mure dans le déni et choisit la solution la moins logique. Pire, notre tendance naturelle est, souvent paradoxalement de bonne foi, de déresponsabiliser. Les exemples sont nombreux : mettre des radars automatiques au lieu d’apprendre à respecter les limites, demander aux enfants de mettre des gilets jaunes au lieu de sécuriser les routes, dire aux filles de se couvrir au lieu de demander aux garçons de cesser de les voir comme des objets sexuels, demander d’éteindre vos box internet au lieu de prendre votre vélo pour aller chercher du pain…

Rassurisme et techno-solutionnisme

Maintenant que le réchauffement climatique est palpable et subi par les plus sceptiques d’entre nous, un nouveau mal est en marche : le rassurisme. Se substituant au climato-scepticisme, il consiste à affirmer que nous nous en sortirons toujours, que l’homme s’est toujours adapté, que nous trouverons des solutions techniques, et cætera, et cætera. Le rassurisme est une plaie parce qu'il incite à l’inaction. Agréable discours à entendre, il est le parfait compagnon du status quo, sous couvert de discours "de bon sens", mais aussi un risque encore plus grand pour l’avenir. Je classe dans le rassurisme le discours techno-solutioniste que nous entendons de plus en plus.

Prenons par exemple l’idée souvent commentée de la séquestration du carbone : pourquoi ne pourrions-nous pas l’extraire de l’atmosphère ? A priori, l’idée n’est pas idiote, si on peut le faire il serait idiot de s’en passer. Pourtant, il suffit d’un tout petit peu de recherche pour se rendre compte des problèmes. Lorsqu’on parle de concentrations de CO², nous parlons de parties par million. Par exemple, 200ppm, ce sont 200 molécules de CO² sur 1 million : c’est une concentration extrêmement faible, mais qui a des effets dévastateurs. Si vous avez quelques souvenirs de physique, je vous conseille d’ailleurs cette excellente vidéo de Science Etonnante sur les mécanismes du réchauffement et le mythe de la saturation du réchauffement. Si tant est que nous disposions d’une technologie de "nettoyage", il faudrait brasser une quantité d’air phénoménale pour l’extraire et pour brasser cet air, il faudrait une quantité non négligeable d’énergie… En clair, il s’agit donc souvent plus de "coups de com'" d’entreprises cherchant avant tout à faire de l’argent sur le dos du climat, peu recommandables… mais pour lesquels des investisseurs peuvent se laisser séduire.

Un autre exemple, c’est la la colonisation de Mars, annoncée par Elon Musk. Peut-on être sérieux seulement 5 minutes ? Il n’est pas surprenant que les seuls qui y croient soient des économistes, c’est avant tout le modèle de la fuite en avant : puisqu’on ne peut pas sauver notre planète, rendons-en une autre vivable ! L’idée est d’autant plus ridicule qu’on ne sait même pas contrôler le climat ici, en premier lieu lutter contre le réchauffement climatique, alors que dire lorsque l’on parle de changer le climat d’une planète dont on ne sait presque rien… C’est une chose que de vouloir fouler Mars du pied, s’en est une autre que de la terraformer.

Mais un des problèmes de la croissance et de l’évolution de la technologie et de l’information, c’est qu’elle rend possible des actions individuelles aux conséquences potentiellement catastrophiques, au premier rang desquelles la géoingénierie. Ainsi, un article particulièrement inquiétant montre que nous disposons de moyens de diffuser économiquement du sulfure dans l’atmosphère, pour refroidir l’atmosphère'. Même avec des conséquences terrifiantes, telles que des pluies acides, ou le simple fait qu’arrêter d’en diffuser entraînerait un réchauffement massif encore plus grand, le fait est que ça pourrait fonctionner en pratique. Nous sommes donc en plein dans ce que j’expliquais plus haut : plutôt que de résoudre le problème à sa source, on continue, quitte à avoir des conséquences bien pires plus tard. Lorsqu’une technologie est disponible, elle est utilisée. La question n’est donc pas de savoir SI, mais QUAND elle sera utilisée, soit par un riche milliardaire qui souhaite continuer à faire du profit en dépit du bon sens, ou d’un Etat qui lutte pour sa survie. Si vous vous rappelez de l’horloge de l’apocalypse, il me semble qu’on se rapproche dangereusement de minuit…

L’autre effet pervers du techno-solutionnisme, c’est que si tant est qu’il fonctionne, il n’incite pas à la sobriété. Ainsi, toutes les évolutions technologiques qui ont permis de faire des économies n’ont in fine pas eu pour conséquence de réduire la consommation. Par exemple, les moteurs thermiques sont aujourd’hui beaucoup plus efficaces qu’avant, mais les économies ont servi à augmenter l’autonomie ou à avoir des voitures plus grosses : c’est l’effet rebond.

Ainsi, toutes ces évolutions technologiques ne sont en pratique là que pour supporter un modèle de consommation constant. Un autre exemple ? Le rendement à l’hectare de la production agricole a été multiplié 3 depuis 1960, par plus de 10 depuis le début de l’ère industrielle :

On se rend bien compte du mensonge qui consiste à dire qu’il faut bien continuer ce modèle pour nourrir la planète : nous pourrions utiliser les gains de productivité à l’hectare pour nourrir plus de monde mais nous avons choisi de les investir dans des biocarburants pour faire rouler des voitures, ou dans de la production industrielle de produits transformés qui menacent notre santé. Pour autant, les revenus des agriculteurs ne cessent de chuter, les famines n’ont pas disparu (essentiellement pour des questions de logistique) et les maladies cardiovasculaires explosent. Nous produisons largement plus que ce que dont nous avons besoin pour survivre, mais nous "produisons de la croissance" en transformant les produits. Là où au début du siècle, l’essentiel de la consommation se faisait du produit brut au consommateur, désormais l’essentiel se fait par des produits transformés. L’abondance est illustrée par ce graphique représentant l’apport calorique par habitant :

A celles et ceux qui répondront que plus d’apport calorique c’est une meilleure santé, rappelons que nous avons besoin d’entre 2000 et 2500 calories par jour, un seuil qui a été franchi au début des années 1820. Depuis, nous sommes bien au-delà, ce qui explique notamment l’explosion de l’obésité et des maladies cardiovasculaires (en combinaison avec la sédentarité permise par l’exploitation des machines).

Ainsi, la croissance n’est pas nécessairement synonyme de progrès : au delà d’un certain seuil, elle devient maladive et entraîne plus de maux que de bien.

Les prémisses de cette constatation ne datent pas d’hier, le club de Rome s’en faisait écho il y a 50 ans déja. De nos jours, rares encore sont les économistes qui défendent la décroissance. En France, des chercheurs comme Timothée Parrique montrent avec brio que la décroissance ne peut plus être considérée comme un gros mot. Au contraire, elle devient nécessaire, comme le laisse entendre ce titre "ralentir ou périr".

Il ne faut pas non plus confondre le techno-solutionnisme avec l’utilisation des techniques permettant de limiter l’impact de notre consommation. Certains outils seront indispensables, mais si nous devons répondre à une solution d’urgence, il est préférable de le faire avec les technologies dont on dispose, pas de celles dont on ne sait pas si elles seront disponibles dans 10 ans.

Pourquoi un effondrement ?

Nous l’avons vu, la conjoncture n’est pas favorable, loin de là. Nous savons, nos gouvernements savent, mais rien ne change. N’était-ce pas Emmanuel Macron qui nous a promis que son quinquennat "sera écologique ou ne sera pas" ? Nous avons la réponse : après le une convention citoyenne sur le Climat vidée de sa substance, après le sacrifice de l’écologie au profit des agroindustriels qui nous confortent dans ce modèle, il n’y a a que du cynisme dans les décisions politiques. Même au niveau local, dans ma commune, la majorité se gausse à coups de croissance verte, un concept qui ne parle qu’aux économistes et qui n’a jamais démontré le moindre succès. Encore une fois, il n’y a rien de surprenant : un niveau de base en mathématiques ou de physique suffit à comprendre qu’on ne peut faire de croissance sans sacrifier de ressources, ce qui se traduit soit par de la pollution, soit par du réchauffement climatique. Dans ma commune, on construit encore des supermarchés en périphérie comme dans les années 70 et on moque les écologistes "décroissants", "contre l’emploi" et "pour l’insécurité". Qu’importe l’état catastrophique des cours d’eau, que l’eau doivent être importée de Loire-Atlantique pour subvenir aux besoins d’industries agroalimentaires locales bien connues, puisqu’on a la croissance ! Qu’importe que l’on doive construire sur des terres agricoles pour loger tous les néo-ruraux attirés par la croissance économique du territoire…

Nous constatons là que malgré toutes les informations disponibles, nous sommes dans la situation du skieur qui ne peut plus s’arrêter; il y aura des dégâts !

Cependant, pas de quoi prophétiser un effondrement civilisationnel, me direz-vous. Certes, mais il y a plus : c’est la conjonction de facteurs qui peut entraîner notre chute.

Nous avons mentionné que dans le dérèglement climatique, l’essentiel du problème était concentré autour des énergies fossiles. Cependant, d’autres problèmes d’ampleur sont eux aussi liés à notre modèle de développement : l’effondrement de la biodiversité par exemple. Nous sommes entrés dans une nouvelle phase d’extinction de masse, tellement bien documentée qu’elle porte un nom: l’extinction de l’Holocène :

En 50 ans, plus de la moitié du monde sauvage a disparu. Je ne sais pas si vous vous rendez bien compte : l’homme est présent sur Terre depuis plus de 4 millions d’années, Homo Sapiens depuis 300000 ans environ, et en seulement 50 ans, nous avons détruit plus de la moitié des espèces. Pour la biodiversité marine, la situation est encore pire, avec la surpêche, l’augmentation de la température et l’acidification des océans. Depuis plus de 6 mois, nous vivons une véritable canicule marine, mais qui en a entendu parler ?

Or, l’homme fait partie d’un écosystème : nous vantons notre adaptabilité, mais nous sommes les premiers dépendants de notre environnement. Le détruire, c’est directement menacer notre survie : il faut être fou pour croire que l’homme peut survivre seul.

Devant l’opposition à la décroissance, ou même simplement les appels à la sobriété, les discours les plus réactionnaires sont en marche. Il n’y a pas de quoi être optimiste, lorsqu’en France, un ministre traite des militants écologistes d’éco-terroristes, mais en même temps, soutient des agro-industriels qui mettent le feu à des mutuelles… Là encore on pourrait croire à de l’ignorance, mais il s’agit d’un cynisme sans nom : tous savent pertinemment ce qui se profile, mais aucun n’a le courage politique pour faire ce qui est vraiment nécessaire : un nouveau modèle de société basé sur la sobriété. Que dire d’un pays où l’on s’émeut plus facilement d’une boîte de soupe versée sur un tableau protégé derrière une vitre en verre, que de plusieurs centaines de morts en Inde suite à la canicule… Pas vraiment de quoi être optimiste.

On pourrait s’arrêter là mais d’autres signaux sont tout aussi inquiétants. Je rappelais par exemple à quel point notre société moderne est basée sur les énergies fossiles. Le graphique ci-dessous montre par exemple la répartition de la consommation énergétique par filière :

La baisse en 2020 est liée au COVID, mais ne nous y trompons pas : à l’échelle mondiale, les énergies fossiles dominent : les renouvelables représentent une part croissante de la production, mais ne sont comparables qu’au parc nucléaire, les énergies fossiles sont largement dominantes. Dans ce portrait, la situation du pétrole est plus critique : nous savons que les réserves s’épuisent. Que l’on décide de s’en passer volontairement ou non, nous arriverons avant la fin du siècle à la fin du pétrole.

Or, dans de nombreux domaines, nous sommes complètement dépendants du pétrole, non pas en tant que source d’énergie, mais de matière première :

pharmacologie
agriculture (machines, mais aussi engrais)
santé
textiles
cosmétiques
…

Il ne vous aura pas échappé que l’Europe ne dispose pas ou peu de cette ressource. Préférons-nous continuer à brûler ce qui nous reste pour voyager pour 500€ à l’autre bout du monde, ou pour produire notre nourriture et fabriquer nos médicaments ? Mon choix est vite fait…

A court ou moyen terme, la dépendance de l’Europe au pétrole signifiera un asservissement aux pays qui en disposent (si tant est qu’ils soient disposés à nous en vendre). Sachant que ces pays ne sont pas ce qu’il y a de plus démocratique, la question du respect des droits humains pourrait à moyen terme devenir un vague souvenir. Dès lors quels choix s’offriront à nous ? Entrer en guerre pour leur "voler" ces ressources ? Les forcer à nous en vendre ? A quel prix ? Comment ?

Le remplacement du pétrole en tant que source d’énergie est possible mais requiert une électrification massive de nos moyens de subsistance (transports, machines outils, industries, …) et une relocalisation de la production, le transport maritime dépendant essentiellement de cette ressource. Or, qui dit électrification dit augmentation de la production. De quels moyens disposons nous pour remplacer une telle quantité d’énergie ? L’écologie n’est jamais aussi mauvaise que lorsqu’elle ignore la réalité physique des choses.

Les pays qui disposent de ces ressources, donc, ne sont pas particulièrement amicaux. Nous parlons de pays dont les actions récentes n’augurent pas d’un avenir radieux pour l’Europe, mais aussi pour leurs propres populations : la Russie, la Chine, … La Russie, qui ne cesse d’étendre son influence sur un continent Africain avide de "revanche" sur la colonisation et l’exploitation de leurs ressources par l’Occident. Certes la Russie n’est pas animée par la bonté de réparer les erreurs de l’Occident, elle souhaite tout autant s’accaparer les ressources minières du continent et par là donc disposer de moyens de pression… pour gagner sa "guerre civilisationnelle", mais le message est passé, la France doit s’en aller !

Cette guerre civilisationnelle, des gens comme moi y participent malgré eux. Ceux qui me connaissent savent à quel point, notamment, je lutte contre mon envie de quitter mon métier. A vrai dire, si je n’avais ni famille, ni crédits sur le dos (comme tout le monde), je crois que j’aurais déjà abandonné ce qui est pourtant une de mes passions.

En effet, en tant que développeur, non seulement je contribue massivement au réchauffement climatique (le numérique représente une part de la consommation d’énergie mondiale en explosion, notamment à cause de l’explosion du nombre de terminaux), mais j’ai aussi donné des outils de manipulation de masse, utilisés comme tels, par des puissances qui cherchent à nous déstabiliser. Les réseaux sociaux, notamment, sont devenus de véritables poubelles où les idées complotistes, antivax, climatosceptiques sont promues bien plus que les autres. Les idées dites "de droite" sont favorisées par les algorithmes de Twitter/X et les manipulations de la Russie déstabilisent nos démocraties. Savoir que ce que je développe sert à l’effondrement de la société et à la propagation des idées les plus nauséabondes me rend malade.

Force est de constater que le populisme monte en flèche : il a gagné au Brésil (Bolsonaro), aux Etats-Unis (Trump), au Royaume-Uni (Jonhson), en Hongrie (Orban), aux Pays-Bas (Wilders), en Argentine (Milei). Il faut être sacrément optimiste pour croire que la France puisse miraculeusement échapper à ce fléau. Pour autant, une fois en place, nous savons ce que ces gouvernements pensent de l’économie et donc de l’écologie. Pire, nous savons ce que ces personnes pensent du droit des femmes et plus largement de toute personne ne pensant pas comme eux. Je vous invite d’ailleurs à écouter cette interview de Véra Nikolski et Jancovici sur la fin de l’ère du pétrole et ses conséquences sur la démocratie et le droit des femmes.

Les idées qui sont promues dans nos sociétés modernes sont incompatibles avec nos objectifs climatiques : nous devons tous travailler, devons être encore et toujours plus productifs. Les outils que nous concevons, en informatique, font gagner de la productivité, mais cette productivité n’est pas rendue au travailleur pour du temps libre, elle est réinvestie en plus de croissance.

Produire, consommer, produire, consommer, produire… "pouvoir d’achat" et zéro chômage érigés en totems.

Ce que je constate donc tous les jours est une fuite en avant, doublée d’un déni, mais pire encore, une réaction exactement inverse à ce qu’il faudrait faire pour maintenir notre consommation sous les limites planétaires. Dès lors, la question de l’effondrement civilisationnel se pose : si tant est que nous prenions le virage, qu’en est-il des pays qui ne le feront pas et auront la capacité à nous menacer, précisément parce qu’ils auront fait le choix inverse ? C’est un dilemme auquel je n’ai pas de réponse, mais d’autres questions ne sont pas agréables à entendre :

Que ferons-nous lorsque les populations qui ne pourront plus vivre dans leur pays frapperont à notre porte ? Que ferons-nous lorsque nous ne pourrons plus importer nos médicaments, faute de moyens de transport longue distance ? Que ferons-nous lorsque l’Afrique nous demandera des comptes pour l’exploitation de ses ressources ? Que ferons-nous si nous n’avons plus de pétrole pour faire rouler nos tanks, voler nos avions et nous défendre contre des pays en quête de conquêtes territoriale, économique ou culturelle ? Que ferons-nous lorsque nous aurons tellement détruit nos services publics (transports, écoles, hôpitaux, production électrique) que nous serons dépendants de services commerciaux dont la seule survie ne dépend que de leurs marges ?

La mondialisation nous a paradoxalement mis dans une situation extrêmement précaire, l’Europe d’aujourd’hui est incroyablement fragile. Tout ça au nom de la sacro-sainte croissance, un concept qui rappelons-le, n’a aucun sens physique, c’est une pure construction mathématique permettant d’évaluer l’activité économique d’un pays.

Or, l’activité économique ne se mesure pas qu’à l’aube de ce qui s’achète. C’est pourtant ce que nous faisons tous les jours. Ainsi, le comptable qui travaille bénévolement dans une association humanitaire fait le même travail que le comptable qui travaille dans une entreprise d’extraction de minerais. L’un ne contribue pas au PIB, l’autre oui. Leurs contributions au bien-être de l’humanité sont elles pour autant comparables ?

Se passer de la croissance n’est pas sans poser des problèmes, tant elle est au cœur de notre société. Sans croissance, point de retraites, tout un monde de solidarité à réinventer ! Quelle entreprise accepterait volontairement de ne pas croître, alors que cela menacerait directement sa survie face aux concurrents qui, eux, prendront la décision de continuer ?

Pourtant, décroître l’industrie automobile, responsable d’une grande partie du réchauffement, de morts sur la route, de la pollution aux particules fines, ne serait que bénéfique pour notre société. Nous ne parlons pas de la supprimer du jour au lendemain, mais de planifier notre sortie. De même, l’opulence de agro-industrie devra disparaître, c’est une question de survie, au profit d’une agriculture raisonnée : produisons moins, mais de meilleure qualité, avec des revenus décents et moins de transformations. A court terme, achetons des Fairphone plutôt que des Vision Pro, achetons des produits réparables plutôt que des produits pas chers mais écologiquement aberrants. Les leviers sont nombreux, il faut "simplement" une volonté politique. Réapprenons à mutualiser, nous pouvons inventer de nouveaux modèles basés sur la coopération.

Plus nous attendons, moins les mesures que nous devrons prendre seront socialement acceptables et plus le risque de révolution sera important. Il n’est pas surprenant qu’on utilise l’expression d’écologie punitive, dans ce contexte, alors que le plus punitif, ce sont les conséquences directes de la surexploitation de notre environnement : inondations, sécheresses, maladies, incendies, … La punition est vécue de plein fouet par les agriculteurs qui perdent leurs récoltes, des entrepreneurs qui voient leur camping partir en fumée, des habitants qui voient leur habitation détruite par des inondations ou des tornades…

En conclusion, tout semble converger vers l’idée d’un crash massif à venir : c’est la combinaison du réchauffement climatique, de l’effondrement de la biodiversité, du mythe de la croissance infinie, de la montée du populisme et de la raréfaction des ressources fossiles qui rend cet avenir possible.

Jamais ne n’aurais pensé, lorsque je discutais dans cette cour de récréation, voir cela de mon vivant.

Désormais, non seulement je le vois venir mais je le vois de plus en plus probable. Je ne saurais quantifier, mais pour moi nous sommes sortis du domaine du possible pour entrer dans celui du probable. Je n’ai pas de solutions miracles, sans changement fondamental de notre mode de vie, mais j’ai des questions, des craintes et aussi un message à faire passer : il est encore temps. La première de nos libertés, c’est encore de voter, faisons le tant que nous en avons encore la possibilité.

Stacking and mosaic creation with JSol’Ex 2.0

Thu, 4 Jan 2024 00:00:00 +0100

A couple days ago I have released JSol’Ex 2.0. This software can be used as an alternative to INTI to process solar images acquired using Christian Buil’s Sol’Ex. This new version introduces 2 new features that I would like to describe in more details in this blog post: stacking and mosaic stitching.

Stacking

Stacking should be something familiar to anyone doing planetary imaging. One of the most popular sofware for doing stacking is AutoStakkert! which has recently seen a new major version. Stacking usually consists of taking a large number of images, selecting a few reference points and trying to align these images to reconstruct a single, stacked image which increases the signal-to-noise ratio. Each of the individual images are usually small and there are a large number of images (since the goal is to reduce the impact of turbulence, typically, videos are taken with a high frame rate, often higher than 100 frames per second). In addition, there are little to no changes between the details of a series (granted that you limit the capture to a few seconds, to avoid the rotation of the planet typically) so the images are really "only" disformed by turbulence. In the context of Sol’Ex image processing, the situation is a bit different: we have a few captures of large images: in practice, capturing an image takes time (it’s a scan of the sun which will consist of a video of several seconds just to build a single image) and the solar details can move quickly between captures. In practice, it means that you can reasonably stack 2 to 5 images, maybe more if the scans are quick enough and that there are not too many changes between scans.

The question for me was how to implement such an algorithm for JSol’Ex? Compared to planetary image stacking, we have a few advantages:

images have a great resolution: depending on the camera that you use and the binning mode, you can have images which range from several hundreds pixels large to a few thousands pixels
the images are well defined : in planetary observation, there are a few "high quality" images in a video of several thousand frames, but most images are either fully or partially disformed
there’s little movement between images: anyone who has stacked a planetary video can see that it’s frequent to see jumps of several pixels between 2 images, just because of turbulence
for each image, we already have determined the ellipse which makes the solar disk and should also have corrected the geometry, so that all solar disks are "perfect circles"

Therefore, a naive approach, which I tried without success a few months ago, is a geometric approach where we simply make all solar disks the same (by resizing), align them then create an average image. To illustrate this approach, let’s look at some images:

Your browser does not support the video tag.

In this video we can see that there is quite some movement visible between each image. Each of them is already of quite good quality, but we can notice some noise and more importantly, a shear effect due to the fact that a scan takes several seconds and that we reconstruct line by line :

The average image is therefore quite blurry:

Therefore, using the average image is not a good option for stacking and that’s why I recommended to use AutoStakkert! instead, which gave better results.

In order to achieve better results, I opted for a simple yet effective algorithm:

first, estimate the sharpness of each image. This is done by computing the Laplacian of each image. The image with the best sharpness is selected as the reference image
divide each image into tiles (by default, a tile has a width of 32 pixels)
for each tile, try to align it with the reference image by computing an error between the reference tile and the image
the error is based on root mean squared error of the pixel intensities : the better the tiles are aligned, the closer to 0 the error will be
this is the most expensive operation, because it requires computing the error for various positions
we’re only looking for displacements with a maximum shift of 2/3 of the tile size (so, by default, 21 pixels maximum)
if the shift between 2 tiles is higher than this limit, we won’t be able to align the tiles properly

We could have stopped here and already reconstruct an image at this stage, but the result wouldn’t be great: the fact that we use tiles would be visible at the edges of the tiles, with square artifacts clearly visible. To reduce the artifacts, I opted for a "sliding window" algorith, where the next tile will overlap the previous one by a factor between 0 (no overlap) and 1 (100% overlap). This means that for each pixel, we will get a "stack" of pixel values computed from the alignment of several tiles. The final pixel value is then computed by taking the median value of the stack. Even so, some stacking vertical or horizontal artifacts can still be sometimes visible, so the last "trick" I used is to build the stacks by only taking pixels within a certain radius, instead of the whole square.

The resulting, stacked image is here:

We can see that:

noise from the original images is gone
shearing artifacts are significantly reduced
the resulting image is not as blurry as the average version

There were, however, some compromises I had to make, in order to avoid that the stacking process takes too long. In particular, the tile alignment process (in particular error computation) is very expensive, since for each tile, we have to compute 21*21 = 441 errors by default. With an overlap factor of 0.3, that’s, for an image of 1024 pixels large, more than 5 million errors to compute. Even computing them in parallel takes long, therefore I added local search optimization: basically, instead of searching in the whole space, I’m only looking for errors within a restricted radius (8 pixels). Then, we take the minimal error of this area and resume searching from that position: step by step we’re moving "closer" to a local optimum which will hopefully be the best possible error. While this doesn’t guarantee to find the best possible solution, it proved to provide very good results while significantly cutting down the computation times.

From several tests I made, the quality of the stacked image matches that of Autostakkert!.

Mosaic composition

The next feature I added in JSol’Ex 2, which is also the one which took me most time to implement, is mosaic composition. To some extent, this feature is similar to stacking, except that in stacking, we know that all images represent the same region of the solar disk and that they are roughly aligned. With mosaics, we have to work with different regions of the solar disk which overlap, and need to be stitched together in order to compose a larger image.

On December 7th, 2024, I had given a glimpse of that feature for the french astrophotograhers association AIP, but I wasn’t happy enough with the result so decided to delay the release. Even today, I’m not fully satisfied, but it gives reasonable results on several images I tried so decided it was good enough for public release and getting feedback about this feature.

Mosaic composition is not an easy task: there are several problems we have to solve:

first, we need to identify, in each image, the regions which "overlap"
then for each image, we need to be able to tell if the pixel value we read at a particular place is relevant for the whole composition or not
then we have to do the alignment
and finally avoid mosaicing artifacts, typically vertical or horizontal lines at the "edges"

In addition, mosaic composition is not immune to the problem that each image can have different illumination, or even that the regions which are overlapping have slightly (or even sometimes significantly) moved between the captures. Therefore, the idea is to "warp" images together in order to make them stitch smoothly.

Preparing panels for integration

Here are the main steps of the algorithm I have implemented:

resize images so that they all solar disks have the same radius (in pixels), and that all images are square
normalize the histograms of each panel so that all images have similar lightness
estimate the background level of each panel, in order to have a good estimate of when a pixel of an image is relevant or not and perform background neutralization
there can be more than 2 panels to integrate. My algorithm works by stitching them 2 by 2, which implies sorting the panels by putting the panels which overlap the most in front, then stitching the 2 most overlapping panels together. The result of the operation is then stitched together with the next panel, until we have integrated all of them.

The stitching part works quite differently than with typical stacking. In stacking, we have complete data for each image: we "only" have to align them. With mosaics, there are "missing" parts in the image that we need to fill in. To do this, we have to identify which part of a panel can be blended into the reconstructed image in order to complete it. This means that the alignment process is significanly more complicated than with typical stacking, since we will work on "missing" data. Part of the difficulty is precisely identifying if something is missing or not, that is to say if the signal of a pixel in one of the panels is relevant to the composition of the final image. This is done by comparing it with the estimated background level, but that’s not the only trick.

Despite the fact that our panels are supposedly aligned and that the circles representing the solar disks are supposed to be the same, in practice, depending on the quality of the capture and the ellipse regression success, the disks may be slightly off, with deformations. There can even be slight rotations between panels (because of flexions at capture time, or processing artifacts). As a consequence, a naive approach consisting of trying to minimize the error between 2 panels by moving them a few pixels in each direction like in stacking doesn’t work:

first of all, while you may properly align one edge of the solar disk, we can see that some regions will be misaligned. If these regions correspond to high contrast areas like filaments, it gives real bad results. If it happens at the edges of the sun, you can even see part of the disk being shifted a few pixels away from the other panel, which is clearly wrong.
second, estimating the error is not so simple, since we have incomplete disks. And in this case, the error has to be computed on large areas, which means that the operation is very expensive.
third, because we have to decide whether to pick a pixel from one panel or the other, this has the tendency to create very strong artifacts (vertical or horizontal lines) at the stitching edges

The stitching algorithm

Giving all the issues I described above, I chose to implement an algorithm which would work similarly to stacking, by "warping" a panel into another. This process is iterative, and the idea is to take a "reference" panel, which is the one which has the most "relevant" pixels, and align tiles from the 2d panel into this reference panel.

To do this, we compute a grid of "reference points" which are in the "overlapping" area. These points belong to the reference image, and one difficulty is to filter out points which belong to "incomplete" tiles. Once we have these points, for each of them, we compute an alignment between the reference tile and the tile of the panel we’re trying to integrate. This gives us, roughly, a "model" of how tiles are displaced in the overlapping area. The larger the overlapping area is, the better the model will be, but experience shows that distorsion on one edge of the solar disk can be significanly different at the other edge.

The next step consists of trying to align tiles of the panel we integrate to the reference panel using this model. This is where the iteration process happens. In a nutshell, we have an area where the solar disk is "truncated". Even if we split the image in tiles like with stacking, we cannot really tell whether a tile is "complete" or not, because it depends both on the pixel intensities of the reference panel and the second panel, and the background level. In particular, calcium images may have dark areas within the solar disk which are sometimes as dark as the background.

If you are struggling to understand how difficult it can be to determine if part of the image we consider is relevant or not, let’s illustrate with this image:

Can you see what’s wrong in this image? Let’s increase constrast to make it clearly visible:

Now it should be pretty obvious that below the south edge of the truncated disk, we have an area which has pixels which are above the value of the background, but do not constitute actual signal! This problem took me quite some time to solve, and it’s only recently that I figured out a solution: before mosaicing, I am performing a background neutralization step, by modeling the background and substracting it from the image. While this doesn’t fully solve the problem, it makes it much less relevant for composition.

In addition, we have to compose the image using tiles which are incomplete, and we don’t know the orientation of the panels: they can be assembled north/south, or west/east, and nothing tells us. Potentially, it can even be a combination of these for a large number of panels.

Therefore, the algorithm works by creating a "mask" of the image being reconstructed. This mask tells us "for this particular pixel, I have reconstructed a value, or the value of the reference image is good enough and we won’t touch it". Then, for each tile, we consider tiles for which the mask is incomplete.

In order to determine how to align the truncated disk with data from the other image, we compute an estimate of the distortion of the tile based on the displacements models we have determined earlier. Basically, for a new "tile" to be integrated, we will consider the sample "reference points" which are within a reasonable distance of the tile. For this set of reference points, we know that they are "close enough" to compute an average model of the distorsion, that I call the "local distorsion": we can estimate, based on the distance of each reference point, how much they contribute to the final distorsion model for that particular point.

The key is really to consider enough samples to have a good distorsion model, but not too many because then the "locality" of alignment would become too problematic and we’d face misalignments. Because there are not so many samples for each "incomplete" tile, we are in fact going to reconstruct, naturally, the image from the edges where there’s missing data: when there are no samples, it basically means we cannot compute a model, so we don’t know how to align tiles. If we have enough samples, then we can compute a reliable model of the distorsion, and then we can reconstruct the missing part of each tile, by properly aligning the tiles together. If the number of samples is not sufficient to consider a good model, then we assume that no distorsion happens, which is often the case for "background" tiles.

Most of the difficulty in this algorithm is properly identifying "when" we can stitch tiles together, that is to say when we can tell that the alignment between tiles makes sense and that the alignment is correct. Often, I got good results for one kind of images (e.g, h-alpha images) but horrible results with others (e.g calcium) or the other way around. I cannot really say I took a very scientific approach to this problem, but more an empirical approach, tweaking parameters of my algorithm until it gave good enough results in all cases.

I mentioned that the algorithm is iterative, but didn’t explain why yet: when we compute the tile alignments, we only do so because we have enough local samples for alignment. We do this for all tiles that match this criteria, but we won’t, for example, be able to align a tile which is in the top of the image, if the bottom hasn’t been reconstructed. Therefore, the iteration happens when we have reconstructed all the tiles we could in one step: then we recompute new reference points, and complete the image, not forgetting, of course, to update our mask to tell that some pixels were completed.

Overall the algorithm is fairly fast, and can be stopped once we have completed all tiles, or after a number of fixed iterations in case of difficulties (often due to the background itself).

One last step

The algorithm we’ve described provides us with a way to "roughly" reconstruct an image, but it doesn’t work like what you’d intuititvely think of mosaic composition, by "moving" 2 panels until they properly align and blend them toghether. Instead, it will reconstruct an image by assembling tiles together, from what is already reconstructed: it is more fine grained, which will fix a number of the issues we’ve faced before: local distorsions, or images which are not properly aligned because the details at the surface at the sun have moved between the moment the first panel was captured and the second one did.

If we stopped there, we would see an image which looks like this:

We can see a clear horizontal line, which is due to the fact that we’re reconstructing using tiles, and that depending on the alignment of tiles with the "missing" areas, we can have strong or weak artifacts at the borders. Errors are even more visible in this image in calcium K line:

This time it’s very problematic and we are facing several of the issues we attempted to avoid: details have significantly moved between the north and south panel were captured, which leads to "shadowing" artifacts, and there are also tiling artifacts visible.

However, the image we get is good enough to perform one last step: use it as a reference image in the stacking algorithm we described in the first section of this blog post. The reason stacking works well is because we know we have complete images that we can align. Here, we have roughly reconstructed an image that we can use as a "complete reference". The idea is therefore to take each tile of each panel and "blend" it using the reconstructed reference. Of course, there is one big difference between the stacking in the first section and the stacking we have to do now. We’re not really going to use the reconstructed image, except for aligning tiles together and computing a "weight" for each tile, which depends on the relative luminosity between the reference image tile we’re considering and the corresponding panel tile.

This gives us a pretty good result:

The images we got there are not perfect, which is why I’m not fully satisfied yet, but they are however already quite good, given that it’s all done in a few seconds, using the same software that you’d use to reconstruct Sol’Ex images! In other words, the goal of these features is not to get the same level of quality that you’d get by using your favorite post-processing or mosaic composition software, but good enough to get you a reasonable result in a reasonable amount of time.

For example, on my machine, it takes less than one minute to:

stack images of the north and south panels (~10 images to stack)
stitch them together in a mosaic

It would have taken several minutes, or even more, using external software, especially for the mosaic part.

Conclusion

In this blog post, I have described how I got to implement 2 algorithms, the stacking algorithm and the mosaic composition one, in JSol’Ex. None of the algorithms were based on any research paper: they were really designed in an "adhoc" way, as my intuition of how things could work. It proved to be quite difficult, and it is very likely that better algorithms are described in the wild: I will consider them for future versions.

Nevertheless, I’m quite happy with the outcome, since, remember, I have started this program as an experiment and for learning purposes only. Now I sincerely hope that it will help you get amazing solar images!

Introducing astro4j

Sat, 22 Apr 2023 00:00:00 +0200

This blog introduces astro4j, my latest toy project, a open source collection of libraries and applications for astronomy, written in Java. In particular, I will discuss JSol’Ex, a program aimed at reconstructing solar disk images from video files captured using the amazing Sol’Ex instrument.

Why astro4j?

I’m a software developer, and if you are following me, you may also know that I’m an amateur astrophotographer. For a long time, I’ve been fascinated by the quality of software we have in astronomy, to process images. If you are french speaking, you can watch a presentation I gave about this topic. Naturally, I have been curious about how all these things work, but it’s actually extremely rare to find open source software, and when you do, it’s rarely written in Java. For example, both Firecapture (software to capture video streams) and Astro Pixel Processor are written in Java, but both of them are closed source, commercial software.

Last month, for my birthday, I got a Sol’Ex, an instrument which combines spectrography and software to realize amazing solar pictures in different spectral lines. To process those images, the easiest solution is to use the amazing INTI software, written in Python, but for which sources are not published, as far as I know, neither on GitHub or GitLab.

Note	After announcing this project, I have been notified that the sources of INTI are indeed available, as GPL. It’s a pity they are not linked on the webpage, this would have helped a lot.

To give you an example of what you can do, here’s the first photography I’ve done with Sol’Ex and processed with INTI (color was added in Gimp):

To get this result, one has to combine images which look like this:

Interesting, no? At the same time, I was a bit frustrated by INTI. While it clearly does the job and is extremely easy to use, there are a few things which I didn’t like:

the first, which I mentioned, is that it’s using Python and that the sources are not published (as far as I understand, some algorithms are not published yet). I am not surprised that Python is used, because it’s a language which is extremely popular in academics, with lots of libraries for image processing, science oriented libs, etc. However, because it’s popular in academics also means that programs are often written by and for academics. When we’re talking about maths, it’s often short variable names, cryptic function names, etc…
second, after processing, INTI pops up a lot of images as individual windows. If you want to process a new file, you have to close all of them. The problem is that I still haven’t figured out in which order you have to do this so that you can restart from the initial window which lets you select a video file! Apparently, depending on the order, it will, or will not, show the selector. And sometimes, it takes several seconds before it does so.
INTI seems to be regenerating a font cache every time I reboot. This operation takes several minutes. It’s probably an artifact of packaging the application for Windows, but still, not very user friendly.
INTI generates a number of images, but puts them alongside the videos. I like things organized (well, at least virtually, because if you looked at my desk right now, it is likely you’d feel faint), so I wish it was creating one directory per processed video.

Confronting the old demons

When I started studying at University, back in 1998, I was planning to do astrophysics. However, I quickly forgot about this idea when I saw the amount of maths one has to master to do modern physics. Clearly, I was reaching my limits, and it was extremely complicated for me. Fortunately, I had been doing software development for years already, because I started very young, on my father’s computer. So I decided to switch to computer science, where I was reasonably successful.

However, not being able to do what I wanted to do has always been a frustration. It is still, today, to the point that a lot of what I’m reading is about this topic, but still, I lack the maths.

It was time for me to confront my old demons, and answer a few questions:

am I still capable of understanding maths, in order to implement algorithms which I use everyday when I do astronomy image processing with software written by others?
can I read academic papers, for example to implement a FFT (Fast Fourier Transform) algorithm, although I clearly remember that I failed to understand the principles when I was at school?
can I do this while writing something which could be useful to others, and publish it as open source software?

Astro4j is there to answer those questions. I don’t have the answers yet and time will tell if I’m successful.

Using modern Java

One question you may have is why Java? If you are not familiar with this language, you may have this old misconception that Java is slow. It’s not. Especially, if you compare to Python, it’s definitely not.

This project is also for me a way to prove that you can implement "serious science" in Java. You can already find some science libraries in Java, but they tend to me impractical to use, because not following the industry standards (e.g published on Maven Central) or platform-dependent.

I also wanted to leverage this to learn something new. So this project:

uses Java 17 (at least for libraries, so that they can be consumed by a larger number of developers, for applications I’m considering moving to Java 20)
uses JavaFX (OpenJFX) for the application UI
experiments with the Vector API for faster processing

As I said, my initial goal is to obtain a software which can basically do what INTI does. It is not a goal to make it faster, but if I can do it, I will.

Introducing JSol’Ex

After a few evenings (and a couple week-ends ;)), I already have something which performs basic processing, that is to say that it can process a SER video file and generate a reconstructed solar disk. It does not perform geometry correction, nor tilt correction, like INTI does. It doesn’t generate shifted images either (for example the doppler images), but it works.

Since the only source of information I had to do this was Christian Buil’s website and Valérie Desnoux INTI’s website, I basically had to implement my own algorithms from A to Z, and just "guess" how it works.

In order to do this, I had to:

implement a SER video file decoder. The library is ready and performs both decoding the SER file and performs demosaicing of images
on top of the decoder, I implemented a SER file player, which is still very basic at this stage, and uses JavaFX. This player can even be compiled to a native binary using GraalVM!

Here’s an example:

Your browser does not support the video tag.

Then I could finally start working on the Sol’Ex video processor. As I said, I don’t know how INTI works, so this is all trial and error, in the end…

In the beginning, as I said, you have a SER video file which contains a lot of frames (for example, in my case, it’s a file from 500MB to 1GB) that we have to process in order to generate a solar disk. Each frame consists of a view of the light spectrum, centered on a particular spectral line.

For example, in the following image, we have the H-alpha spectral line:

Because of optics, you can see that the line is not horizontal: each frame is distorted. Therefore, in order to reconstruct an image, we have to deal with that distortion first. For this, we have to:

detect the spectral line in the frame, which I’m doing by implementing a simple contrast detection
perform a linear regression in order to compute a 2d order polynomial which models the distortion

Note that before doing this, I had no idea how to do a 2d order regression, but I searched and found that it was possible to do so using the least squares method, so I did so. The result is that we can identify precisely the line with this technique:

In the beginning, I tought I would have to perform distortion correction in order to reconstruct the image, because I was (wrongly) assuming that, because each frame represents one line in the reconstructed image, I had to compute the average of the colums of each frame to determine the color of a single pixel in the output. I was wrong (we’ll come to that later), but I did implement a distortion correction algorithm:

When I computed the average, the resulting image was far from the quality and constrast of what I got with INTI. What a failure! So I thought that maybe I had to compute the average of the spectral line itself. I tried this, and indeed, the resulting image was much better, but still not the quality of INTI. The last thing I did, therefore, was to pick the middle of the spectral line itself, and then, magically, I got the same level of quality as with INTI (for the raw images, as I said I didn’t implement any geometry or tilt correction yet).

The reason I was assuming that I had to compute an average, is that it wasn’t clear to me that the absorption ray would actually contain enough data to reconstruct an image. As it was an absorption ray, I assumed that the value would be 0, and therefore that nothing would come out of using the ray itself. In fact, my physics were wrong, and you must use that.

A direct consequence is that there is actually no need to perform a distortion correction. Instead, you can just use the 2d order polynomial that we’ve computed, and "follow the line", that’s it!

Now, we can generate an image, but it will be very dark. The reason is obvious: by taking the middle of the spectral line, we’re basically using dark pixels, so the dynamics of the image are extremely low. So, in order to have something which "looks nice", you actually have to perform brightness correction.

The first algorithm I have used is simply a linear correction: we’re computing the max and min value of the image, then rescaling that so that the max value is the maximum representable (255).

Here’s the result:

However, I felt that this technique wouldn’t give the best results, in particular because linear images tend to give results which are not what the eye would see: our eye performs a bit like an "exponential" accumulator, the more photos you get, the "brighter" we’ll see it.

So I implemented another algorithm which I had seen in PixInsight, which is called inverse hyperbolic (Arcsinh) correction:

Last, you can see that the image has lots of vertical line artifacts. This is due to the presence of dust either on the optics or the sensors. INTI performs correction of those lines, and I wanted to do something similar.

Again, I don’t know what INTI is doing, so I figured out my own technique, which is using "multipass" correction. In a nutshell, for each row, I am computing the average value of the row. Then, for a particular row, I compute the average of the averages of the surrounding lines (for example, 16 rows before and after). If the average of this line is below the average of the averages(!), then I’m considering that the line is darker than it should be, computing a correction factor and applying it.

The result is a corrected image:

We’re still not a the level of quality that INTI produces, but getting close!

So what’s next? I already have added some issues for things I want to fix, and in particular, I’m looking at improving the banding reduction and performing geometry correction. For both, I think I will need to use fast fourier transforms, in order to identify the noise in one case (banding) and detect edges in the other (geometry correction).

Therefore, I started to implement FFT transforms, a domain I had absolutely no knowledge of. Luckily, I could ask ChatGPT to explain to me the concepts, which made it faster to implement! For now, I have only implemented the Cooley-Tukey algorithm. The issue is that this algorithm is quite slow, and requires that the input data has a length which is a power of 2. Given the size of the image we generate, it’s quite costly.

I took advantage of this to learn about the Vector API to leverage SIMD instructions of modern CPUs, and it indeed made things significantly faster (about twice as fast), but still not at the level of performance that I expect.

I am trying to understand the split radix but I’m clearly intimidated by the many equations here… In any case I printed some papers which I hope I’ll be able to understand.

Conclusion

In conclusion, in this article, I’ve introduced astro4j, an open source suite of libraries and applications written in Java for astronomy software. While the primary goal for me is to learn and improve my skills and knowledge of the maths behind astronomy software processing, it may be that it produces something useful. In any case, since it’s open source, if you want to contribute, feel free!

And you can do so in different domains, for example, I pretty much s* at UI, so if you are a JavaFX expert, I would appreciate your pull requests!

Finally, here is a video showing JSol’Ex in action:

Your browser does not support the video tag.

How the Micronaut team leverages Gradle’s version catalogs for improved developer productivity

Sun, 12 Mar 2023 00:00:00 +0100

This blog post discusses how the Micronaut development team makes use of a feature of Gradle, version catalogs, to improve the team’s developer productivity, reduce the risks of publishing broken releases, coordinate the releases of a large number of modules and, last but not least, provide additional features to our Gradle users.

The backstory

The Micronaut Framework is a modern open-source framework for building JVM applications. It can be used to build all kinds of applications, from CLI applications to microservices or even good old monoliths. It supports deploying both to the JVM and native executables (using GraalVM), making it particularly suitable for all kind of environments. A key feature of the Micronaut framework is developer productivity: we do everything we can to make things faster for developers. In particular, Micronaut has a strong emphasis on easing how you test your applications, even in native mode. For this we have built a number of tools, including our Maven and Gradle plugins.

When I joined the Micronaut team almost a couple years back, I was given the responsibility of improving the team’s own developer productivity. It was an exciting assignment, not only because I knew the team’s love about Gradle, but because I also knew that there were many things we could do to reduce the feedback time, to provide more insights about failures, to detect flaky tests, etc. As part of this work we have put in place a partnership with Gradle Inc which kindly provides us with a Gradle Enterprise instance, but this is not what I want to talk about today.

Lately I was listening to an interview of Aurimas Liutikas of the AndroidX team, who was saying that he didn’t think that version catalogs were a good solution for library authors to share their recommendations of versions, and that BOMs are probably a better solution for this. I pinged him saying that I disagreed with this statement and offered to provide more details why, if he was interested. This is therefore a long answer, but one which will be easier to find than a thread on social media.

What are version catalogs?

Let’s start with the basics: a version catalog is, like the name implies, a catalog of versions to pick from, nothing more. That doesn’t sound too much exciting, and what versions are we talking about? That’s version of libraries or plugins that you use in your build.

As an illustration, here is a version catalog, defined as a TOML file:

[versions]
javapoet = "1.13.0"

[libraries]
javapoet = { module = "com.squareup:javapoet", version.ref = "javapoet" }

Then this library can be used in a dependencies declaration block in any of the project’s build script using a type-safe notation:

dependencies {
    implementation(libs.javapoet) {
        because("required for Java source code generation")
    }
}

which is strictly equivalent to writing:

dependencies {
    implementation("com.squareup:javapoet:1.13.0") {
        because("required for Java source code generation")
    }
}

There are many advantages of using version catalogs to declare your library versions, but most notably it provides a single, standard location where those versions are declared. It is important to understand that a catalog is simply a list of dependencies you can pick from, a bit like going to the supermarket and choosing whatever you need for your particular meal: it’s not because a catalog declares libraries that you have to use them. However, a catalog provides you with recommendations of libraries to pick from.

Version catalogs for Micronaut users

An interesting aspect of version catalogs is that they can be published, for others to consume: they are an artifact. Micronaut users can already make use of catalogs, as I have explained in a previous blog post. This makes it possible for a user who doesn’t know which version of Micronaut Data to use, to simply declare:

dependencies {
    implementation mn.micronaut.data
}

People familiar with Maven BOMs can easily think that it is the same feature, but there are key differences which are described in the Gradle docs.

In the rest of this post we will now focus on how we generate those catalogs, and how they effectively help us in improving our own developer productivity.

How the Micronaut team uses version catalogs

One catalog per module

As I said, the Micronaut framework consists of a large number of modules which live in their own Git repository. All the projects share the same layout, the same conventions in order to make things easier to maintain. For this purpose, we use our own collection of internal build plugins as well as a project template.

Those build plugins provide features like:

defining the default Java language level, setting up code conventions and code quality plugins
standardizing how documentation is built (using Asciidoctor)
setting up integration with Gradle Enterprise, to publish build scans, configure the build cache and predictive test selection
implementing binary compatibility checks between releases
configuring publication to Maven Central
providing a high-level model of what a Micronaut module is

The last item is particularly important: in every Micronaut project, we have different kind of modules: libraries (which are published to Maven Central for users to consume), internal support libraries (which are not intended for external consumption), or a BOM module (which also publishes a version catalog as we’re going to see).

Long story short: we heavily rely on conventions to reduce the maintenance costs, have consistent builds, with improved performance and higher quality standards. If you are interested in why we have such plugins, Sergio Delamo and I gave an interview about this a few months ago (alert: the thumbnail shows I have hair, this is fake news!).

Each of our projects declares a version catalog, for example:

this one for Micronaut core
this one for Micronaut Test Resources
this one for Micronaut Kafka
or this one for the Micronaut Gradle Plugin

Automatic version upgrades

One of the advantages of version catalogs is that it provides a centralized place for versions, which can be easily used by bots to provide pull requests for dependency upgrades. For this, we use Renovatebot which integrates particularly well with version catalogs (GitHub’s dependabot lacks behind in terms of support). This allows us to get pull requests like this one which are very easy to review.

BOM and version catalog generation

Each of the Micronaut projects is now required to provide a BOM (Bill of Materials) for users. Another term for a BOM that is used in the Gradle ecosystem is a platform: a platform has however slightly different semantics in Maven and Gradle. The main goal of a BOM is to provide a list of dependencies a project works with, and, in Maven, it can be used to override the dependency versions of transitive dependencies. While in Maven, a BOM will only influence the dependency resolution of the project which imports the BOM, in Gradle a platform fully participates in dependency resolution, including when a transitive dependency depends on a a BOM. To simplify, a user who imports a BOM may use dependencies declared in the BOM without specifying a version: the version will be fetched from the BOM. In that regards, it looks exactly the same as a version catalog, but there are subtle differences.

For example, if a user imports a BOM, any transitive dependency matching a dependency found in the BOM will be overridden (Maven) or participate in conflict resolution (Gradle). That is not the case for a catalog: it will not influence the dependency resolution unless you explicitly add a dependency which belongs to the catalog.

That’s why Micronaut publishes both a BOM and a catalog, because they address different use cases, and they work particularly well when combined together.

In Micronaut modules, you will systematically find a project with the -bom suffix. For example, Micronaut Security will have subprojects like micronaut-security-jwt, micronaut-security-oauth2 and micronaut-security-bom.

The BOM project will aggregate dependencies used by the different modules. In order to publish a BOM file, the only thing a project has to do is to apply our convention plugin:

plugins {
    id "io.micronaut.build.internal.bom"
}

Note how we don’t have to declare the coordinates of the BOM (group, artifact, version), nor that we have to declare how to publish to Maven Central, what dependencies should be included in the BOM, etc: everything is done by convention, that’s the magic of composition over inheritance.

Should we want to change how we generate the BOM, the only thing we would have to do is to update our internal convention plugin, then all projects would benefit from the change once they upgrade.

Convention over configuration

In order to determine which dependencies should be included in our BOM, we defined conventions that we use in our catalog files. In our internal terminology, when we want a dependency to be handled by the Micronaut framework, we call that a managed dependency: a dependency that is managed by Micronaut and that users shouldn’t care about in most cases: they don’t have to think about a version, we will provide one for them.

This directly translates to a convention in the version catalogs of the Micronaut projects: dependencies which are managed need to be declared with a managed- prefix in the catalog:

[versions]
...
managed-kafka = '3.4.0'
...
zipkin-brave-kafka-clients = '5.15.0'

[libraries]
...
managed-kafka-clients = { module = 'org.apache.kafka:kafka-clients', version.ref = 'managed-kafka' }
managed-kafka-streams = { module = 'org.apache.kafka:kafka-streams', version.ref = 'managed-kafka' }
...
zipkin-brave-kafka-clients = { module = 'io.zipkin.brave:brave-instrumentation-kafka-clients', version.ref = 'zipkin-brave-kafka-clients' }

Those dependencies will end up in the version catalog that we generate, but without the managed- prefix. This means that we would generate a BOM which contains the following:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <!-- This module was also published with a richer model, Gradle metadata,  -->
  <!-- which should be used instead. Do not delete the following line which  -->
  <!-- is to indicate to Gradle or any Gradle module metadata file consumer  -->
  <!-- that they should prefer consuming it instead. -->
  <!-- do_not_remove: published-with-gradle-metadata -->
  <modelVersion>4.0.0</modelVersion>
  <groupId>io.micronaut.kafka</groupId>
  <artifactId>micronaut-kafka-bom</artifactId>
  <version>5.0.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <name>Micronaut Kafka</name>
  <description>Integration between Micronaut and Kafka Messaging</description>
  <url>https://micronaut.io</url>
  <licenses>
    <license>
      <name>The Apache Software License, Version 2.0</name>
      <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
      <distribution>repo</distribution>
    </license>
  </licenses>
  <scm>
    <url>scm:git@github.com:micronaut-projects/micronaut-kafka.git</url>
    <connection>scm:git@github.com:micronaut-projects/micronaut-kafka.git</connection>
    <developerConnection>scm:git@github.com:micronaut-projects/micronaut-kafka.git</developerConnection>
  </scm>
  <developers>
    <developer>
      <id>graemerocher</id>
      <name>Graeme Rocher</name>
    </developer>
  </developers>
  <properties>
    <micronaut.kafka.version>5.0.0-SNAPSHOT</micronaut.kafka.version>
    <kafka.version>3.4.0</kafka.version>
  </properties>
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>${kafka.compat.version}</version>
      </dependency>
      <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-streams</artifactId>
        <version>${kafka.version}</version>
      </dependency>
      <dependency>
        <groupId>io.micronaut.kafka</groupId>
        <artifactId>micronaut-kafka</artifactId>
        <version>${micronaut.kafka.version}</version>
      </dependency>
      <dependency>
        <groupId>io.micronaut.kafka</groupId>
        <artifactId>micronaut-kafka-streams</artifactId>
        <version>${micronaut.kafka.version}</version>
      </dependency>
    </dependencies>
  </dependencyManagement>
</project>

Note how we automatically translated the managed-kafka property into a BOM property kafka.version, which is used in the <dependencyManagement> block. Dependencies which do not start with managed- are not included in our generated BOM.

Let’s now look at the version catalog that we generate:

#
# This file has been generated by Gradle and is intended to be consumed by Gradle
#
[metadata]
format.version = "1.1"

[versions]
kafka = "3.4.0"
kafka-compat = "3.4.0"
micronaut-kafka = "5.0.0-SNAPSHOT"

[libraries]
kafka = {group = "org.apache.kafka", name = "kafka-clients", version.ref = "kafka-compat" }
kafka-clients = {group = "org.apache.kafka", name = "kafka-clients", version.ref = "kafka" }
kafka-streams = {group = "org.apache.kafka", name = "kafka-streams", version.ref = "kafka" }
micronaut-kafka = {group = "io.micronaut.kafka", name = "micronaut-kafka", version.ref = "micronaut-kafka" }
micronaut-kafka-bom = {group = "io.micronaut.kafka", name = "micronaut-kafka-bom", version.ref = "micronaut-kafka" }
micronaut-kafka-streams = {group = "io.micronaut.kafka", name = "micronaut-kafka-streams", version.ref = "micronaut-kafka" }

Given a single input, the version catalog that we use to build our Micronaut module, our build conventions let us automatically declare which dependencies should land in the output BOM and version catalogs that we generate for that project! Unlike Maven BOMs which either have to be a parent POM or redeclare all dependencies in an independent module, in Gradle we can generate these automatically and completely decouple the output BOM from what is required to build our project.

In general, all api dependencies must be managed, so in the example above, the Micronaut Kafka build scripts would have an API dependency on kafka-clients, which we can find in the main project build script:

dependencies {
    api libs.managed.kafka.clients
    ...
}

The benefit of generating a version catalog for a user is that there is now a Micronaut Kafka version catalog published on Maven Central, alongside the BOM file.

This catalog can be imported by a user in their settings file:

settings.gradle

dependencyResolutionManagement {
    versionCatalogs {
         create("mnKafka") {
             from("io.micronaut.kafka:micronaut-kafka-bom:4.5.2")
         }
    }
}

Then the dependency on Micronaut Kafka and its managed dependencies can be used in a build script using the mnKafka prefix:

build.gradle

dependencies {
    implementation mnKafka.micronaut.kafka
    implementation mnKafka.kafka.clients
}

A user doesn’t have to know about the dependency coordinates of Kafka clients: the IDE (at least IntelliJ IDEA) would provide completion automatically!

BOM composition

In Micronaut 3.x, there is a problem that we intend to fix in Micronaut 4 regarding our "main" BOM: the Micronaut core BOM is considered as our "platform" BOM, in the sense that it aggregates BOMs of various Micronaut modules. This makes it hard to release newer versions of Micronaut which, for example, only upgrade particular modules of Micronaut.

Therefore in Micronaut 4, we are cleanly separating the "core" BOM, from the new platform BOM. It is interesting in this blog post because it offers us the opportunity to show how we are capable of generating aggregating BOMs and aggregated catalogs.

In the platform BOM module, you can find the "input" catalog that we use, and only consists of managed- dependencies. Most of those dependencies are simply dependencies on other Micronaut BOMs: this is an "aggregating" BOM, which imports other BOMs. This is, therefore, the only BOM that a user would effectively have to use when migrating to Micronaut 4: instead of importing all BOMs for the different Micronaut modules they use, they can simply import the Micronaut Platform BOM, which will then automatically include the BOMs of other modules which "work well together".

This allows us to decouple the releases of the framework from the releases of Micronaut core itself.

However, there is a subtlety about aggregating BOMs in Maven: they are not regular dependencies, but dependencies with the import scope. This means that we must make a difference between a "managed dependency" and an "imported BOM" in our input catalog.

To do this, we have another naming convention, which is to use the boms- prefix for imported BOMs:

[versions]
...
managed-micronaut-aws = "4.0.0-SNAPSHOT"
managed-micronaut-azure = "5.0.0-SNAPSHOT"
managed-micronaut-cache = "4.0.0-SNAPSHOT"
managed-micronaut-core = "4.0.0-SNAPSHOT"
...

[libraries]
...
boms-micronaut-aws = { module = "io.micronaut.aws:micronaut-aws-bom", version.ref = "managed-micronaut-aws" }
boms-micronaut-azure = { module = "io.micronaut.azure:micronaut-azure-bom", version.ref = "managed-micronaut-azure" }
boms-micronaut-cache = { module = "io.micronaut.cache:micronaut-cache-bom", version.ref = "managed-micronaut-cache" }
boms-micronaut-core = { module = "io.micronaut:micronaut-core-bom", version.ref = "managed-micronaut-core" }
...

This results in the following BOM file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>io.micronaut.platform</groupId>
  <artifactId>micronaut-platform</artifactId>
  <version>4.0.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <name>Micronaut Platform</name>
  <description>Bill-Of-Materials (BOM) and Gradle version catalogs for Micronaut</description>

  ...

  <properties>
    ...
    <micronaut.aws.version>4.0.0-SNAPSHOT</micronaut.aws.version>
    <micronaut.azure.version>5.0.0-SNAPSHOT</micronaut.azure.version>
    <micronaut.cache.version>4.0.0-SNAPSHOT</micronaut.cache.version>
    <micronaut.core.version>4.0.0-SNAPSHOT</micronaut.core.version>
    ...
  </properties>
  <dependencyManagement>
    <dependencies>
      ...
      <dependency>
        <groupId>io.micronaut.aws</groupId>
        <artifactId>micronaut-aws-bom</artifactId>
        <version>${micronaut.aws.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      <dependency>
        <groupId>io.micronaut.azure</groupId>
        <artifactId>micronaut-azure-bom</artifactId>
        <version>${micronaut.azure.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      <dependency>
        <groupId>io.micronaut.cache</groupId>
        <artifactId>micronaut-cache-bom</artifactId>
        <version>${micronaut.cache.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      <dependency>
        <groupId>io.micronaut</groupId>
        <artifactId>micronaut-core-bom</artifactId>
        <version>${micronaut.core.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      ...
    </dependencies>
  </dependencyManagement>
</project>

A more interesting topic to discuss is what we can do with version catalogs that we publish for users: we can inline dependency aliases from each of the imported catalogs into the platform catalog. All dependencies in the catalog files of each modules are directly available in the platform catalog:

[versions]
dekorate = "1.0.3"
elasticsearch = "7.17.8"
...
micronaut-aws = "4.0.0-SNAPSHOT"
micronaut-azure = "5.0.0-SNAPSHOT"
micronaut-cache = "4.0.0-SNAPSHOT"
micronaut-core = "4.0.0-SNAPSHOT"
...

[libraries]
alexa-ask-sdk = {group = "com.amazon.alexa", name = "ask-sdk", version = "" }
alexa-ask-sdk-core = {group = "com.amazon.alexa", name = "ask-sdk-core", version = "" }
alexa-ask-sdk-lambda = {group = "com.amazon.alexa", name = "ask-sdk-lambda-support", version = "" }
aws-java-sdk-core = {group = "com.amazonaws", name = "aws-java-sdk-core", version = "" }
aws-lambda-core = {group = "com.amazonaws", name = "aws-lambda-java-core", version = "" }
aws-lambda-events = {group = "com.amazonaws", name = "aws-lambda-java-events", version = "" }
aws-serverless-core = {group = "com.amazonaws.serverless", name = "aws-serverless-java-container-core", version = "" }
awssdk-secretsmanager = {group = "software.amazon.awssdk", name = "secretsmanager", version = "" }
azure-cosmos = {group = "com.azure", name = "azure-cosmos", version = "" }
azure-functions-java-library = {group = "com.microsoft.azure.functions", name = "azure-functions-java-library", version = "" }
...

The alexa-ask-sdk is for example an alias which was originally declared in the micronaut-aws module. Because we aggregate all catalogs, we can inline those aliases and make them directly available in user build scripts:

settings.gradle

dependencyResolutionManagement {
    versionCatalogs {
         create("mnKafka") {
             from("io.micronaut.platform:micronaut-platform:4.0.0-SNAPSHOT")
         }
    }
}

build.gradle

dependencies {
...
    implementation(mn.micronaut.aws.alexa)
    implementation(mn.alexa.sdk)
}

Generating a version catalog offers us a very pragmatic way to define all dependencies that users can use in their build scripts with guarantees that they work well together.

Technical details

If you survived reading up to this point, you may be interested in learning how, technically, we implemented this. You can take a look at our internal build plugins, but more specifically at the BOM plugin.

In order to generate our BOM and version catalogs, we have mainly 2 inputs:

the list of subprojects which need to participate in the BOM: in a Micronaut modules, we explained that we have several kinds of projects: libraries which are published, test suites, etc. Only a subset of these need to belong to the BOM, and we can determine that list automatically because each project applies a convention plugin which determines its kind. Only projects of a particular kind are included. Should exceptions be required, we have a MicronautBomExtension which allows us to configure more precisely what to include or not, via a nice DSL.
the list of dependencies, which is determined from the project’s version catalog

One issue is that while Gradle provides automatically the generated, type-safe accessors for version catalogs, there is actually no built-in model that you can access to represent the catalog model itself (what is an alias, references to versions, etc): the type-safe API represents a "realized" catalog, but not a low-level model that we can easily manipulate. This means that we had to implement our own model for this.

We have also seen that we can generate a single platform, aggregating all Micronaut modules for a release, that the users can import into their build scripts. Unfortunately it is not the case for the Micronaut modules themselves: for example, Micronaut Core must not depend on other Micronaut modules, but, for example, Micronaut Data can depend on Micronaut SQL and use dependencies from the Micronaut SQL catalog. Those modules cannot depend on the platform BOM, because this is the aggregating BOM, so we would create a cyclic dependency and wouldn’t be able to release any module.

To mitigate this problem, our internal build plugins expose a DSL which allows each projects to declare which other modules they use:

settings.gradle

micronautBuild {
    importMicronautCatalog() // exposes a `mn` catalog
    importMicronautCatalog("micronaut-reactor") // exposes a `mnReactor` catalog
    importMicronautCatalog("micronaut-rxjava2") // exposes a `mnRxjava2` catalog
    ...
}

While this is simple from the declaration site point of view, it is less practical from a consuming point of view, since it forces us to use different namespaces for each imported catalog:

dependencies {
    ...
    testImplementation mn.micronaut.inject.groovy
    testImplementation mnRxjava2.micronaut.rxjava2
    ...
}

It would have been better if we could actually merge several catalogs into a single one, but unfortunately that feature has been removed from Gradle. I still have hope that this will eventually be implemented, because not having this creates unnecessary boilerplate in build scripts and redundancy in names (e.g implementation mnValidation.micronaut.validation).

Additional benefits and conclusion

All that I described in this article aren’t the only benefits that we have on standardizing on version catalogs. For example, we have tasks which allow us to check that our generated BOM files only reference dependencies which are actually published on Maven Central, or that there are no SNAPSHOT dependencies when we perform a release. In the end, while most of the Micronaut developers had no idea what a version catalog was when I joined the team, all of them pro-actively migrated projects to use them because, I think, they immediately saw the benefits and value. It also streamlined the dependency upgrade process which was still a bit cumbersome before, despite using dependabot.

We now have a very pragmatic way to both use catalogs for building our own projects, and generating BOMs and version catalogs which can be used by both our Maven and Gradle users. Of course, only the Gradle users will benefit from the version catalogs, but we did that in a way which doesn’t affect our Maven users (and if you use Maven, I strongly encourage you to evaluate building Micronaut projects with Gradle instead, since the UX is much better).

I cannot end this blog post without mentioning a "problem" that we have today, which is that if you use Micronaut Launch to generate a Micronaut project, then it will not use version catalogs. We have an issue for this and pull requests are very welcome!

Gradle’s flexibility in action

Mon, 6 Feb 2023 00:00:00 +0100

I often say that flexibility isn’t the reason why you should select Gradle to build your projects: reliability, performance, reproducibility, testability are better reasons. There are, however, cases were its flexibility comes in handy, like last week, when a colleague of mine asked me how we could benchmark a Micronaut project using a variety of combination of features and Java versions. For example, he wanted to compare the performance of an application with and without epoll enabled, with and without Netty’s tcnative library, with and without loom support, building both the fat jar and native binary, etc. Depending on the combinations, the dependencies of the project may be a little different, or the build configuration may be a little different.

It was an interesting challenge to pick up and the solution turned out to be quite elegant and very powerful.

Conceptual design

I have tried several options before this one, which I’m going to explain below, but let’s focus with the final design (at least at the moment I write this blog post). The matrix of artifacts to be generated can be configured in the settings.gradle file:

combinations {
    dimension("tcnative") {   (1)
        variant("off")
        variant("on")
    }
    dimension("epoll") {      (2)
        variant("off")
        variant("on")
    }
    dimension("json") {       (3)
        variant("jackson")
        variant("serde")
    }
    dimension("micronaut") {  (4)
        variant("3.8")
        variant("4.0")
    }
    dimension("java") {       (5)
        variant("11")
        variant("17")
    }
    exclude {                 (6)
        // Combination of Micronaut 4 and Java 11 is invalid
        it.contains("micronaut-4.0") && it.contains("java-11")
    }
}

1	a dimension called `tcnative` is defined with 2 variants, `on` and `off`
2	another dimension called `epool` also has `on` and `off` variants
3	the `json` dimension will let us choose 2 different serialization frameworks: Jackson or Micronaut Serde
4	we can also select the version of Micronaut we want to test
5	as well as the Java version!
6	some invalid combinations can be excluded

The generates a number of synthetic Gradle projects, that is to say "projects" in the Gradle terminology, but without actually duplicating sources and directories on disk. With the example above, we generate the following projects:

:test-case:tcnative-off:epoll-off:json-jackson:micronaut-3.8:java-11
:test-case:tcnative-off:epoll-off:json-jackson:micronaut-3.8:java-17
:test-case:tcnative-off:epoll-off:json-jackson:micronaut-4.0:java-17
:test-case:tcnative-off:epoll-off:json-serde:micronaut-3.8:java-11
:test-case:tcnative-off:epoll-off:json-serde:micronaut-3.8:java-17
:test-case:tcnative-off:epoll-off:json-serde:micronaut-4.0:java-17
:test-case:tcnative-off:epoll-on:json-jackson:micronaut-3.8:java-11
:test-case:tcnative-off:epoll-on:json-jackson:micronaut-3.8:java-17
:test-case:tcnative-off:epoll-on:json-jackson:micronaut-4.0:java-17
… and more

To build the fat jar of the "tcnative on", "epoll on", "Jackson", "Micronaut 4.0" on Java 17 combination, you can invoke:

$ ./gradlew :test-case:tcnative-on:epoll-on:json-jackson:micronaut-4.0:java-17:shadowJar

And building the native image of the "tcnative off", "epoll on", "Micronaut Serde", "Micronaut 3.8" on Java 17 combination can be done with:

$ ./gradlew :test-case:tcnative-off:epoll-on:json-serde:micronaut-3.8:java-17:nativeCompile

Cherry on the cake, all variants can be built in parallel by executing either ./gradlew shadowJar (for the fat jars) or ./gradlew nativeCompile (for the native binaries), which would copy all the artifacts under the root projects build directory so that they are easy to find in a single place.

How does it work?

In a typical project, say the Micronaut application we want to benchmark, you would have a project build which consists of a single Micronaut application module. For example, running ./gradlew build would build that single project artifacts. In a multi-project build, you could have several modules, for example core and app, and running :core:build would only build the core library and :app:build would build both core and app (assuming app depends on core. In both cases, single or multi-project builds, for a typical Gradle project, there’s a real directory associated for each project core, app, etc, where we can find sources, resources, build scripts, etc.

For synthetic projects, we actually generate Gradle projects (aka modules) programmatically. We have a skeleton directory, called test-case-common, which actually defines our application sources, configuration files, etc. It also contains a build script which applies a single convention plugin, named io.micronaut.testcase. This plugin basically corresponds to our "baseline" build: it applies the Micronaut plugin, adds a number of dependencies, configures native image building, etc.

Then the "magic" is to use Gradle’s composition model for the variant aspects. For example, when we define the tcnative dimension with 2 variants on and off, we’re modeling the fact that there are 2 possible outcomes for this dimension. In practice, enabling tcnative is just a matter of adding a single dependency at runtime:

io.micronaut.testcase.tcnative.on.gradle.kts

dependencies {
    runtimeOnly("io.netty:netty-tcnative-boringssl-static::linux-x86_64")
}

The dimension which handles the version of Java (both to compile and run the application) makes use of Gradle’s toolchain support:

io.micronaut.testcase.java.17.gradle.kts

java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(17))
    }
}

This can be done in a convention plugin which is named against the dimension variant name: io.micronaut.testcase.tcnative.on. In other words, the project with path :test-case:tcnative-off:epoll-off:json-jackson:micronaut-3.8:java-11 will have a "synthetic" build script which only consists of applying the following plugins:

plugins {
    id("io.micronaut.testcase")               (1)
    id("io.micronaut.testcase.tcnative.off")  (2)
    id("io.micronaut.testcase.epoll.off")     (3)
    id("io.micronaut.testcase.json.jackson")  (4)
    id("io.micronaut.testcase.micronaut.3.8") (5)
    id("io.micronaut.testcase.java.11")       (6)
}

1	Applies the common configuration
2	Configures `tcnative` off
3	Configures `epoll` off
4	Configures Jackson as the serialization framework
5	Configures Micronaut 3.8
6	Configures build for Java 11

Each of these plugins can be found in our build logic. As you can see when browsing the build logic directory, there is actually one small optimization: it is not necessary to create a variant script if there’s nothign to do. For example, in practice, tcnative off doesn’t need any extra configuration, so there’s no need to write a io.micronaut.testcase.tcnative.off plugin which would be empty in any case.

Variant specific code

The best case would have been that we only have to tweak the build process (for example to add dependencies, disable native image building, etc), but in some cases, we have to change the actual sources or resource files. Again, we leveraged Gradle’s flexibility to define custom conventions in our project layout. In a traditional Gradle (or Maven) project, the main sources are found in src/main/java. This is the case here, but we also support adding source directories based on the variants. For example in this project, some DTOs will make use of Java records on Java 17, but those are not available in Java 11, so we need to write 2 variants of the same classes: one with records, the other one with good old Java beans. This can be done by putting the Java 11 sources under src/main/variants/java-11/java, and their equivalent Java 17 sources under src/main/variants/java-17/java. This is actually generic: you can use any variant name in place of java-11: we could, for example, have a source directory for the epoll-on folder. The same behavior is available for resources (in src/main/variants/java-11/resources).

This provides very good flexibility while being totally understandable and conventional.

The settings plugin

So far, we explained how a user interacts with this build, for example by adding a dimension and a variant or adding specific sources, but we didn’t explain how the projects are actually generated. For this purpose, we have to explain that Gradle supports multiple types of plugins. The typical plugins, which we have used so far in this blog post, the io.micronaut.testcase.xxx plugins, are project plugins, because they apply on the Project of a Gradle build. There are other types of plugins, and the other one which we’re interested in here is the settings plugin. Unlike project plugins, these plugins are applied on the Settings object, that is to say thay they would be typically applied on the settings.gradle(.kts) file. This is what we have in this project:

settings.gradle.kts

// ...

plugins {
    id("io.micronaut.bench.variants")
}


include("load-generator-gatling")

configure<io.micronaut.bench.AppVariants> {
    combinations {
        dimension("tcnative") {
            variant("off")
            variant("on")
        }
        dimension("epoll") {
            variant("off")
            variant("on")
        }
        dimension("json") {
            variant("jackson")
            //variant("serde")
        }
        dimension("micronaut") {
            variant("3.8")
            //variant("4.0")
        }
        dimension("java") {
            //variant("11")
            variant("17")
        }
        exclude {
            // Combination of Micronaut 4 and Java 11 is invalid
            it.contains("micronaut-4.0") && it.contains("java-11")
        }
    }
}

The io.micronaut.bench.variants is another convention plugin defined in our build logic. It doesn’t do much, except for creating an extension, which is what lets us configure the variants:

import io.micronaut.bench.AppVariants

val variants = extensions.create<AppVariants>("benchmarkVariants", settings)

The logic actually happens within that AppVariants class, for which you can find the sources here. This class handles both the variants extension DSL and the logic to generate the projects.

The entry point is the combinations method which takes a configuration block. Each of the call to dimension registers a new dimension, which is itself configured via a variant configuration block, where each individual variant is declared. When we return from this call, we have built a model of dimension of variants, for which we need to compute the cartesian product.

We can check each of the entry that we have generated against the excludes, and if the combination is valid, we can use the Gradle APIs which are available in settings script to generate our synthetic projects.

For example:

val projectPath = ":test-case:${path.replace('/', ':')}"
settings.include(projectPath)

computes the project path (with colons) and includes it, which is equivalent to writing this manually in the settings.gradle file:

include(":test-case:tcnative-off:epoll-off:json-jackson:micronaut-3.8:java-11")
include(":test-case:tcnative-off:epoll-off:json-jackson:micronaut-3.8:java-17")
include(":test-case:tcnative-off:epoll-off:json-jackson:micronaut-4.0:java-17")

If we stopped here, then we would have defined projects, but Gradle would expect the sources and build scripts for these projects to be found in test-case/tcnative-off/epoll-off/json-jackson/micronaut-3.8/java-11. This isn’t the case for us, since all projects will share the same project directory (test-case-common). However, if we configure all the projects to use the same directory, then things could go wrong at build time, in particular because we use parallel builds: all the projects would write their outputs in the same build directory, but as we have seen, they may have different sources, different dependencies, etc. So we need to set both the project directory to the common directory, but also change the build directory to a per-project specific directory. This way we make sure to reuse the same sources without having to copy everything manually, but we also make sure that up-to-date checking, build caching and parallel builds work perfectly fine:

settings.project(projectPath).setProjectDir(File(settings.rootDir, "test-case-common"))
gradle.beforeProject {
    if (this.path == projectPath) {
        setBuildDir(File(projectDir, "build/${path}"))
    }
}

Note that we have to use the gradle.beforeProject API for this: it basically provides us with the naked Project instance of our synthetic projects, before its configuration phase is triggered.

The next step is to make sure that once the java plugin is applied on a project, we configure the additional source directories for each dimension. This is done via the withPlugin API which lets use react on the application of a plugin, and the SourceSet API:

project.plugins.withId("java") {
    project.extensions.findByType(JavaPluginExtension::class.java)?.let { java ->
        variantNames.forEach { variantName ->
            java.sourceSets.all {
                this.java.srcDir("src/$name/variants/$variantName/java")
                this.resources.srcDir("src/$name/variants/$variantName/resources")
            }
        }
    }
}

Last, we need to apply our convention plugins, the plugins which correspond to a specific combination variant, to our synthetic project:

gradle.afterProject {
    if (this.path == projectPath) {
        variantSpecs.forEach {
            val pluginId = "io.micronaut.testcase.${it.dimensionName}.${it.name}"
            val plugin = File(settings.settingsDir, "build-logic/src/main/kotlin/$pluginId.gradle.kts")
            if (plugin.exists()) {
                plugins.apply(pluginId)
            }
        }
    }
}

As you can see, for each variant, we basically compute the name of the plugin to apply, and if a corresponding file exists, we simply apply the plugin, that’s it!

It only takes around 100 lines of code to implement both the DSL and logic to generate all this, which is all the power Gradle gives us!

Limitations

Of course, there are limitations to this approach. While we could handle the Java version easily, we can’t, however, add a dimension we would have needed : GraalVM CE vs GraalVM EE. This is a limitation of Gradle’s toolchain support, which cannot make a difference between those 2 toolchains.

Another limitation is that this works well for a single project build, or a project like here where there’s a common application, a support library, but all modifications happen in a single project (the application). Supporting multi-project builds and variants per module would be possible in theory, but would add quite a lot of complexity.

It was also lucky that I could support both Micronaut 3 and Micronaut 4: in practice, the Gradle plugin for Micronaut 4 isn’t compatible with Micronaut 3, so I would have to either use Micronaut 3 or Micronaut 4. However, we can use the Micronaut 4 plugin with Micronaut 3, provided some small tweaks.

Last, there is one unknown to this, which is that building synthetic projects like that makes use of APIs which are stable in Gradle, but likely to be deprecated in the future (event based APIs).

Alternatives

Before going to the "final" solution, I have actually tried a few things (which could be spiked in a couple hours or so). In particular, the first thing I did was actually to use a single project, but configure additional artifacts (e.g jar and native binary) for each variant. While I could make it work, the implementation turned out to be more complicated, because you have to understand how each of the plugins work (Micronaut, GraalVM, the Shadow plugin) and create exotic tasks to make things work. Also this had a number of drawbacks:

impossible to build variants in parallel (at least without the experimental configuration cache)
configuring each of the variant specific build configuration (e.g adding dependencies) was more complicated. It was in particular only possible to add additional runtime dependencies. If something else was needed, for example compile time dependencies or additional resources, this wasn’t possible to do because a single main jar was produced.

Conclusion

In this blog post, we have seen how we can leverage Gradle’s flexibility to support what seemed to be a complicated use case: given a common codebase and some "small tweaks", generate a matrix of builds which are used to build different artifacts, in order to benchmark them.

The solution turned out to be quite simple to implement, and I hope pretty elegant, both in terms of user facing features (adding dimensions and configuring the build should be easy), maintenance (composition over inheritance makes it very simple to understand how things are combined) and implementation.

Many thanks to Jonas Konrad for the feature requests and for reviewing this blog post!

Petit voyage en électrique

Fri, 20 Jan 2023 00:00:00 +0100

Hier, je me déplaçais sur Tours pour la conférence Touraine Tech, où j’ai donné un talk sur Micronaut Test Resources. Je remercie encore l’organisation d’avoir accepté ce talk, qui, d’après les commentaires que j’ai reçu, a plutôt été bien reçu ! Mais ça n’est pas le sujet de ce billet : je souhaite simplement vous parler de mon expérience avec ma voiture électrique, que j’ai utilisé pour me rendre à la conférence.

Le déplacement

Tours, ça n’est pas si loin de chez moi, environ 200km. J’avais donc décidé de m’y rendre avec mon e-208, dont l’autonomie théorique, avec sa batterie de 50kW (disponible 46kW), est annoncée à 340km. J’ai fais l’acquisition de cette voiture il y a 2 ans, et j’en suis globalement très content : j’habite en zone rurale, nous n’avons pas de transports en commun, et cette voiture sert donc pour tous les trajets du quotidien. Je peux la recharger à la maison sans problème. Jusqu’ici, les trajets les plus longs que j’avais effectué étaient sans recharge : des allez-retours à Pornic, où j’ai de la famille, soit environ 160 km aller/retour, et ça se passait très bien, en particulier l’été.

Maintenant, entre l’autonomie théorique et la réalité, il y a un monde, en particulier en hiver. J’étais donc assez nerveux à l’idée de me retrouver "en rade" avant d’arriver sur Tours, et j’ai donc planifié mon déplacement avec l’application ChargeMap (j’ai une carte chez eux et l’application Peugeot est franchement pas top, impossible de planifier aussi bien).

Je voulais faire l’aller-retour dans la journée, ce qui impliquait de pouvoir recharger en arrivant sur Tours. Un des problèmes, c’est que les bornes de recharge "rapides" ne sont pas si nombreuses. Autre problème : il est impossible de savoir si une borne va être occupée lorsqu’on y arrivera. La 208 dispose d’une prise combo CCS qui accepte une charge à 100kW.

J’avais donc 2 choix:

m’arrêter à une charge rapide (50kW et +) avant de me rendre à la conférence
ou déposer ma voiture sur une borne lente à proximité de la conférence et revenir plus tard dans la journée pour libérer la borne

J’ai choisi la première option, parce que j’avais un doute que la borne soit occupée en arrivant, et que je doive donc faire 10 min de route de plus pour me rendre à la borne rapide et donc perdre du temps. Par ailleurs, ça n’est pas super pratique que de devoir quitter la conférence et marcher 1km (potentiellement sous la pluie) dans la journée.

En bref, j’ai planifié pour être tranquille. Voici les conditions du trajet:

départ 5h32, tout le trajet en mode éco
je suis parti avec une batterie chargée à 100% (je sais qu’il faut éviter, mais d’une, je n’allais pas risquer de devoir m’arrêter sur une borne lente en cours de trajet, je ne souhaitais pas me retrouver à moins de 10% de batterie à l’arrivée, trop stressant, et d’autre part, le logiciel Peugeot ne permet pas d’interrompre une charge lorsque la batterie atteint une certaine limite, par exemple 80% !)
j’ai choisi un itinéraire sans autoroute
j’ai roulé à 80km sur les départementales (y compris celles limitées à 90km/h, en Maine et Loire), entre 90 et 100km/h sur les nationales
je roule en conduite souple : pas d’accélérations brutales, utilisation du mode B pour freiner, etc…
la température extérieure oscillait entre 0 et 3 degrés, chauffage réglé à 18

L’application ChargeMap dispose d’une fonctionnalité qui lui permet d’envoyer le trajet planifié sur Google Maps, que j’ai utilisé pour le guidage. Ça se passait très bien, jusqu’à ce que j’arrive après Saumur où je me rends compte que le GPS avait décidé de me faire prendre l’autoroute ! Problème, je n’avais clairement pas assez d’autonomie pour rouler à 130 km/h. Ne pouvant pas rouler à 90 km/h sur autoroute, trop dangereux, je suis donc monté à 110 km/h et autant dire que vu les conditions météo (froid), mon autonomie restante fondait comme neige au soleil. Je suis donc sorti un peu plus loin pour finir le trajet en passant par les bords de Loire, comme c’était initialement prévu.

Au final, je suis arrivé sur ma borne de recharge Allego, au Casino de La Riche, à 8h08 : 180km en 2h36. De mémoire (l’appli Peugeot récupère les trajets, mais pas la consommation, incroyable ce retard du logiciel par rapport à la concurrence !), ma consommation moyenne était de l’ordre de 16kWh/100km. Je me suis branché et j’ai chargé pendant 49 minutes pour atteindre 90% de batterie, soit 29kW, facture: 31,38€, pas franchement économique (1.082€ du kWh !). Je me suis arrêté à 90% parce que la recharge "ralentit" à mesure qu’on s’approche de la charge maximale : il m’aurait fallu rester encore une bonne demi-heure (voire plus) pour atteindre les 100%, et je souhaitais me rendre à la conférence.

Je suis donc arrivé sur Polytech’Tours à 9h12, soit 3h40, à comparer aux 2h25 si j’étais parti avec ma 407 diesel, qui ferait l’aller-retour sans aucun pb sans faire le plein (autonomie environ 950km…).

Pour le retour, je savais donc que je serai très juste et qu’il faudrait probablement que je fasse un arrêt supplémentaire pour recharger (à cause des 10% de batterie en moins au départ). Je ne suis pas passé par l’autoroute au retour, et donc suivi les bords de Loire. Les conditions météo étaient similaires, mais avec plus de pluie. J’ai surveillé mon autonomie, et si au départ, j’avais une marge de 100km entre l’autonomie annoncée par la voiture (c’est à dire qu’en suivant ses indications, je serais à la maison avec 100km d’autonomie restante), au fur et à mesure du trajet, cette estimation a sensiblement baissé. Arrivé à Cholet (environ 40km de chez moi), il ne restait plus que 60 km de marge, alors que j’avais baissé la température dans l’habitacle à 16 degrés. Encore une fois, je roulais en mode éco, souple, pas de bouchons, rien. En clair, l’estimation d’autonomie, c’est du grand n’importe quoi et complètement irréaliste (à noter, qu’en été, c’est bien plus proche de la réalité).

Bref, j’avais aussi faim et n’étant pas très joueur, je me suis arrêté sur une borne rapide en chemin, à côté d’une pizzeria, au SIEML de l’Ecuyère. Comme je savais que quel que soit le temps de charge, j’aurais de quoi rentrer large, j’ai juste pris le temps de manger. Je récupère ma voiture, pas mal, 22,7kW de récupérés en 35 minutes de charge, pour 9.62€ : 3 fois moins cher que la recharge à Tours (mais à comparer aux 0.14€/kWh quand je charge à la maison…).

Conclusion

L’expérience fut concluante : je sais que je peux faire ce genre de trajets, moyennant quelques concessions (heure d’arrivée tardive à cause de la recharge à l’arrivée, trajet sans autoroute, confort "limité", etc), mais c’est à peu près la distance maximale que je puisse faire sans que ça ne devienne trop pénible. En revanche, je reste très mitigé sur l’autonomie "réelle" : ici, j’étais plus proche des 200 km en faisant tout pour économiser. Même en conditions idéales, jamais, ô grand jamais, je n’atteindrais les 340 km (le mieux, c’est environ 300km). Le logiciel est aussi bien trop basique comparé à la concurrence (oui, Tesla) : l’application mobile manque de fonctionnalités de base (blocage de charge, récupération des consommations, …) et le GPS pour planifier un trajet est franchement nul. L’estimation de l’autonomie restante est irréaliste et pire, on ne sait pas vraiment où on en est de charge: l’indicateur à la "jauge d’essence" n’est pas adapté à une batterie (dites moi le % restant, c’est plus parlant !). Enfin, un des gros soucis reste la recharge et les tarifs hallucinants qui sont pratiqués : il est pour ainsi dire impossible de savoir combien va vous coûter un trajet, puisqu’en fonction des conditions, vous allez devoir vous arrêter, ou non, et que les tarifs varient en fonction de la puissance de recharge, du fournisseur, etc.

Lorsqu’on part en thermique, on sait que le carburant coûte environ 1.9€/L, à 25% près : en électrique, oubliez. Vous pouvez faire du simple au quadruple. Est-ce à dire que je ne recommande pas l’électrique ? Pas du tout ! Déjà, je préfère 100 fois le confort de la conduite en électrique au thermique. La voiture est aussi super agréable à conduire et la puissance disponible immédiatement est un indéniable atout de l’électrique.

Mon alternative, sur ce trajet, aurait été de prendre ma voiture thermique, mais ça aurait été la solution de facilité. Et compte-tenu de l’urgence climatique, j’ai fais le choix de perdre un peu de confort, pour le bien de ma conscience :)

Un an d’astrophotographie

Sat, 31 Dec 2022 00:00:00 +0100

Note	An english version of this blog post is available here

Et voilà, 2022, c’est terminé ! Je poste régulièrement des images de l’univers que je prends depuis mon jardin, mais il est possible que vous en ayez raté quelques unes (je les postais sur Twitter, mais désormais, vous pouvez me suivre sur un compte astro dédié sur Mastodon), c’est donc l’opportunité de faire une rétrospective !

Prendre de belles photo du ciel demande un peu d’effort : bien entendu, il faut du matériel (télescope, caméras, …), de bonnes conditions (météo, phase de la lune, …) mais il faut aussi de l’expérience dans le traitement logiciel. Les photos que je prends ces temps-ci sont déja bien meilleures que celles que je prenais il y a quelques années. Néanmoins, j’y vois de nombreux défauts, du bruit, des problèmes de focus, etc. Il y a donc encore beaucoup de choses à améliorer.

Deux sessions Twitch

Cette année, j’ai fait deux sessions lives sur Twitch, où j’ai expliqué de mon point de vue les bases de l’astrophotographie. Il s’agit de deux sessions d’environ 90 minutes chacune, où je donne des informations sur le matériel, mon setup, mais aussi le traitement logiciel. Je les ai faites pour répondre à des questions qu’on me pose souvent, comme:

quel télescope choisir ?
combien de temps de pose faut-il ?
combien ça coûte ?
comment fais-tu pour prendre une photo ?
etc.

Si vous avez un peu de temps, voici les liens:

2022: Les photos

Je vais les lister par ordre chronologique.

Note	Les photos que je poste ici sont protégées par le droit d’auteur: vous ne pouvez ni les recopier, ni les revendre, sans ma permission.

De temps en temps, il peut s’écouler des semaines entre les photos, auquel cas il s’agit très certainement de problèmes météo : dès que j’en ai l’opportunité, vous pouvez être certains que je sors le matériel ! Cependant, il peut arriver qu’il s’agisse aussi d’un échec : paramètres de capture incorrects, mauvais focus, mauvaises poses de flats, … Dans ce cas, il n’y a guère d’autre choix que de mettre tout le travail à la poubelle…

Pour chacune des photos que je vous propose ici, il s’agit d’une version basse résolution. Pour voir les versions hautes résolution, cliquez sur la photo, ce qui vous amènera vers sa version sur mon compte Astrobin. Vous aurez alors accès:

à tous les détails d’acquisition (quand, temps de pose, matériel utilisé, filtres, etc)
à une version "full" en haute résolution.

La plupart de mes photos font 62 megapixels. Parfois moins, parfois plus, mais il s’agit grossièrement de 7 fois plus de pixels que sur votre télé 4K ! Lorsque vous arriverez sur Astrobin, vous tomberez sur une version "annotée", mais basse résolution de la photo. Cliquez dessus et vous aurez alors accès à la version "full", qui est considérablement plus détaillée.

C’est parti !

Janvier

Nous sommes en période de fêtes, alors commençons par NGC 2264, aussi connu sous le nom de "l’arbre de Noël". Il s’agit d’un amas ouvert à environ 2300 années lumières, dans la constellation de la Licorne, qui se trouve accompagné d’une magnifique nébuleuse appelée la nébuleuse du cône.

Pendant l’hiver, les nuits sont relativement longues, ce qui donne souvent l’opportunité de travailler plusieurs sujets en une nuit. J’en ai donc profité pour capturer une autre cible, NGC 4565, la "galaxie de l’aiguille", un classique. Je n’ai pas pris suffisamment de données pour bien faire ressortir les couleurs, néamnoins le résultat est intéressant:

Février

A la fin de l’hiver, presque un mois plus tard, j’ai choisi une cible bien connue des astrophotographes : NGC 1893. Cette photo a été assez difficile à prendre et à traiter, à cause de la présence d’étoiles très brillantes dans le champ, ce qui, avec mon matériel, crée des reflets qu’il faut supprimer en traitement. Ce traitement pourrait même être améliorer pour réduire un peu plus les étoiles et faire ressortir la nébuleuse plus clairement.

Nous débutons alors la "saison des galaxies", qui dure tout le printemps. La prochaine image est un autre classique, le couple de galaxies M81 et M82. Je ne suis pas complètement satisfait du traitement de cette photo, notamment au niveau des couleurs, mais il s’agissait pour moi de faire ressortir ce qu’on appelle l’IFN, les Nébuleuses de flux intégré, qui ne sont ni plus ni moins que de très larges nuages de gaz "inerte" qui s’étendent sur de très grandes distances. C’est particulièrement visible dans cette photo.

Mars

La prochaine photo est un autre classique du printemps, connu sur le nom de "la chaîne de Markarian". Bien que le grand public connaisse l’existence des galaxies et parfois des amas d’étoiles (notre soleil fait partie de la galaxie de la Voie Lactée), peu de personnes savent que les galaxies elle-mêmes s’organisent en "groupes locaux" puis sous forme de structures plus larges s’étendant sur des milliers d’années lumières. Cette photographie a le mérite de montrer ces groupes de façon évidente : sur une seule photographie, nous pouvons voir des dizaines de galaxies, parfois en intéraction directe (elles tournent les unes autour des autres et pourront fusionner), parfois simplement en arrière plan.

Mai

Le mois d’avril ne fut pas un bon mois, sans la moindre photo ! Alors quand Mai fut venu et le beau temps de retour, j’en ai profité et pris plein les mirettes !

La saison des galaxies n’est pas complètement terminée en Mai. J’ai choisi de capturer une galaxie dont la forme rappelle celle de la galaxie de l’aiguille: NGC 5746. Encore une fois sur cette photo, le traitement a été difficile à cause de la présence de cette étoile très brillante.

Continuons avec une des photos les plus spectaculaires de cette année. Elle n’est pas particulièrement nette et il y a des efforts à faire sur le traitement, mais il s’agissait là d’une occasion à ne pas manquer : la photographie d’un événement rare, une supernova, c’est à dire une étoile qui meurt dans une explosion phénoménale, en émettant une quantité de lumière si impressionnante qu’elle surpasse celle du reste de la galaxie : une supernova dans une galaxie va atteindre une luminosité apparente aussi forte que des étoiles de notre propre galaxie, alors qu’elle peut se situer à des centaines de milliers d’années lumières de nous !

La photo est aussi spectaculaire au sens où j’a utilisé une focale longue, pour "zoomer" au plus près, mais que malgré celà, de nombreuses autres galaxies sont visibles en arrière plan dans le champ ! Voici donc la supernova 2022hfs, dans la galaxie NGC 4647.

Sur la photo, au centre, nous avons la grande galaxie M60, et dessous, son compagnon NCG 4647. Dans cette dernière, on distingue en haut un point brillant: il s’agit de la supernova !

Afin de changer des galaxies, j’ai aussi capturé en mai le magnifique amas globulaire nommé M5. Un amas globulaire est un ensemble de "vieilles" étoiles, regroupées dans un espace très compact. On trouve de nombreux amas globulaires au sein de notre galaxie.

J’ai une fascination pour les "nébuleuses sombres". Ce sont des nébuleuses de différents types, qui présentent des couleurs sombres pour différentes raisons: poussière interstellaire, qui bloque la lumière, ou plus simplement des régions du ciel dénuées des gas qui émettent typiquement dans le rouge. Faire ressortir ces nébuleuses sombre requiert en général beaucoup de temps de pose. En mai, ça reste possible, mais c’est difficile, et il faut même parfois cumuler les données de plusieurs nuits.

J’ai pris deux photos de telles nébuleuses en Mai. La première montre B347 et LDN 889, dans la constellation du Cygne. Il s’agit d’une nébuleuse en émission (elle émet sa propre lumière) qui montre une région plus sombre en son centre:

La seconde est une magnifique structure complexe, mélange de gaz bloquants la lumière, de nébuleuses à réflexion (qui reflètent la lumière des étoiles) et à émission : IC 4603.

En règle générale, j’évite la lune : je préfère photographier le ciel profond (galaxies, nébuleuses, …), ce qui requiert un ciel sombre, sans lune et sans pollution lumineuse. Cependant, je m’accorde de temps en temps une pause, ce que j’ai fait le 10 mai, en capturant notre voisine:

Juin

En juin débute la saison des nébuleuses et en particulier des nébuleuses à émission (dans le rouge). Pendant quelques mois, nous allons pouvoir capturer beaucoup d’entre elles : ceci s’explique par la position de la voie lactée, qui en été se trouve au zénith au-dessus de nos têtes. Nous observons alors vers le centre de notre galaxie, où la densité d’étoiles et de nuages de gas est la plus forte.

Un des classique, c’est la nébuleuse de l’Aigle, que j’avais déja capturée. Je me suis donc lancé le défi de réaliser une mosaïque qui couvrirait une région assez large du ciel, de M16 à M24. La photo qui en résulte ne fait pas moins de 7360 pixels de large, pour 13200 de haut !

La photo suivante est ma "favorite" en 2022. En comparaison à d’autres photos, elle ne m’a pas demandé beaucoup de temps de pose : seulement une heure, mais elle montre une des plus belles régions du ciel.

J’étais particulièrement envieux de prendre cette photo, parce qu’il est assez difficile de regrouper les conditions nécessaires là où j’habite : entre la météo et le fait que cette cible ne monte pas très haut dans le ciel, le timing était serré et je l’ai ratée plusieurs années de suite.

Trèves de bavardages, il s’agit donc des nébuleuses de la Trifide et du Lagon, qui, comme je l’ai dit, sont pour moi une des plus belles régions du ciel avec un mélange subtil de couleurs bleutées (nébuleuse en émission), rosée et rouge, avec de nombreux détails visibles dans les nébulosités.

Lorce qu’on est astronome amateur, un des premiers objets du ciel profond que l’on essaie de voir l’été en Europe est la nébuleuse de la Lyre, M57. Il s’agit d’une nébuleuse planétaire (nommée ansi parce que les premiers observateurs les ont confondu avec des planètes) assez petite, mais très lumineuse, visible avec des instruments très modestes. La prendre en photo et faire ressortir des détails, en revanche, est une autre paire de manches. Pour cette photo, j’ai utilisé une technique inhabituelle, réservée d’habitude aux planètes : capturer de très nombreuses poses très courtes et "assembler" le tout. Voici le résultat:

Le 28 juin est une date spéciale, puisqu’il s’agit de mon anniversaire de mariage (20 ans en 2023 !). En 2022, le 28 juin me fit un beau cadeau avec de nombreuses photos.

La première est la nébuleuse du croissant, dans le Cygne: il s’agit d’une étoile qui a explosé et dont l’enveloppe gazeuse s’étend au milieu d’une gigantesque milieu gazeux rouge où naîtront de nouvelles étoiles.

La 2ème est un autre classique des astrophotographes : la nébuleuse de la trompe d’éléphant (Tr-37). Je vous ai dis que j’aimais les nébuleuses sombres, cette photo montre de superbes structures sombres au sein d’une nébuleuse brillante, le contraste est saisissant !

Un autre exemple d’une telle nébuleuse est NGC 7822, capturée la même nuit :

Enfin, pour conclure le mois de juin, il s’agit de vous montrer ce que l’on finit par voir après quelques milliers d’années, lorsqu’une supernova relativement proche explose. Le ciel d’été dispose d’un magnifique exemple dans la constellation du Cygne : la nébuleuse des "dentelles du Cygne", nommée ainsi en référence à ses magnifiques structures en filaments, de véritables "dentelles".

Juillet

Le mois de juillet est la période idéale pour capturer notre voisine la plus proche, la galaxie d’Andromède. Légèrement plus grande que notre propre galaxie, la Voie Lactée, nos deux comparses finiront par fusionner dans quelques millions d’années. Andromède est visible à l’oeil nu sous un bon ciel sans pollution lumineuse. Sa taille apparente est même supérieuse à celle de la Lune !

Il y a de nombreuses comètes dans notre système solaire, et de temps en temps, une d’entre elles devient suffisamment brillante pour qu’on puisse la capturer. C’est arrivé pour moi au début du mois de Juillet, quand la comète C2017 K2 Panstarrs est devenue visible :

(pour une raison inconnue je n’ai pas posté cette photo sur Astrobin).

Ma cible suivante est un nouvel exemple de "nébuleuse sombre", cette fois ci sur une nébuleuse qui l’est de part des nuages de poussières interstellaires, qui empêchent la lumière des étoiles en fond de nous parvenir. Celle-ci est connue sous le nom de la nébuleuse de l’hippocampe. Mon champ est cependant suffisamment large pour montrer, dans la même photo, 2 autres objets très intéressants: l’amas ouvert NGC 6939, ainsi que la magnifique galaxie spirale NGC 6946, vue du dessus.

Pour ma prochaine cible, j’ai choisi un objet plus difficile à prendre en photo : une nébuleuse à émission diffuse, connue sous le nom de Shapless 115 (du catalogue de nébuleuses du même nom). Prendre en photo de telles nébuleuses requiert l’utilisation de filtres en bande étroite, en particulier dans les endroits soumis à la pollution lumineuse, ce qui est mon cas (bien qu’elle soit bien plus raisonnable qu’à Nantes, 45 km plus au nord). Elle requiert aussi des temps de pose plus longs (ici, 4 heures) :

Le même jour, j’ai essayé de prendre en photo la "nébuleuse du sorcier", mais je fus assez déçu du résultat, que je n’ai donc pas posté sur Astrobin. Mais puisqu’il s’agit d’une rétrospective, je vous la pose ici, sachant que je réessairai certainement cette cible en 2023:

Août

La prochaine photographie est une autre de mes favories en 2022, et une autre nébuleuse sombre. Il s’agit de la "nébuleuse du requin", j’espère que vous voyez pourquoi :

Vous ne voyez pas le requin ? Essayez avec cette version "sans étoiles" :

Pour la prochaine photographie, vous reconnaîtrez un visage familier : Andromède, une nouvelle fois ! Pourquoi la prendre en photo plusieurs fois ? Eh bien parce bien que ce soit une cible très grande et très lumineuse, il s’agit aussi d’une des plus difficile à traiter. Il faut donc essayer plusieurs techniques, plusieurs temps de pose, etc pour obtenir un bon résultat. Alors, quelle version préférez vous ?

Nous voici déja à la fin de l’été, mais il restait une photographie que je ne pouvais pas me permettre de rater : Messier 33, la galaxie du triangle. Il s’agit d’une autre galaxie de notre groupe local, avec Andromède, qui est absolument magnifique avec ses couleurs bleutées. Si vous zoomez sur la photos, vous y découvrirez de nombreuses régions rougeâtres ou rosées: il s’agit des mêmes types de nébuleuses que celles que j’ai montré précédement, où naissent de nombreuses nouvelles étoiles !

Septembre

Le mois de Septembre marque le début de la "saison des pluies", qui peut durer jusque février. Il reste encore quelques bonnes opportunités en septembre avant que cette saison ne commence.

Pour la première photo, nous avons NGC 7822, dont je ne suis pas particulièrement fier du traitement, avec de nombreux gradients encore visibles…

La suivante est un autre "inmanquable", les nébuleuses de l’Amérique du Nord et du Pélican : cette photo est une mosaique de 2 images, n’hésitez donc pas à vous promener sur la version "full", qui révèle un nombre impressionnant de détails.

Pour la suivante, vous retrouverez encore une fois une figure familière : Messier 33. Mais cette fois-ci, j’ai utilisé une autre focale, plus longue, permettant de zoomer sur la galaxie qui prenait alors l’intégralité du capteur, c’était juste parfait ! Là encore je vous invite à vous balader sur la version "full", qui montre nombre de détails…

Novembre

Je vous avais dit qu’après Septembre, ça devenait compliqué… Entre fin septembre et fin novembre, pas la moindre photo ! Alors quand j’ai eu une petite fenêtre le 29, et peu de temps avant le retour de la pluie, j’ai choisi de partir sur une cible "facile" et très lumineuse, qui ne requiert donc pas beaucoup de temps de pose: la fameuse nébuleuse d’Orion !

La nuit suivante, j’ai eu une nouvelle chance, où j’ai choisi de cibler la très photogénique NGC 1333, une région du ciel qui montre dans un même champ des nébuleuses sombres, mais aussi de lumineuses nébuleuses à réflection, presque blanches, et une région rougeâtre où naissent de nombreuses étoiles formant un magnifique contraste avec la région sombre alentour !

Decembre

Pour terminer l’année, une photo qui n’a que quelques jours. J’ai dù attendre le lendemain de Noël pour pouvoir ressortir le matériel (et profiter de mes vacances). Les conditions étaient loin d’être parfaites : de nombreux passages nuageux, un ciel pas super transparent, et une humidité qui finit par tomber en brume. Cependant, cette cible me fait de l’oeil depuis de nombreuses années, et là encore, il s’agit d’une cible qu’on ne peut pas capturer facilement de part sa hauteur, très basse sur l’horizon. La "fenêtre de capture" est donc très courte, et j’aurais aimé disposer de plus de temps de pose pour faire ressortir des couleurs du "casque de Thor" que voici:

Et nous y voilà ! J’aurais bien sûr aimé en faire plus. Mais entre les pleines lunes, les conditions météo et n’oublions pas mon travail, c’est toujours un peu compliqué. Il y a de très nombreux objets que je souhaite prendre en photo, et que j’ai soit raté, soit pas eu l’occasion ou le temps de faire lorsqu’ils étaient visibles. Qu’à celà ne tienne, c’est aussi pour ça que l’astrophotographie, c’est une passion d’une vie !

One year of astrophotography

Sat, 31 Dec 2022 00:00:00 +0100

Note	Une version en français est disponible ici !

This is it, 2022 is coming to an end! I am regularly posting pictures of the universe I take from my backyards, but you may have missed some of them (note: I used to post them on Twitter, but now that I moved to Mastodon, I have a dedicated account for this), so this is the opportunity to recap!

Making great photos of the sky takes some effort: of course you need some gear (telescope, camera, …), you need good conditions (weather, moon, …) but you also need skills in software post-processing. The photos I take those days are already much better than the ones I used to take a few years ago, but I see a lot of defects, noise, focusing problems, etc. so there’s still a LOT of room for improvement.

A couple of Twitch sessions (in french)

This year, for the french speaking, I have done a couple of sessions on Twitch where I explained the basics of astrophotography, from my point of view. It’s a couple of ~90 minutes videos each, where I give some details about my setup, the gear, and software processing. I’ve done this because I’m always getting the same questions:

what telescope to choose?
how long does it take?
how much does it cost?
how do you proceed?
etc.

If you are interested (and french speaking), here are the links:

Let me know if you are interested in seeing something similar in English.

2022: The photos

I’m going to list the photos in chronological order.

Note	The photos I’m posting here are copyrighted, you may not use them or sell them without my permission.

Sometimes, there are weeks between photos, and that’s almost certainly because of the weather: when I have the opportunity, you can be sure I always take the setup out. However, sometimes, it’s also because I failed: you capture data, but you made a mistake (e.g, incorrect focus, not enough exposure, wrong flat frames, …), and you have to drop everything…

For each of the photos I’m posting here, it’s a low resolution version. You can click on each of the photos to get a link to their version on Astrobin, which will also give you:

acquisition details (when, how many exposures, filters, …)
a full resolution version : most of the photos are 62-megapixel (sometimes less because of cropping, sometimes more because of mosaics), which is roughly 7 times as many pixels as what you have in a 4K UHD TV!

Note that when you’re on Astrobin, you can click on the photo to go to the full version, which has considerably higher resolution than the preview.

January

It’s the holiday season, so let’s get started with NGC 2264 also known as the "Christmas Tree Cluster": it’s an open star cluster about 2300 light years away, located in the Monoceros constellation, which also shows a beautiful nebula called the "Cone nebula".

During winter nights are quite long, which also gave me the opportunity to shot another target, NGC 4565 aka the Needle Galaxy, a classic. I didn’t get enough data to properly capture the colors, but it’s still an interesting shot!

Februrary

Late winter, more than a month after the previous photos, I went for a target which is well known of astrophotographers: NGC 1893. It was an image which proved difficult to make and process, because of the presence of the bright stars which created colorful reflections on my very fast setup. Processing can probably also be improved to reduce the stars and show the nebula better.

This opens the "galaxy season", which is lasting almost all spring. The next image is a "classic", the M81 and M82 couple, but I’m not super happy with the processing and resulting colors. This image has lots of details, and especially, I wanted to highlight the Integrated Flux Nebula which is clearly visible in this sector of the sky.

March

The next photo is another classic of the galaxy season, known as Markarian’s chain. It is lesser known to the public that while stars are grouped in clusters and galaxies, galaxies themselves are organized in larger, extremely wide structures called local groups or chains. Those galaxies are bound together by gravity, and Markarian’s chain is a spectacular example of this, with a small area of the sky showing dozens of galaxies in interaction.

April wasn’t a good period, with no photo whatsoever, unfortunately, but May was the most prolific of period of the year!

May

Galaxy season isn’t over in May. I chose to image another galaxy which looks similar to the needle galaxy: NGC 5746. Yet, again, processing proved to be difficult because of the very bright star below it.

Galaxy season continues with probably one of the most interesting photos of this year. It isn’t particularly sharp, and there’s a lot of progress to be made on processing this image, but it was an opportunity I couldn’t miss: taking a picture of a Supernova, that is to say a star which dies by exploding and emitting so much light that its brightness literally makes it stand out of the galaxy hosting it. The photo is also spectacular in the sense that it’s taken with a long focal length, in order to "zoom" as much as possible, but still shows a lot of other galaxies in this small area of the sky! Here is Supernova 2022hfs in NGC 4647: in the middle we can see the M60 galaxy and its companion NGC 4647. In that galaxy, at the upper top, we can see a bright spot which is the supernova:

Another classic of the dark skies is the beautiful M5 globular cluster:

I have a fascination for "dark nebulas", which are nebulas which show dark structures, usually because of the presence of dust which blocks light, or simply because some large gas structures are not uniform. Usually taking photos of such structures requires time, long exposures, which is still possible during that period of the year, but becoming complicated.

I took a couple pictures of such nebulas in May.

First one is B347 and LDN 889 in Cygnus which is showing an emission nebula with a nice dark area in its center:

The second one is a beautiful complex structure showing dust blocking light and creating uncanny colored structures: IC 4603.

Usually, I’m avoiding the moon: I prefer the beauties of deep sky objects which require very dark skies and no light pollution. However, from time to time, I like to give the moon a shot, which is what I did on May 10th:

June

June is the beginning of the "emission nebula" season. There are a lot of good targets that we can start shooting during summer. One of the classics is the Eagle nebula, which I had already shot. So I wanted to capture this area in wider field. This is why I started with a mosaic which is extremely detailed if you go to the full picture below: the image is 7360 pixels wide, for 13200 pixels height! The area covers the region from M16 to M24:

The next photo is probably my favorite of 2022. In comparison to other photos which literally requires hours of exposure, this one "only" took an hour, but it shows one of the most beautiful areas of the sky. I was particularly eager to do it, because it’s difficult to have good conditions to capture it where I live: it’s relatively low in the sky, so there is unfortunately a very short period of time when the conditions are met to capture it, and when you do so, you can’t capture a lot of data. In any case, here are the Trifid and Lagoon nebulas, as I said, probably one of my favorites, showing beautiful colors and well visible gas structures:

When you are an amateur astronomer, one of the first targets you try to see (especially in European summer) is the ring nebula (M57) in Lyrae. It is a very bright nebula which is easily visible even with modest instruments. Taking a picture of it and getting details of its structure is harder. Here, I took an ususual approach, by going full focal length and taking lots of short exposures with my planetary camera and processing the image as if it was a planet. Here’s what I got:

June 28 is a special day for me as it’s my wedding’s day (it’s going to be 20 years in 2023!). June 28, 2022 was also a prolific day with a couple of beautiful nebulas.

The first one is the crescent nebula in Cygnus, which is an incredibly dense area of the sky, in the middle of the Milky Way, with lots of gas and stars:

The 2d photo I took that day was another classic of astrophotographers, the Elephant Trunk Nebula (Tr-37). I told you I loved dark nebulas, this one is a good example of dark structures in a bright nebula:

Another example of such a nebula is NGC 7822:

Last but not least of June, what happens after a star explodes as a supernova? The Veil Nebula, in Cygnus, is a great illustration of the result. This nebula is one of the most remarkable example of its kind, because it’s particularly large and shows very nice structures of filaments of matter.

July

Summer is the best period of time to take a photo of our nearest neighbor galaxy, the famous Andromeda galaxy. It is visible with the naked eye (if no moon is present) and larger than the apparent size of the moon in the sky.

Sometimes you get the opportunity to capture comets. This happened for me in the beginning of July, when comet C2017 K2 Panstarrs became bright enough to be shot:

(for some reason I didn’t put this one on Astrobin).

My next target was another example of "dark nebula", this time with a nebula which is dark because of dust blocking the light around it. This one is known as the Seahorse Nebula. Interestingly, the field I captured was wide enough to add a couple of interesting targets in the area: the open cluster NGC 6939 and the beautiful galaxy NGC 6946:

For the next one I chose a more difficult target, a diffuse emission nebula known as Sharpless 115. Such nebulas require filters to be captured properly under conditions like mine where there is light pollution (reasonable but still), and longer exposures (4 hours in this case):

The same day I captured the "Wizard Nebula" but wasn’t happy enough with the result, so I didn’t post it on Astrobin. Here’s the photo, though. I will definitly retry this target in 2023:

August

The next image is another of my 2022 favorites, and another dark nebula. This one is called the shark nebula, and I hope you’ll figure out why:

Can’t see the shark? Maybe it’s easier with this "starless" version?

The next image will be familiar, it’s the Andromeda galaxy, once again! Why capturing it multiple times? Well, as I said, the more data you gather, the better. Also processing is difficult, and you have to make a lot of trial to find the right exposure and number of frames to get a good result. Here’s my 2d attempt, which one do you prefer?

Summer is ending, but there was one target I couldn’t miss: another galaxy of our local group, Messier 33, the Triangulum galaxy. This spiral galaxy is particularly magnificient with its blue’ish colors. If you zoom in the photo, you’ll also notice red clouds, which are nebulas where new stars are regularly formed!

September

With the end of summer comes the rainy season, which lasts up to february. September gives us a few last opportunities. The first image, NGC 7822, I’m not super happy with, there are still gradients visible, but let me share it anyway, it’s a retro!

The next one is another classic of astrophoto, the North America and Pelican nebulas, which has the particularity of being a mosaic of 2 images, so if you go to the full version and zoom in, you will see a LOT of details:

The next one will again sound familiar: it’s the triangulum galaxy again, but with a longer focal length, which, in practice, means that it’s "zoomed in" compared to the previous version. I can only encourage you to dig into the full version, which shows an incredible amount of details, definitely another of my favorites of 2022!

November

I told you that the rainy season starts after September, so clearly, between october, and even till the end of November, I wasn’t able to capture a single target! I had a small window on November 29, not much time before rain, so I chose an "easy target", the famous Orion Nebula, which is so bright it doesn’t take long to capture:

The next day I was able to go to a target I wanted to shoot for a long time. NGC 1333 is an area of the sky which shows a variety of nebulas in the same field of view: dark nebulas, but also a light, blue’ish emission nebula, and a red area where new stars are born!

December

Finally, the last picture of this year is from a few days ago. I had to wait after Christmas to get another window, and I chose to try to capture the so-called "Thor’s Helmet". Unfortunately the weather conditions weren’t great, with lots of clouds and eventually fog, so I didn’t get as much data as I would like. This target is particularly difficult because it’s very low, so if I couldn’t capture it now, it would be too late after…

And it’s a wrap! I wish I could do much more photos, but between the moon, the weather conditions and… my daily job, it’s always a bit complicated. There are many objects I would have liked to capture, but either failed or missed the window when they are visible. This is why astrophotography is a lifetime hobby, after all!

Monitoring pour optimiser ma conso électrique

Sat, 26 Nov 2022 00:00:00 +0100

En juillet dernier, j’ai fait installer des panneaux solaires. Cette installation change considérablement nos habitudes de consommation: au lieu de consommer de préférence aux heures creuses, il est préférable de consommer lorsque la production locale le permet. Dans ce billet, je vous explique ce que j’ai mis en place pour nous aider dans l’optimisation de notre consommation, en particulier un outil de monitoring avec un Raspberry PI. L’objectif de mon projet est de réduire ma facture électrique en maximisant l’autoconsommation. Nous allons donc voir comment l’installation des panneaux peut changer nos habitudes.

Installation de panneaux solaires

Il y a longtemps que l’idée nous trottait dans la tête. En effet, ici, nous chauffions au fioul, ce qui avait un certain nombre d’inconvénients: pollution (fort émetteur de CO₂), prix très variable (entre 800€ et 1400€ les 1000L selon les saisons), bruit (la chaudière), odeurs de combustion, etc. Cette chaudière au fioul nous servait à la fois à chauffer l’eau chaude, mais aussi au chauffage domestique. La maison étant relativement bien isolée, nous arrivions cependant, à 4 personnes, à ne consommer qu’environ 1000L de fioul par an. Une vieille chaudière à fioul était tombée en panne en plein hiver, il y a 5 ans, nous avions dû la changer en urgence, et, je me souviens très bien de cela, il se trouve que j’étais à ce moment à l’étranger pour le "Gradle World Meeting", une semaine de travail avec l’ensemble de l’équipe. Nous n’avions donc pas eu le temps de faire différents devis, notamment de chauffage alternatif. Mais en Mars dernier, patatras, nouvelle panne. Cette fois-ci, la chaudière elle-même n’était pas en cause: c’est le système d’alimentation en fioul, entre la cuve et la chaudière, qui était encrassé. La chaudière se mettait constamment en défaut faute d’avoir une arrivée de fioul propre. Nous avions alors le choix entre faire nettoyer la cuve et la tuyauterie, pour une facture de l’ordre de 2k€, ou de changer.

En même temps, mon épouse et moi-même sommes tous les 2 en télétravail 5j/5. Notre consommation électrique est donc importante, entre l’alimentation des ordinateurs, des écrans, la cuisine le midi, le chauffage d’appoint dans mon bureau, … Certains de nos équipements consomment aussi régulièrement: n’ayant pas le tout à l’égout, par exemple, nous avons une microstation de traitement des eaux usées, avec une pompe de recirculation et une pompe de relevage, qui ont des consommations non négligeables à l’année. Nous possédons aussi une voiture électrique (une e-208) qui est chargée à domicile. Enfin, l’été, nous avons une pompe de circulation pour la piscine qui consomme beaucoup. Au final, notre facture électrique est donc beaucoup plus importante que celle de fioul.

Cette dernière panne a donc été l’occasion de revoir notre projet. Après divers devis, nous avons opté pour l’installation de 16 panneaux solaires de la société française DualSun : 10 panneaux classiques "Flash" et 6 panneaux hybrides électricité/eau chaude, pour une production totale de 6kWC (6 kW crète). Les panneaux hybrides permettent de produire de l’électricité et de préchauffer l’eau chaude. Le tout a été associé à une pompe à chaleur air-eau, une Alfea Extensa A.I de la marque française Atlantic. Il s’agit d’une pompe à chaleur air-eau moyenne température (55⁰C) qui nous permet de conserver notre installation de chauffages en fonte, au prix (à déterminer) d’une consommation supérieure lors des grands froids (ce qui n’arrive pas souvent ici). La pompe est associée à un ballon d’eau chaude qui exploite le circuit préchauffé par les panneaux.

Les 6kWc nous permettent d’être en auto-consommation totale en journée l’été (et probablement une partie du printemps/automne, à confirmer avec le temps) et je devrais pouvoir revendre une partie de la surproduction à Enedis (mais pour des raisons administratives, mon dossier est toujours en attente…).

Mon calcul de retour sur investissement m’a donné 12 ans, en prenant en compte une augmentation annuelle de 5% des prix de l’électricité. Si les tarifs augmentent plus vite, alors ça sera rentabilisé plus vite, mais il est impossible de savoir si ça sera le cas… a priori avec la crise énergétique, j’ai tendance à croire que ça n’est pas un mauvais calcul…

Enfin, nous n’avons pas fait installer de batteries: bien que cela serait extrêmement intéressant dans mon cas, pour récupérer la nuit le surplus de la production en journée, le prix des batteries est encore bien trop élevé (de l’ordre de 7k€ pour 10kW).

Changer ses habitudes de consommation

Dans mon cas, nos habitudes de consommation étaient assez simples, au final:

en journée, nous n’avions pas trop le choix: il faut bien alimenter les ordinateurs, etc.
les machines à laver, lave-vaisselle, etc tournaient la nuit pour profiter des heures creuses

Malheureusement, notre consommation en heures creuses était assez limitée comparée au reste. Tout à changé avec les panneaux solaires:

cet été et même fin septembre, lorsque le ciel est dégagé, mes panneaux produisent jusqu’à 5.5kW, ce qui dépasse largement ma consommation "live"
nous chargeons la voiture en journée via la prise "domestique" (~2kW) lorsque c’est possible, et la nuit en charge "rapide" (6kW) lorsque la batterie est trop basse
les machines tournent en journée au lieu de la nuit

Mais optimiser tout cela est compliqué, et je souhaitais un système qui permette à mon épouse et mes enfants de savoir si, par exemple, "c’est le moment" de lancer une machine.

Je me suis donc lancé dans un projet de "bidouille" pour faire un système de monitoring qui donnerait en direct ma production, ma consommation, et indiquerait de manière simple si on a de la marge ou pas.

Petite complication

Évidemment, vous pourriez vous dire que ça devrait être simple, avec toutes les applications connectées qui existent. Oui et non. Chez moi, l’installation est particulière: mes panneaux solaires sont sur le toit de ma maison, ainsi que les onduleurs et optimiseur. Ces appareils sont capables de vous donner en direct la production, et mesurent aussi la consommation instantanée, ce qui permet donc de mesurer directement son auto-consommation: on peut savoir si on produit plus qu’on ne consomme, ou l’inverse, en temps réel, c’est parfait non ? Voici par exemple un graphique fournit par SolarEdge en été:

En rouge, nous avons la consommation. En vert, la production solaire, et en bleu, l’autoconsommation. Vous noterez d’ores et déja un problème, si, comme moi, vous êtes un tant soit peu curieux: lorsqu’il y a production solaire, la courbe de consommation se met à suivre celle de production. Elle monte lorsque la production augmente et diminue lorsqu’elle baisse: ça n’est pas logique et probablement un bug quelque par chez SolarEdge. J’ai demandé via mon installateur des explications (ils ne comprennent pas non plus), mais SolarEdge n’est pas revenu vers eux.

A titre de comparaison, voici un autre graphe ce mois-ci:

On note déja qu’il y a des pics de consommation, correspondant à l’enclenchement de la pompe à chaleur, ou des appareils de cuisson. On constate aussi que la période de production est plus resserrée, mais qu’il existe encore, en journée, de la surproduction par moment (ça n’est pas toujours vrai, dès qu’il pleut, la production est pour ainsi dire nulle).

Si on oublie la fausse valeur de consommation, c’est parfait me direz-vous ! Et bien oui mais non. Chez moi, il y a un hic: le capteur de consommation est au niveau de l’onduleur, près de mon tableau électrique. Or ici, j’ai plusieurs tableaux électriques: un dans mon habitation principale, mais aussi un dans mon garage (bâtiment indépendant) et un autre dans un local technique du jardin. Lorsque j’ai acheté cette maison, elle était installée en triphasé, il y avait donc les 3 phases, et de mon compteur EDF principal, dans mon allée, partent des lignes électriques vers 3 bâtiments distincts. Ces phases étaient particulièrement déséquilibrées, nous avions donc fait repasser en monophasé, mais ce qu’il faut retenir, c’est qu’il aurait fallu que le capteur soit au niveau du compteur électrique, et non dans ma maison, pour qu’on puisse mesurer correctement la consommation en live. Pour des raisons techniques, il n’était pas possible de le faire: j’ai donc une mesure imparfaite de ma consommation, qui ne donne que la consommation de mon habitation principale.

Ma solution

Si on résume, je dispose pour l’instant d’un outil qui me donne la production (nous verrons plus loin comment l’obtenir), mais pas la consommation. Mais mon fournisseur d’électricité (Total Energies) propose une clé ATOME qui permet de connaître ce que je consomme du réseau en live, clé qui se branche sur le compteur Linky. Malheureusement, Enedis ne propose pas d’APIs permettant de connaître sa consommation live, il n’y a donc pas d’autre choix que de louer la clé de Total Energies… J’ai donc fait l’acquisition de cette clé, et je dispose donc des 2 mesures dont j’ai besoin:

la consommation live donnée par ma clé ATOME: attention, il ne s’agit donc pas de ma consommation totale, mais de ce que j’ai besoin en plus de ma production, du réseau EDF
la production live donnée par SolarEdge

Il suffit donc de faire la différence entre les 2 pour savoir de quelle marge on dispose, mais je ne peux jamais, en pratique, savoir combien je consomme.

Mon idée a donc été d’utiliser un vieux Rapsberry PI qui traînait dans mon armoire, combiné à un écran e-ink, pour afficher cette production et cette consommation, ainsi qu’une "note" permettant de suggérer de lancer une machine à laver, par exemple.

J’ai donc fait l’acquisition de cet écran, un écran e-ink dont la consommation en veille est nulle, ce qui, pour un outil de monitoring de consommation d’énergie, me semblait le minimum. Nous verrons cependant que ça ne fut pas sans inconvénients.

Les APIs, toujours le point faible

Bien, maintenant que nous savons que les données sont disponibles, via le site de SolarEdge pour la production, et via l’application Total Energies pour la clé live, il fallait disposer de ces données via des API que je puisse interroger via mon Raspberry.

Là, douche froide:

SolarEdge propose bien une API pour développeurs, mais elle n’est ni super bien documentée (il faut comprendre soit-même à quoi correspondent les champs retournés), ni illimitée : on ne peut effectuer que 300 requêtes par jour, soit un peu moins d’une requête toutes les 5 minutes ! C’est d’autant plus regrettable que l’information est disponible en continu et en live via leur interface web!
pour Total Energies, c’est encore pire: il n’y a pas d’API officielle. Il faut donc "hacker" pour avoir accès, en simulant une connexion via l’application, qui donne la consommation live

Bref, ni SolarEdge, ni Total ne proposent d’API de type push, ou d’event bus qu’on puisse écouter pour obtenir les informations. C’est très décevant, à une heure où ce genre d’optimisations devient critique pour une bonne gestion de notre consommation électrique: c’est un outil pour le climat !

L’autre problème, c’est que même si j’utilise l’API officielle de SolarEdge et que je réussis à récupérer l’information de Total Energies, ces APIs sont instables : elles tombent très souvent "en panne" et ne renvoient aucune info. Bref, lorsque ça marche, c’est parfait, mais souvent, ça ne fonctionne tout simplement pas, par exemple en ce moment, ma production indique 0 alors que ça n’est pas le cas:

Ceci me permet au moins de savoir qu’en ce moment, je demande 1135W du réseau, ce qui signifie que je consomme sensiblement plus (entre le chauffe-eau, l’ordinateur de mon fils qui joue à Minecraft et la PS5 de mon autre garçon).

Un peu de technique

Alors comment récupérer en pratique ces informations ? En fait j’ai adapté un script en Python qui, toutes les 5 minutes, se connecte à ces 2 APIs, récupère les informations et déclenche le rafraîchissement de l’écran. Alors, Python, personnellement c’est pas ma tasse de thé. J’ai l’impression de refaire du PHP, avec des scripts cracras et des variables globales de partout. Il y a sûrement possibilité de faire mieux, mais en bidouille en se connectant par SSH à mon Raspberry, c’est pour l’instant tout ce que j’ai.

Vous trouverez donc le script ici.

Parmi les problèmes, je vous mentionnait celui du choix de l’écran. J’avoue avoir été relativement naïf, parce qu’à l’utilisation, si l’écran e-ink est très sympa à voir, son rafraîchissement prend… jusqu’à 30s ! En effet, la façon de dessiner sur ces écrans est assez particulière: on écrit des modèles mémoire, puis on envoie des instructions à l’écran pour effacer telle zone, etc. Ces instructions sont très lentes à s’exécuter, mais surtout, elles provoquent systématiquement un effet bizarre à l’écran, qui passe du blanc au noir plusieurs fois, commence à afficher des choses, puis la couleur, etc… Bref, pas super sympa pour du "live", mais, au final, suffisant pour mon usage.

Conclusion

Au final, j’ai quand même un outil proche de ce que je souhaitais. Il nous a permis, d’ores et déjà, d’adapter notre consommation: n’importe qui peut regarder l’écran et décider de lancer une machine si la production est importante, alors que cette info n’était avant uniquement disponible que pour moi, via une application sur mobile: ici, l’information est donnée en continu, de manière passive, via cet écran.

Nous avons, par exemple, pu adapter notre consommation en automne: la pompe à chaleur ne fonctionnait pas (pas besoin de chauffer) et donc nous avons pu mettre notre voiture à charger, plus travailler tous les 2 à domicile et lancer une machine à laver, sans consommer un seul kW du réseau EDF ! Lorsque le monitoring a indiqué que nous commencions à consommer du réseau, il a suffit de couper la charge de la voiture (malheureusement, l’application MyPeugeot est extrêmement limitée et ne permet pas d’interrompre une charge mais simplement de la différer, mais c’est un autre problème).

Depuis Juillet, nous sommes à -70% de consommation électrique, ce qui est énorme. Cependant, les conditions météo ont, jusqu’ici, été très favorables: beaucoup de soleil et des températures record en Octobre et Novembre (malheureusement pour le climat…). Depuis une semaine, la pompe à chaleur se met régulièrement en route pour tenir les 19⁰C, mais j’ai des résultats surprenants:

90kWh consommés en Novembre pour chauffer l’eau
et seulement 9kWh pour le chauffage !

L’avenir nous dira si le passage à une pompe à chaleur pour le chauffage était une bonne idée ou non (en même temps, ici nous n’avions pas trop le choix). Il serait pertinent, compte tenu du surplus de production qui arrive souvent en journée, d’installer des batteries pour maximiser l’autoconsommation. Malheureusement comme je l’ai indiqué, le prix est à ce jour bien trop élevé. Je devrais donc me contenter de revendre ma production (au prix de 10cts le kWh, prix fixé pour 20 ans (!!) contre environ 15cts lorsque je consomme). Attention cependant, lorsqu’on revend et que notre installation, comme la nôtre, dépasse les 3kWc, alors nous devons déclarer celà en revenus. Cela rend la revente bien moins intéressante, puisque malgré le fait qu’on va "consommer" notre surplus, mais à des heures différentes, le tarif de rachat ansi que le fait de devoir déclarer réduit sensiblement la rentabilité.

Hello, Mastodon (goodbye Twitter!)

Sun, 6 Nov 2022 00:00:00 +0100

It should be no news to anyone that Elon Musk finally took control of Twitter. There were reasons to be worried about this move, but the recent events at Twitter made it worse than most of us would have thought.

TL/DR: I am moving primarily to Mastodon. You can now follow me at @melix@mastodon.xyz.

Twitter as a self-promotion tool

Lots of people have a complicated story with Twitter, I am no different. I joined Twitter back in January 2010, and I’ve grown up to more 4800 followers nowadays, and from time to time, I suffered from being exhausted by the negativity of the network. Last year, I was even in the middle of an harassment story, because some trolls thought I was someone else.

However, Twitter has been my main communication channel for professional work: this is where I explain what I work on, announce software updates, events I will attend or talk to, or publish blog posts links like this one. This is also my main feedback channel and where I do most of my technology intelligence.

My community of followers doesn’t make me an "influencer" at large scale, like a rock star, but it definitely had an influence on my professional career. For example, different potential employers explicitly mentioned my "influence" and social media presence, for example, as a major reason why they’d like to work with me. I have no doubt that without Twitter, my career wouldn’t have been the one it is today. For that, I am forever grateful.

I am also using my Twitter account for personal stuff. Most notably, some of you have seen my astronomy pictures. You may also have seen some political posts and point of views being expressed. While I’m using Twitter professionally, I always made it clear that the opinions I’m expressing are not representative of my employer’s point of view. Said differently, I see my Twitter account as a tool, but a tool which "survives" my different employers. To some extent, it’s part of my digital identity.

Then came Elon Musk

I was already very critical of Elon Musk, and not just because of the infamous Starlink project, for which I have so much to say. One can admire what he created with Tesla or SpaceX (and I do admire), but at the same time, we can recognize that he’s a terrible human being. This is all about business, all about money, and “freedom of speech” only when it suits his point of view. Elon is making people work as hard as they can until they burnout or simply get fired once they have "done their job". Elon is also a "special" view of free speech, where those who pay will have right for free speech, while those who don’t won’t be visible anymore. He’s also the one blocking anyone criticizing him.

As a consequence, the head of Twitter, and therefore Twitter itself, now represents everything I fight against in my personal life. This is the world I don’t want for my children. This is the world which puts money before human beings, the world where power, masculinity, is emphasized. Elon doesn’t care that we exhaust resources, as long as part of the population can survive, be it on Mars (!).

But now, he fired half of the company, by email, ignoring the law, because, you know, Musk does whatever he wants. It happens that I know people who got fired and I also know folks who used to work for Twitter before. I, for one, am respectful of what people produce. I am respectful of what efforts it takes to build a site like Twitter, and, in general, I am someone who puts trust in other people at the top of my hierarchy of beliefs. So I find it extremely annoying, or, to be clear, disgusting when someone who has absolutely no understanding of how to build such a large community over the years, decides to layoff half of a company, change the spirit of the website so that paying customers have more power than the others, so that it matches more his political point of view. Even if the company is losing $4M a day, compared to B44$, seriously, what would have it taken to show the people who built Twitter a bit of respect and simply build a plan, say, of volunteer departures.

Elon Musk’s behavior is everything I hate: someone coming because they have power (understand, money), and then ruining other people lives just because he can, without any respect.

This puts me in a difficult situation: at the same time, I need Twitter, because it’s a professional tool which can’t be easily replaced and I have to go because there’s no way I’m going to pay 8 dollars a month to a company who treats human beings like that, and simply to get more visibility over others just because I can afford it.

Hello, Mastodon!

As a FOSS supporter, back in 2017, I had registered to Mastodon. My account was left more or less inactive for years, because, honestly, not many people used it so it was … annoying. This was like talking in an echo chamber. Until the rumor of Elon Musk acquiring Twitter started: we’ve seen more and more people joining, and I’m super glad that this week alone, many other people, including from the Tech industry, decided to make the move.

No more ads. No more recommendations. A clean timeline, as it should be.

It’s really refreshing to be on Mastodon, it feels like the Twitter from the early days.

However, Mastodon has a number of key differences, which make it more "complicated" for users to understand. First of all, and that’s the biggest issue for joining, Mastodon is not a single site like Twitter: it’s a federation of servers. Just like you have a provider for your email (say GMail, Hotmail, etc.), you can choose your Mastodon server provider. As an OSS project, you can go even as far as hosting your own instance.

Therefore, Mastodon is, by nature, distributed, which makes it completely immune to what just happened to Twitter. But it also means that there are a few things you should be aware when you join a server:

content moderation is the responsibility of the server administrator: they can block you, read your posts, private messages. They can block other providers too, which means, for example, that if you choose to go to an instance which has a Code of Conduct, you have to follow the rules.
the good news is that you own the content: if you are not happy with your server moderation rules, you can move to a different instance, and you can move your data with it: toots, folks you follow, followers, …
the cost of maintenance is distributed on the community: there are lots of free servers out there, but you can choose to participate to the bills. You can even build your own server, for a price which wouldn’t exceed much what Elon wants us to pay to get a blue mark
Hashtags are much more important in Mastodon than Twitter: there’s no global search: the only things which are indexed are hash tags. You won’t find contents which is not with a hash tag.

This is the internet I remember of. It’s not suitable for everyone, but it’s what I call open. What I also like is that it respects freedom of speech, while preserving your freedom of not seeing assholes (there are instances which are full of alt-right folks, but at least, we can block the whole server and not have to suffer their nauseating posts).

Conclusion

So, for the time being, I’m transitioning to Mastodon. You can follow me at @melix@mastodon.xyz, and I strongly suggest that you do the same and find a server which suits you. I will not support a company which now represents the worst of human beings. I will not pay $8 a month to a company which shows no respect to its employees, content creators and business partners.

Because it’s hard to say goodbye to a professional network just like that, I do not plan to shutdown my account on Twitter yet, though: I still didn’t find a way to solve this cognitive dissonance.

So what I plan on doing is moving to Mastodon first. I have already updated my screen name in Twitter to give a link to my Mastodon account. I suggest you do the same. I will now limit my tweets to what is strictly required for my professional career (announcements, etc.). I will also mostly answer to tweets which are related to moving to Mastodon and use Mastodon for everything else, until I can completely get rid of Twitter. I am conscious that I may not be able to completely get rid of it, if too many people stay on Twitter. So be it.

Congratulations M. Musk, you ruined Twitter!

Astrophotographie: la suite !

Wed, 5 Oct 2022 00:00:00 +0200

Salut à tous !

Je vous avais promis de faire un deuxième live sur Twitch pour parler d’astrophotographie, cette fois-ci sur la partie logicielle. J’ai décidé d’arrêter de procrastiner et je vous annonce le mercredi 12 octobre à 20h sur ma chaîne Twitch.

Si vous avez raté la première partie sur le matériel astrophoto, c’est dispo en replay sur Youtube.

Dans ce live j’aborderais comment, à partir des données brutes acquises pendant la nuit, on peut obtenir de belles photos grâce à la magie du logiciel. Nous parlerons de flats, darks, pré-traitement, stacking, … plein de termes compliqués mais qui au final ne sont pas si difficiles à comprendre.

Bref, il est temps que je prépare tout ça !

Introducing Micronaut Test Resources

Thu, 4 Aug 2022 00:00:00 +0200

The new release of Micronaut 3.6 introduces a new feature which I worked on for the past couple of months, called Micronaut Test Resources. This feature, which is inspired from Quarkus' Dev Services, will greatly simplify testing of Micronaut applications, both on the JVM and using GraalVM native images. Let’s see how.

Test resources in a nutshell

Micronaut Test Resources simplifies testing of applications which depend on external resources, by handling the provisioning and lifecycle of such resources automatically. For example, if your application requires a MySQL server, in order to test the application, you need a MySQL database to be installed and configured, which includes a database name, a username and a password. In general, those are only relevant for production, where they are fixed. During development, all you care about is having one database available.

Here are a couple of traditional solutions to this problem:

document that a MySQL server is a pre-requisite, and give instructions about the database to create, credentials, etc. This can be simplified by using Docker containers, but there’s still manual setup involved.
Use a library like Testcontainers in order to simplify the setup

In general, using Testcontainers is preferred, because it integrates well with the JVM and provides an API which can be used in tests to spawn containers and interact with them. However, a better integration between Micronaut and Testcontainers can improve the developer experience in several ways:

simplify the container lifecycle configuration by providing an opinionated framework-specific default way, making you think less of how to setup it in the individual tests : tests shouldn’t need to deal with the container lifecycle: we’d like to have test containers/resources management as transparent as possible.
isolate it better from your application making it simpler to reason about dependencies (and transitive dependencies), not just for the developer, but for example tools enabling native mode as well: Testcontainers APIs "leak" to the test classpath, making it difficult to run tests in native mode. This is not a problem specific to the Testcontainers library though: many libraries are not yet compatible with GraalVM. Our solution makes it possible to use Testcontainers in native tests without the hassle of configuring it!
enable support for "development mode", that is to say when you run the application locally (not the tests, the application itself) or even several distinct projects at once that can benefit from sharing access to the same running containers (for example, an MQTT client and a server may want to use the same container, even if they are individual projects living in distinct Git repositories).

The goal of Micronaut Test Resources is to achieve all of these at once:

zero-configuration: without adding any configuration, test resources should be spawned and the application configured to use them. Configuration is only required for advanced use cases.
classpath isolation: use of test resources shouldn’t leak into your application classpath, nor your test classpath
compatible with GraalVM native: if you build a native binary, or run tests in native mode, test resources should be available
easy to use: the Micronaut build plugins for Gradle and Maven should handle the complexity of figuring out the dependencies for you
extensible: you can implement your own test resources, in case the built-in ones do not cover your use case
technology agnostic: while lots of test resources use Testcontainers under the hood, you can use any other technology to create resources

In addition, Micronaut Test Resources support advanced development patterns, which are useful in the microservices era. As an example, it is capable of sharing containers between submodules of a single build, or even between independent projects, from different Git repositories! Say that you have 2 projects, one built with Gradle, the other with Maven, both needing to communicate using the same message bus: Micronaut is capable of handling this use case for you, making it extremely easy to test components interacting with each other!

Because of these constraints, we decided to use Testcontainers, because the library is just perfect for the job, but in an isolated process instead, as I’m going to describe below. Note that this solution is also 100% compatible with Testcontainers Cloud, which makes container provisioning even easier!

Using Micronaut Test Resources

Enabling test resources support

Micronaut Test Resources integrates with build tools. In both Maven and Gradle, you need to enable test resources support. If you create a new project using Micronaut Launch or the Micronaut CLI, test resources will be configured for you, but if you migrate an existing application to test resources, here’s what you need to do:

If you are using Maven, you will need to upgrade to the Micronaut 3.6 parent POM and add the following property:

<properties>
   <micronaut.test.resources.enabled>true</micronaut.test.resources.enabled>
</properties>

For Gradle, you can use test resources with Micronaut 3.5+ and you simply need to use the test resources plugin:

plugins {
    id 'io.micronaut.application' version '3.5.1'
    id 'io.micronaut.test-resources' version '3.5.1'
}

Our first test resources contact

In this blog post we will write an application which makes use of Micronaut Data and connects to a MySQL server to list books. The whole application code is available on GitHub, so I’m only going to show the relevant parts for clarity.

In such an application, we typically need a repository:

@JdbcRepository(dialect = Dialect.MYSQL)
public interface BookRepository extends CrudRepository<Book, Long> {
    @Override
    List<Book> findAll();
}

And this repository makes use of the Book class:

@MappedEntity
public class Book {
    @Id
    @GeneratedValue(GeneratedValue.Type.AUTO)
    private Long id;

    private String title;

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title;
    }
}

In order for Micronaut to use the database, we need to add some configuration to our application.yml file:

datasources:
  default:
    schema-generate: CREATE
    db-type: mysql

The most important thing to see is that we don’t specify any username, password or URL to connect to our database: the only thing we have to specify is the database type of our datasource. We can then write the following test:

@MicronautTest
class DemoTest {

    @Inject
    BookRepository bookRepository;

    @Test
    @DisplayName("A MySQL test container is required to run this test")
    void testItWorks() {
        Book book = new Book();
        book.setTitle("Yet Another Book " + UUID.randomUUID());
        Book saved = bookRepository.save(book);
        assertNotNull(saved.getId());
        List<Book> books = bookRepository.findAll();
        assertEquals(1, books.size());
    }

}

The test creates a new book, stores it in the database, then checks that we get the expected number of books when reading the repository. Note, again, that we didn’t have to specify any container whatsoever. In this blog post I’m using Gradle, so we can verify the behavior by running:

./gradlew test

Then you will see the following output (cleaned up for clarity of this blog post):

i.m.testresources.server.Application - A Micronaut Test Resources server is listening on port 46739, started in 128ms
i.m.t.e.TestResourcesResolverLoader - Loaded 2 test resources resolvers: io.micronaut.testresources.mysql.MySQLTestResourceProvider, io.micronaut.testresources.testcontainers.GenericTestContainerProvidereted
o.testcontainers.DockerClientFactory - Connected to docker:
  Server Version: 20.10.17
  API Version: 1.41
  Operating System: Linux Mint 20.3
  Total Memory: 31308 MB
🐳 [testcontainers/ryuk:0.3.3] - Creating container for image: testcontainers/ryuk:0.3.3
🐳 [testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 is starting: 1f5286fa728aca74a7d6d4c0eb2148a3bc81f5c028027496d7aabda7b7ed45e8
🐳 [testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 started in PT0.655476S
o.t.utility.RyukResourceReaper - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
🐳 [mysql:latest] - Creating container for image: mysql:latest
🐳 [mysql:latest] - Container mysql:latest is starting: d796c7a1ce10f393a4181f12967ee77ac9864f45595f97967c700f022e86ac7d
🐳 [mysql:latest] - Waiting for database connection to become available at jdbc:mysql://localhost:49209/test using query 'SELECT 1'
🐳 [mysql:latest] - Container is started (JDBC URL: jdbc:mysql://localhost:49209/test)
🐳 [mysql:latest] - Container mysql:latest started in PT7.573915S

BUILD SUCCESSFUL in 11s
7 actionable tasks: 2 executed, 5 up-to-date

What does this tell us? First, that a "Micronaut Test Resources server" was spawned, for the lifetime of the build. When the test was executed, this service was used to start a MySQL test container, which was then used during tests. We didn’t have to configure anything, test resources did it for us!

Running the application

What is also interesting is that this also works if you run the application in development mode. Using Gradle, you do this by invoking ./gradlew run (mvn mn:run with Maven): as soon as a bean requires access to the database, a container will be spawned, and automatically shut down when you stop the application.

Note	Of course, in production, there won’t be any server automatically spawned for you: Micronaut will rely on whatever you have configured, for example in an `application-prod.yml` file. In particular, the URL and credentials to use.

What is even nicer is that you can use this in combination with Gradle’s continuous mode!

To illustrate this, let’s create a controller for our books:

@Controller("/")
public class BookController {
    private final BookRepository bookRepository;

    public BookController(BookRepository bookRepository) {
        this.bookRepository = bookRepository;
    }

    @Get("/books")
    public List<Book> list() {
        return bookRepository.findAll();
    }

    @Get("/books/{id}")
    public Book get(Long id) {
        return bookRepository.findById(id).orElse(null);
    }

    @Delete("/books/{id}")
    public void delete(Long id) {
        bookRepository.deleteById(id);
    }
}

Now start the application in continuous mode: ./gradlew -t run

You will see that the application starts a container as expected:

INFO  io.micronaut.runtime.Micronaut - Startup completed in 9166ms. Server Running: http://localhost:8080

Notice how it took about 10 seconds to start the application, most it it spent in starting the MySQL test container itself. You definitely don’t want to pay this price for every change you make, so this is where the continuous mode is helpful. If we ask for the list of books, we’ll get an empty list:

$ http :8080/books
HTTP/1.1 200 OK
Content-Type: application/json
connection: keep-alive
content-length: 2
date: Tue, 26 Jul 2022 16:59:51 GMT

[]

This is expected, but notice how we didn’t have a method to actually add a book to our store. Let’s fix this by editing the BookController.java class without stopping the server. Add the following method:

    @Get("/books/add/{title}")
    public Book add(String title) {
        Book book = new Book();
        book.setTitle(title);
        return bookRepository.save(book);
    }

Save the file and notice how Gradle instantly reloads the application, but doesn’t restart the database: it’s already there so it’s going to reuse it!

In the logs you will see something like this:

INFO  io.micronaut.runtime.Micronaut - Startup completed in 1086ms. Server Running: http://localhost:8080

This time the application started in just a second! Let’s add a book:

$ http :8080/books/add/Micronaut%20in%20action
HTTP/1.1 200 OK
Content-Type: application/json
connection: keep-alive
content-length: 38
date: Tue, 26 Jul 2022 17:03:57 GMT

{
    "id": 1,
    "title": "Micronaut in action"
}

However, if we stop the application (by hitting CTRL+C) and start again, you will see that the database will be destroyed when the application shuts down. Let’s see how we can "survive" different build invocations.

Keeping the service alive

By default, the test resources service is short lived: it’s going to be started at the beginning of a build, and shutdown at the end of a build. This means, that it will live as long as you have tests running, or, if running in development mode, as long as the application is alive. However, you can make it survive the build, and reuse the containers in several, independent build invocations.

To do this, you need to explicitly start the test resources service:

./gradlew startTestResourcesService

This starts the test resources service in the background: it did not start our application, nor did it run tests. This means that now, we can start our application:

./gradlew run

And, because it’s the first time the application is launched since we started the test resources service, it’s going to spawn a test container:

INFO  io.micronaut.runtime.Micronaut - Startup completed in 9211ms. Server Running: http://localhost:8080

We can add our book:

$ http :8080/books/add/Micronaut%20in%20action
HTTP/1.1 200 OK
Content-Type: application/json
connection: keep-alive
content-length: 38
date: Tue, 26 Jul 2022 17:03:57 GMT

{
    "id": 1,
    "title": "Micronaut in action"
}

The difference is now that if we stop the application (e.g hit CTRL+C) and start it again, it will reuse the container:

INFO  io.micronaut.runtime.Micronaut - Startup completed in 895ms. Server Running: http://localhost:8080

If we list our books, the database wasn’t cleaned, so we’ll get the book we created from the previous time we started the app:

$ http :8080/books
HTTP/1.1 200 OK
Content-Type: application/json
connection: keep-alive
content-length: 40
date: Tue, 26 Jul 2022 17:14:40 GMT

[
    {
        "id": 1,
        "title": "Micronaut in action"
    }
]

Nice, right? However there’s a gotcha if you do this: what happens if we run tests?

$ ./gradlew test

> Task :compileTestJava
Note: Creating bean classes for 1 type elements

> Task :test FAILED

DemoTest > A MySQL test container is required to run this test FAILED
    org.opentest4j.AssertionFailedError at DemoTest.java:28

Why is that? This is simply because our tests expect a clean database, and we had a book in it, so keep this in mind if you’re using this mode.

At some point, you will want to close all open resources. You can do this by explicitly stopping the test resources service:

./gradlew stopTestResourcesService

Now, you can run the tests again and see them pass:

$ ./gradlew test

...
INFO  🐳 [testcontainers/ryuk:0.3.3] - Creating container for image: testcontainers/ryuk:0.3.3
INFO  🐳 [testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 is starting: ea2aa1c7f1e66a9c7306b00443e8a6693451f3f02bd780b3e2ed7b96ed59936a
INFO  🐳 [testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 started in PT0.553559699S
INFO  o.t.utility.RyukResourceReaper - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
INFO  o.testcontainers.DockerClientFactory - Checking the system...
INFO  o.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
INFO  🐳 [mysql:latest] - Creating container for image: mysql:latest
INFO  🐳 [mysql:latest] - Container mysql:latest is starting: 1c6437a55b8f9e5668bcec4aef27087c889b8a77ca18d2ddf58809853482a422
INFO  🐳 [mysql:latest] - Waiting for database connection to become available at jdbc:mysql://localhost:49227/test using query 'SELECT 1'
INFO  🐳 [mysql:latest] - Container is started (JDBC URL: jdbc:mysql://localhost:49227/test)
INFO  🐳 [mysql:latest] - Container mysql:latest started in PT7.469460173S

BUILD SUCCESSFUL in 11s
7 actionable tasks: 2 executed, 5 up-to-date

Native testing

Did you know that you can run your test suite in native mode? That is to say, that the test suite is going to be compiled into a native binary which runs tests? One issue with this approach is that it’s extremely complicated to make it work with Testcontainers, as it requires additional configuration. With Micronaut Test Resources, there is no such problem: you can simply invoke ./gradlew nativeTest and the tests will properly run. This works because Testcontainers libraries do not leak into your test classpath: the process which is responsible for managing the lifecycle of test resources is isolated from your tests!

Under the hood

How does that work?

In a nutshell, Micronaut is capable of reacting to the absence of a configured property. For example, a datasource, in order to be created, would need the value of the datasources.default.url property to be set. Micronaut Test Resources work by injecting those properties at runtime: when the property is read, it triggers the creation of test resources. For example, we can start a MySQL server, then inject the value of the JDBC url to the datasources.default.url property. This means that in order for test resources to work, you need to remove configuration (note that for production, you will need to provide an additional configuration file, for example application-prod.yml, which provides the actual values).

The entity which is responsible for resolving missing properties is the "Test Resources Server": it’s a long lived process which is independent from your application and it is responsible for managing the lifecycle of test resources. Because it’s independent from the application, it means it can contain dependencies which are not required in your application such as, typically, the Testcontainers runtime. But it may also contain additional classes, like JDBC drivers, or even your custom test resources resolver!

Because this test resources server is a separate process, it also means it can be shared by different applications, which is the reason why we can share the same containers between independent projects.

Once you understand that Micronaut Test Resources work by resolving missing properties, it becomes straightforward to configure. In particular, we offer configuration which makes it very easy to support scenarios which are not supported out of the box. For example, Micronaut Test Resources supports several JDBC or reactive databases (MySQL, PostgreSQL, MariaDB, SQL Server and Oracle XE), Kafka, Neo4j, MQTT, RabbitMQ, Redis, Hashicorp Vault and ElasticSearch, but what if you need a different container?

In that case, Micronaut Test Resources offer a conventional way to create such containers, by simply adding some configuration lines: in the documentation we demonstrate how to use the fakesmtp SMTP server with Micronaut Email for example.

Custom test resources

If the configuration-based support isn’t sufficient, you also have, in addition, the ability to write your own test resources. If you use Gradle, which I of course recommend, this is made extremely easy by the test resources plugin, which creates an additional source set for this, named testResources. For Maven, you would have to create an independent project manually to support this scenario.

As an illustration, let’s imagine that we have a bean which reads a configuration property:

@Singleton
public class Greeter {
     private final String name;

     public Greeter(@Value("${my.user.name}") String name) {
         this.name = name;
     }

     public String getGreeting() {
     	return "Hello, " + name + "!";
     }

     public void sayHello() {
         System.out.println(getGreeting());
     }
}

This bean requires the my.user.name property to be set. We could of course set it in an application-test.yml file, but for the sake of the exercise, let’s imagine that this value is dynamic and needs to be read from an external service. We will implement a custom test resources resolver for this purpose.

Let’s create the src/testResources/java/demo/MyTestResource.java file with the following contents:

package demo;

import io.micronaut.testresources.core.TestResourcesResolver;

import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Optional;

public class MyTestResource implements TestResourcesResolver {

    public static final String MY_TEST_PROPERTY = "my.user.name";

    @Override
    public List<String> getResolvableProperties(Map<String, Collection<String>> propertyEntries, Map<String, Object> testResourcesConfig) {
        return Collections.singletonList(MY_TEST_PROPERTY); // (1)
    }

    @Override
    public Optional<String> resolve(String propertyName, Map<String, Object> properties, Map<String, Object> testResourcesConfiguration) {
        if (MY_TEST_PROPERTY.equals(propertyName)) {
            return Optional.of("world");                    // (2)
        }
        return Optional.empty();
    }

}

Tells that this resolver can resolve the my.user.name property
Returns the value of the my.user.name property

And in order for the resolver to be discovered, we need to create the src/testResources/resources/META-INF/services/io.micronaut.testresources.core.TestResourcesResolver file with the following contents:

demo.MyTestResource

Now let’s write a test for this by adding the src/test/java/demo/GreeterTest.java file:

package demo;

import io.micronaut.context.annotation.Requires;
import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
import jakarta.inject.Inject;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;

@MicronautTest
class GreeterTest {

    @Inject
    Greeter greeter;


    @Test
    @DisplayName("Says hello")
    void saysHello() {
        greeter.sayHello();
        Assertions.assertEquals("Hello, world!", greeter.getGreeting());
    }

}

Now if you run ./gradlew test, you will notice that Gradle will compile your custom test resource resolver, and when the test starts, you will read the following line:

Loaded 3 test resources resolvers: demo.MyTestResource, io.micronaut.testresources.mysql.MySQLTestResourceProvider, io.micronaut.testresources.testcontainers.GenericTestContainerProvider

So when the Greeter bean is created, it will read the value of the my.user.name property by calling your custom test resolver! Of course this is a very simple example, and I recommend that you take a look at the Micronaut Test Resources sources for more examples of implementing resolvers.

Conclusion

In this blog post, we’ve explored the new Micronaut Test Resources module, which will greatly simplify development of Micronaut applications which depend on external services like databases or messaging queues. It works by simplifying configuration, by removing lines which used to be present, but now are dynamically resolved, like datasources.default.url. Test resources are handled in a separate process, the test resources server, which is responsible for handling their lifecycle. This also makes it possible to share the resources (containers, databases, …) between independent builds. For advanced use cases, Micronaut Test Resources provides configuration based resources creation.

Last but not least, Micronaut Test Resources is an extensible framework which will let you implement your own test resources in case the built-in ones miss a feature.

Special thanks to Tim Yates for his hard work on upgrading the Micronaut Guides to use test resources, and Álvaro Sanchez-Mariscal for his support on the Maven plugin!

Je suis le gars chiant

Thu, 28 Jul 2022 00:00:00 +0200

La science. Si j’avais eu les capacités pour comprendre les abstractions mathématiques nécessaires (j’essaie encore, de temps en temps), je serais, peut-être, aujourd’hui assis dans un laboratoire de recherche en astrophysique. J’ai toujours trouvé fascinant la capacité à émettre des hypothèses, puis les vérifier expérimentalement, pour en faire des théories, théories qui, constamment, peuvent être remises en cause lorsque de nouvelles observations les contredisent. Je suis assis derrière un bureau, mais dans le développement informatique. Ma passion, mon loisir, c’est l’astronomie.

Alors oui, mais non

J’ai grandi dans un monde empli de graphologie, de magnétisme, d’ufologie, de pyramidologie, d’expériences de mort imminente, qui m’ont aussi profondément marqué. J’ai entendu des témoignages, j’ai constaté des guérisons, j’ai cru à des beaucoup de choses. Mais pendant une grande partie de mon enfance, je n’avais pas le bagage intellectuel suffisant pour comprendre. En un mot, ma fascination pour les sciences se mélait allègrement à celle pour les pseudo-sciences. Parce que j’ai toujours cherché des explications, j’en suis revenu et je pense que cette expérience a renforcé mon sens critique si tant est que je me sens, de nos jours, extrêmement proche du mouvement zététique et combats fermement l’anti-science.

En soirée, je suis le gars chiant qui ne supporte pas qu’on introduise du surnaturel là où il existe des explications rationnelles, celui qui soulève un sourcil amusé lorsqu’on lui parle de séances de spiritisme.

Celui qui vous explique que l’homéopathie ne fonctionne pas et qui vous rappelle que s’il existe un effet placébo, il est moralement discutable de le rembourser.

Je suis le gars chiant qui, il y a déja des années, avertissait mes amis sur la montée du mouvement antivax. Je suis le gars chiant qui, en tant qu’astronome amateur, vous démontre que l’astrologie n’a aucun sens. Je suis le gars chiant qui s’indigne qu’on puisse vous vendre des absurdités comme des colliers en ambre pour les douleurs dentaires des bébés ou des pierres à recharger sous la lumière de la lune.

Je suis le gars chiant qui vous invite à douter de la médecine chinoise, responsable du massacre des rhinocéros et de la maltraitance d’ours pour des produits dont il est démontré qu’il n’ont aucun effet. Je suis le gars chiant qui, lorsqu’on lui parle de produits naturels, vous rappelle que naturel ne veut pas nécessairement dire bon pour la santé. Je suis le gars chiant qui vous explique que si les électrosensibles sont réellement malades, ça n’est pas à cause des ondes mais de mécanismes psychologiques proches des phobies. Je suis le gars chiant qui s’insurge contre la naturopathie et ses tendances complotistes et invite à prendre connaissance des dérives sectaires de la méditation pleine conscience.

Je suis le gars chiant qui se dit écolo, mais aussi pro-nucléaire: je suis effondré lorsque les écolos incitent à relancer des centrales à charbon alors que les rapports du GIEC nous disent que chaque tonne de CO2 compte. Je suis le gars chiant qui choque ses amis écolos lorsqu’il affirme que le stockage géologique des déchets nucléaire n’est pas un problème pour les générations futures.

Je me sens écologiste et je suis convaincu que nous devons aller vers la sobriété: je recherche des modes de vie alternatifs, moins impactants pour l’environnement, moins industriels et à ce titre j’ai été attiré par les concepts des colibris et ses nombreuses variantes. Mais je vérifie, je confronte mes croyances. Je suis le gars chiant qui découvre que presque tous les mouvements écologistes sont infiltrés par des groupes anti-science, au premier rang desquels on trouve la secte anthroposophique. Je suis le gars chiant qui vous fait remarquer que la permaculture ne permet pas d’être auto-suffisant, que la plupart de ces mouvements dits alternatifs ne survivent que grâce à des "formations" qui s’auto-entretiennent et/ou financent des sectes.

Je suis l’écolo qui vous déculpabilise de ne pas effacer vos emails car celà ne réduira pas votre emprunte carbone, mais en profite pour vous faire remarquer que ne pas utiliser votre sèche-linge, oui.

Je suis le gars chiant qui s’inquiète lorsque des proches suivent des formations de développement personnel ou aux ennéagrammes, parce qu’il cherche qui sont ceux derrière ces mouvements et se rend compte qu’il est très facile de tomber dans des dérives sectaires et ne pas s’en rendre compte.

Je suis aussi le gars chiant qui fait confiance aux autorités sanitaires, y compris quand elles concluent qu’il n’y a statistiquement pas de surnombre de cancers pédiatriques au Sud de Nantes alors qu’on était en droit de s’inquiéter, mais aussi prêt à revoir ma position si de nouvelles données étaient disponibles.

Je suis celui qui heurte la sensibilité de ses amis, les choque, les énerve parfois (souvent), parce que j’apparais comme ne croyant en rien.

Triste vie que celle de celui qui ne croît en rien

Est-ce vrai ? Est-ce que je ne crois réellement en rien ? Bien évidemment non. J’ai mes propres croyances. Par exemple, je crois qu’il existe une vie extra-terrestre. Sous quelle forme, je n’en sais rien, mais j’en suis intimement convaincu. Mais il s’agit d’une croyance, et en tant que tel, je n’essaierai pas de vous l’imposer. Je peux vous donner des idées sur le pourquoi j’y crois, mais il n’existe aucune preuve. J’ai cru à beaucoup de choses aussi: gamin j’ai cru au bisou magique, j’ai cru au Père Noël, j’ai cru que la graphologie fonctionnait (alors que je suis horrifié de constater que cet outil est toujours utilisé de nos jours !). Je crois qu’il n’existe pas de vie après la mort, mais je ne peux pas vous en apporter la preuve. Je crois aussi qu’on n’a pas besoin de revenir 200 ans en arrière en agriculture pour "sauver la planète".

Alors on pourrait dire "mais c’est pas bien grave", "mais les plantes c’est quand même bien, c’est mieux que les produits chimiques", "la science n’explique pas tout", "faut il se fâcher pour ça ?". Tant de lieux communs, qui n’apportent rien, mais qui, en revanche, cachent de tristes réalités: des morts, souvent par milliers.

Dur d’être celui qui gâche la fête

Alors je si je suis suis le gars chiant, je ne cache pas que c’est parfois fatigant, moralement: si j’écris ce billet, c’est aussi pour exprimer cette fatigue. C’est fatigant parce que je vais contre les idées reçues, je "casse l’ambiance" mais surtout je peux me fâcher avec des personnes que j’apprécie énormément. C’est fatigant parce qu’on me regarde en sachant d’avance que je vais intervenir, mais on ne croira pas ce que je dis, même preuves à l’appui. C’est fatigant parce que je constate la montée de l’obscurantisme, du complotisme et que je me sens impuissant. C’est fatigant parce qu’il est urgent d’agir pour sauver notre avenir (je ne parlerai pas de sauver la planète, c’est l’espèce humaine, et les espèces actuelles en général qui est en jeu) et que les décisions prises ne sont pas à la hauteur. C’est fatigant parce que j’aspire à plus d’écologie, plus de simplicité, mais que tous les mouvements que je trouve sont plombés par de l’anti-science et des sectes déguisées en "spiritualité". C’est fatigant parce qu’on me caricature: je suis le "rationnel" qui ne comprend rien aux sciences humaines ou à l’humain en général. C’est fatigant parce qu’on se bat constamment contre l’intolérance. Tout ce que je vous raconte est présent partout: famille, amis, collègues, politiques, mais chez moi y compris.

Alors, parfois, parce que je ne veux pas me fâcher ou faire honte à madame, je me tais et je deviens d’humeur sombre, ne m’en voulez pas.

Nous sommes tous pleins de croyances: la question est de savoir si nous sommes prêts à les remettre en question, pour le bien de tous, et si ces croyances doivent influencer les décisions que nous prenons pour la communauté. Et parce que nous sommes tous susceptibles de faire des erreurs, je ne porte aucun jugement moral à celles et ceux qui croient ceci ou celà: nous sommes tous pleins de contradictions et je ne suis pas mieux qu’un autre. Je n’en veux pas à ceux qui croient, je ne doute pas qu’ils aient de bonnes intentions. Je pourrais en vouloir à ceux qui continuent de croire lorsqu’on leur montre qu’il y a de quoi douter, ou mieux qu’on leur apporte des preuves, mais notre cerveau est câblé pour rendre la remise en question difficile. J’en veux, en revanche, à ceux qui utilisent sciemment l’ignorance des gens pour en tirer profit financièrement ou pour psychologiquement de personnes.

Je n’ai pas de chiffres à vous donner, mais je crois que ces ordures sont nombreuses.

There is no single dependency graph

Mon, 4 Jul 2022 00:00:00 +0200

What are the dependencies of Micronaut? While the question may seem obvious, the answer is not. I realized, over the years, that many developers tend to ignore the complexity of dependency management in the Java ecosystem. In particular, I faced several occasions when discussing dependency upgrades (in particular in the context of security updates) when it became obvious that there was a big gap between the mental model that some people have, and the reality. In this blog post I want to address some misconceptions around dependencies in the Java ecosystem.

Libraries vs applications

Applications are easier

First and foremost, we have to make a difference between a library and an application. If I install, say, a desktop application, then the notion of "dependencies of the application" seem quite obvious: they are often bundled with the application in a libs directory, so all we have to do is look into that. Therefore, the question "what are the dependencies of X" can be conflated to a simpler question: "what are the dependencies that application X requires at runtime to run properly".

This is already a simplfication, though: nothing prevents the application from using a plugin system. This would make the answer to that question more complicated, but let’s focus on the simplest case. Because we have that file list at hand, answering questions like "is X vulnerable to log4shell" is easy: we can look into the libs directory, and if we find a vulnerable version of log4j in there, then we know for sure if the application uses a vulnerable dependency.

This model, the "Windows installer" one, is what lots of developers have in mind. The problem is often that they assume that the libraries work the same, but they don’t.

Library dependencies

For libraries (and frameworks, which can be seen as "super libraries") the answer is indeed more complex. A library is intended to be consumed by an application, or another library. Therefore, when we ask the question "what are the dependencies of library L", then the answer is "it depends".

In particular, libraries will introduce us to the world of transitive dependencies: an application will depend on libraries, which themselves have dependencies on other libraries.

For example, you can have the following dependency graph:

We can already see that we have a pelicular situation, where 2 libraries depend on the same module c. If they require the same version of c, then we are lucky. If they don’t, then we run in a situation called a version conflict, where different tools will use different strategies to solve them (no, Maven and Gradle, in particular, don’t have the same way of dealing with dependency conflicts).

Say that a requires c 1.0 and that d requires c 1.1: a reasonable strategy is to do optimistic upgrades and to choose c 1.1 because it’s the highest version. This is the default strategy that Gradle uses, for example. Maven, on the other hand, takes a simpler, but non prectictable strategy of "closest first", where "closest" actually means "first seen wins". In other words, if a is seen first, then c 1.0 will be used. Reverse the order of dependencies and c will use version 1.1. Gradle’s strategy is immune to those problems, but we can already see that there can be a difference between the dependencies which are declared in a build file and the ones which will effectively be used.

As a consequence, it’s a mistake to look at the declared dependencies to determine what are the effective dependencies of a project. You must always look at the resolved dependency graphs. This is why I wrote, a few months ago, that Dependabot gave a false sense of security (hopefully, now, they provide an API which can be used to mitigate that problem).

Building from sources

Now that we understand that the declared dependencies can be different from the resolved dependencies, and that we reckon that it’s the build tool’s responsibility to solve those conflicts, let’s address an elephant in the room: building a proejct entirely from sources (including transitive dependencies).

Let’s imagine that, for legal reasons, you don’t want to use Maven Central. Instead you want to build your project against the sources of your dependencies and build themselves from dependencies, and so on. Obviously, if your build tool doesn’t support resolving a dependency graph first (which implies having metadata available for transitive dependencies in some form) then replacing the dependencies with their sources instead of the binaries, then you have a problem: you’re going to have to figure out yourself what libraries to build, in which versions, and do everything by hand. Spoiler alert: no build tool supports that (in the Java ecosystem, some ecosystems like Rust always use source dependencies, which come with a number of other issues I won’t address in this blog post).

This means that in order to build your project, you must:

resolve all dependencies, including transitive dependencies, to figure out what version they need
find a way to fetch the sources for the particular version of each dependency that is used
update the build scripts of that project to use source dependencies
compile each project independently, and recurse to 1.

But hey, don’t you see the problem already (except from the fact you’d have to rewrite all build files, and figure out how to build each project according to their CI specification)? Because resolved dependency graphs only depend on the top level project being compiled, you have absolutely no guarantee that you’ll use the same, resolved versions everywhere. In the dependency graph above, if you resolve the dependency graph for app, then you will determine that you need to build c in version 1.1 from sources. Alright, but to build a, we will need to build c… with version 1.0! In other words, there’s no way you can consistently build such a dependency graph from source without having a single, globally resolved dependency graph, and a single build. The only alternative to that is basically to go down the tree, build some artifacts, then replace transitive dependencies with file dependencies and cross fingers that everything compiles up to the top. Of course this is completely unrealistic in the real world, unless you have millions of dollars to spend on rebuilding artifacts (and yes, some organizations do that, that’s the strategy for debian, for example, which has the "nice" side effect of having applications which have bugs which are not in the initial release, because all applications need to use the same dependency version).

That’s why I think the preferred solution, for security, is still to use precompiled binaries (which also would make builds faster in any case), but combine that with dependency verification: it’s a good tradeoff, which offers the right level of security, while not having to spend incredible amounts of money in rebuilding the entire world (also, it’s better for the planet). Note that this also guarantees trust, as your "custom built" binaries will clearly use different signatures, and possibly checksums, than what the users normally expect.

While we were talking about building from sources, we also forgot about one extremely important bit: there is no single dependency graph, even in a single project.

Building requires multiple dependency graphs

The most obvious way to illustrate that there’s no single dependency graph in a project is to thing about tests. When you compile your application, there shouldn’t be any test dependency on the compile classpath. When you compile your tests, then you’d get the dependencies of the application, plus the dependencies of your test framework, plus your additional test dependencies.

Therefore, you have at least 2 distinct dependency graphs:

the application compile classpath (in Gradle, it’s the compileClasspath configuration which represents that dependency graph)
the application test compile classpath (in Gradle, it’s the testCompileClasspath configuration which represents that dependency graph)

Maven itself makes a difference, with dependency scopes (compile, runtime, test, …). You can already see that in practice, we have many more dependency graphs: compile classpath, runtime classpath, test compile classpath, test runtime classpath, annotation processing path, functional tests compile/runtime classpath, etc. A project can literally have dozens of different dependency graphs.

More importantly, there can be conflicts in those graphs: for example, when you compile your tests, you may introduce a dependency which will accidentally trigger an upgrade of a dependency, so you would get a different dependency version during testing and actual run time! Similarly, your runtime dependencies can introduce transitive dependencies which would have the consequence of having different versions of dependencies at compile time and run time. Note that Gradle offers different ways to mitigate those real world problems, for example consistent resolution or version alignment.

Conclusion

I hope that after reading this, it becomes quite clear that there is no single dependency graph. Therefore, it’s a mistake to ask for "what are the dependencies of Micronaut", because the answer depends on the consumer. Not only does it depend on the consumer, but it also depends on either the order of dependencies (Maven for example), or the strategies being used to "force" dependency versions (which are also consumer dependent), or the kind of dependency graph which is resolved (runtime dependencies, compile dependencies). Of course, we didn’t mention more advanced features like optional dependencies, for which I like to remind that they are not optional, nor did we talk about more advanced, runtime based systems like OSGi which add another layer of complexity to the problem.

If you can take away one thing from this blog post, it’s to never ask "what are the dependencies of X" anymore: minimally, the question should be more targetted:

what are the dependencies that X need to compile?
what are the dependencies that X uses at runtime to run its test suite?

The question "what are the dependencies that X uses at runtime" is only valid for some applications (for example those which are not subject to platform dependent dependencies), not for libraries.

Gradle quickie: laziness

Tue, 24 May 2022 00:00:00 +0200

Yesterday, I wrote this tweet:

I got a surprisingly high number of answers, so I thought it would be a good idea to expand a bit on the topic.

Gradle introduced lazy APIs several years ago. Those APIs are mostly directed at plugin authors but some build authors may have to deal with them too. Lazy APIs are designed to improve performance, by avoiding to create tasks which would never be invoked during a build. While lots of users wouldn’t notice the difference between a build using lazy APIs and a build which doesn’t, in some ecosystems like Android or with large projects, this makes a dramatic difference. In other words, while Gradle’s performance is often praised, it’s easy to break performance by unintentionally trigerring configuration of tasks which shouldn’t.

Task configuration

The discussion was trigerred when I was doing a code review yesterday. I saw the following block:

tasks.withType(Test) {
    testLogging {
        showStandardStreams = true
        exceptionFormat = 'full'
    }
}

This block configures logging for all test tasks of the project. At first glance, this seems appropriate, but there’s this gotcha: you should use .configureEach:

tasks.withType(Test).configureEach {
    testLogging {
        showStandardStreams = true
        exceptionFormat = 'full'
    }
}

If you don’t, then all tasks of type Test will always be configured, even if you don’t call them in a build. In other words, lazy configuration is about only configuring tasks which are going to be invoked.

Unfortunately, there are no warnings about eager configuration, or "unnecessary" configuration in a build. If you use Build Scans, you can have insights about configuration and realize that, but casual users wouldn’t.

Similarly, this code:

test {
    testLogging {
        showStandardStreams = true
        exceptionFormat = 'full'
    }
}
----

Will configure the test task (not all test tasks) eagerly: even if the test task isn’t executed in a build, it would be configured. Now you see the problem: this configuration pattern has been there basically forever, so it’s hard to remove. To do lazy configuration, you have to write:

tasks.named('test') {
    testLogging {
        showStandardStreams = true
        exceptionFormat = 'full'
    }
}
----

Obviously, this isn’t as nice, DSL-wise. One thing you may wonder is why Gradle’s DSL default to the lazy version? In other words, why doesn’t it call the lazy version instead of the eager one?

It’s because of backwards compatiblity: because this pattern has been present since day one in Gradle, eager configuration is everywhere in older builds. If you search for configuration blocks in Stack Overflow, it’s very likely that you’ll end up copy and pasting eager configuration samples. But, as the name implies, lazy configuration has a different behavior than eager: in the lazy case, the configuration block is invoked only when the task is needed, either because it’s going to be executed, or that another task depends on its configuration to configure itself. In the eager case, configuration is executed immediately: unfortunately there are lots of builds which accidentally depend on this order of execution, so changing from eager to lazy could result in breaking changes!

What should you use?

The consequence is that there’s a mix of lazy and eager APIs in Gradle, and making the difference between what is going to trigger configuration or not isn’t obvious, even for Gradle experts. Let’s summarize a few patterns:

If you want to configure one particular task by name, you should write:

tasks.named("myTask") {
   // configure the task
}

tasks.named("myTask", SomeType) {
   // configure the task
}

If you want to all tasks of a particular type, you should write:

tasks.withType(SomeType).configureEach {
   // configure the task
}

If you want to create a new task, don’t use create, but register instead:

tasks.register("myTask", SomeType) {
    ...
}

In the DSL, the following code that you find in many tutorials would immediately create a task:

task hello {
   doLast {
       println "Hello!"
   }
}

So the correct way to do this is:

tasks.register("hello") {
    doLast {
         println "Hello!"
    }
}

Note that the return type of both calls is different: the eager version will return a Task, while the 2nd one returns a TaskProvider. This is the reason why upgrading plugins isn’t that trivial, since it’s a binary breaking change!

Task collections and implicit dependencies

In a previous blog post I explained that the provider API is the right way to handle implicit inputs. For example, you can pass directly a TaskProvider as an element of a file collection: Gradle would automatically resolve dependencies and trigger the configuration of that task, include it in the task graph and use its output as an input of the task you’re invoking.

Therefore, understanding lazy APIs means that you should understand when things are executed. In the example above, the call tasks.withType(Test) by itself does not configure anything. You can see it as a lazy predicate: it returns a live task collection, it’s a declaration of intent: "this models all tasks of type `Test`".

Therefore, the following blocks of code are strictly equivalent:

tasks.withType(Test) {
   // configure
}

tasks.withType(Test).each {
    // configure
}

def testTasks = tasks.withType(Test)
testTasks.each {
    // configure
}

In other words, the last version explains the "magic" behind the traditional Gradle DSL. The first line is lazy, returns a task collection, and it’s the fact of calling .each which triggers configuration of all tasks! Replace .each with .configureEach and you are now lazy!

Newer APIs like named are lazy from day one, but are not necessarily user friendly.

A Gradle puzzle

In effect, named is lazy in terms of configuration, but eager in terms of lookup: it will fail if the task that you’re looking for doesn’t exist. It’s a bit strange, since in Gradle everything is now supposed to be lazy, so you can’t know when a task is going to be available or not. As an illustration, let’s explore the following script (don’t write this in your own builds, this is for demonstration purposes!):

tasks.register("hello") {
   doLast {
       println "Hello,"
   }
}

tasks.named("hello") {
   doLast {
        println "World!"
   }
}

If you run gradle hello, then the output is what you expect:

> Task :hello
Hello,
World!

Now, invert the position of the 2 tasks:

tasks.named("hello") {
   doLast {
        println "World!"
   }
}

tasks.register("hello") {
   doLast {
       println "Hello,"
   }
}

and run again. Boom!

* Where:
Build file '/tmp/ouudfd/build.gradle' line: 1

* What went wrong:
A problem occurred evaluating root project 'ohnoes'.
> Task with name 'hello' not found in root project 'ohnoes'.

That is very unexpected: I think what most people would expect is, if any change, that the World! and Hello outputs would be exchanged. But because named eagerly searches for a task registed with a particular name, it fails if not found.

As a consequence, plugin authors who want to react to other plugins, or react to tasks which may be present or not, tend to use the following API instead:

tasks.matching { it.name == 'hello' }.configureEach {
    doLast {
        println "World!"
   }
}

tasks.register("hello") {
   doLast {
       println "Hello,"
   }
}

Now let’s run our hello task:

> Task :hello
World!
Hello,

Yay! No failure anymore, and the output is in the order we expected. Problem solved, right?

Well, not so fast. You’ve used configureEach, so everything should be lazy, right? Sorry, nope: the matching API is an old, eager API! Actually, if you look at what the predicate uses, it becomes obvious:

// T is a Task!
TaskCollection<T> matching(Spec<? super T> var1)

Because it works on Task instances, it needs to create and configure the tasks so that you can run an arbitrary predicate on them!

That’s why if you have to write things like this, you must guard calls to matching with a withType before, which will restrict the set of tasks which will be configured. For example:

tasks.withType(Greeter).matching { it.name == 'hello' }.configureEach {
   messages.add("World!")
}

tasks.register("hello", Greeter) {
   messages.add("Hello,")
}

Of course the example is a bit stupid, but it makes sense when you’re not the one in control of when a task is configured or even if you don’t know if it will ever be.

Unfortunately, Gradle doesn’t provide an API which is fully lazy and lenient to tasks being present or not. If you simply want to configure a task, that is not a big deal since you can simply use configureEach:

tasks.configureEach {
    if (it.name == 'hello') { ... }
}

This is fine because the configuration block will be called for each task being configured. However, this configureEach block is a configurer, not a predicate, so you can’t use it as an input to another task:

tasks.named("md5") {
    inputFiles.from(tasks.named("userguide"))
}

The code above would fail if the userguide task doesn’t exist before the md5 task is configured…

Conclusion

In this blog post, I have explained why you should use the new lazy APIs instead of their eager counterparts. I have also described that while they are more verbose, they make it possible to have faster builds by avoiding configuration of tasks which would not be executed. However, Gradle doesn’t warn you if you eagerly configure tasks, and it’s easy to shoot yourself in the foot. Some would blame the docs, some would blame the APIs.

As a former Gradler, I would blame none of those: the docs are here, and changing the APIs to be lazy everywhere is either a binary breaking change (return type of methods which create instead of register), or a behavior change (deferred configuration vs immediate configuration). This makes it particularly complicated to upgrade builds without pissing off a number of users!

Astrophotographie: rendez-vous sur Twitch !

Tue, 3 May 2022 00:00:00 +0200

Nous ne sommes pas que des développeurs !

Ma passion, en dehors du développement, c’est l’astronomie. Depuis quelques années, je me suis lancé dans l’astrophotographie: je ne fais quasiment plus que ça. Pour un développeur comme moi, c’est assez intéressant de constater que lorsque je poste sur Twitter une photo que j’ai faite, j’ai bien souvent plus de réponses et de likes que sur mes tweets professionnels (ce qui est parfois vexant, lol !).

Des lives sur Twitch

Edit: C'est fait ! L'expérience fut enrichissante pour moi, retrouvez le replay sur Youtube.

J’ai, au final, souvent les mêmes questions qui reviennent:

Qu’est ce que c’est ?
Quel matériel tu utilises ?
Combien de fois ça grossit ?
Est-ce que ce sont de vraies couleurs ?
Combien de temps de pose ?

et bien d’autres !

Alors, l’an dernier, je me suis lancé ce défi de faire un talk dans une conférence de développeurs (Devoxx) sur le sujet des "miracles du logiciel" en termes d’astrophotographie. Malheureusement, le talk n’a pas été retenu, mais j’ai conservé en tête l’idée de présenter quelque chose.

Alors, envie de savoir comment on fait des photos comme celle-ci ?

Figure 1. The flaming star nebula

Aujourd’hui, je vous annonce donc un premier live sur Twitch pour parler d’astrophotographie ! Je dis premier, parce que je pense qu’il y a de quoi en faire plusieurs avant d’avoir fait le tour du sujet.

Attention, ça sera sans prétention, pas aussi préparé qu’un talk en conférence. Ca sera aussi mon tout premier live et donc probablement plein de problèmes techniques, mais il faut bien se lancer un jour !

J’annonce donc le jeudi 12 mai à 20h sur ma chaîne Twitch.

Le premier sujet sera autour de mon setup photographique: quel matériel j’utilise, les principes de base de l’acquisition photo. Attention, il ne s’agira pas d’une présentation générique sur l’astrophoto, mais bien d’une présentation spécifique à mon matériel, soupoudrée de détails sur comment ça fonctionne en règle générale.

En fonction du succès et/ou de ce que j’arrive à couvrir, d’autres lives seront programmés.

A bientôt !

Conditional dependencies with Gradle

Mon, 21 Mar 2022 00:00:00 +0100

Introduction

If you ever wrote a Gradle plugin for a framework (e.g Micronaut) or a plugin which needs to add dependencies if the user configures a particular flag, then it is likely that you’ve faced some ordering issues.

For example, imagine that you have this DSL:

micronaut {
    useNetty = true
}

Obviously, at some point in time, you have to figure out if the property useNetty is set in order to transparently add dependencies. A naive solution is to use the good old afterEvaluate block. Many plugins do this:

afterEvaluate {
    dependencies {
        if (micronaut.useNetty.get()) {
            implementation("io.netty:netty-buffer:4.1.75.Final")
        }
    }
}

The problem is that while afterEvaluate seems to fix the problem, it’s just a dirty workaround which defers the problem to a later stage: depending on the plugins which are applied, which themselves could use afterEvaluate, your block may, or may not, see the "final" configuration state.

In a previous post, I introduced Gradle’s provider API. In this post, we’re going to show how to use it to properly fix this problem.

Using providers for dependencies

Let’s start with the easiest. It’s a common requirement of a plugin to provide the ability to override the version of a runtime. For example, the checkstyle plugin would, by default, use version of checkstyle by convention, but it would still let you override the version if you want to use a different one.

Micronaut provides a similar feature:

micronaut {
    version = "3.3.1"
}

The Micronaut dependencies to be added on the user classpath depend on the value of the version in our micronaut extension. Let’s see how we can implement this. Let’s create our Gradle project (we’re assuming that you have Gradle 7.4 installed):

$ mkdir conditional-deps && cd conditional-deps
$ gradle init --dsl groovy \
   --type java-library \
   --package me.champeau.demo \
   --incubating \
   --test-framework junit-jupiter

Now we’re going to create a folder for our build logic, which will contain our plugin sources:

$ mkdir -p build-logic/src/main/groovy/my/plugin

Let’s update the settings.gradle file to include that build logic:

settings.gradle

pluginManagement {
    // include our plugin
    includeBuild "build-logic"
}
rootProject.name = 'provider-dependencies'
include('lib')

For now our plugin is an empty shell, so let’s create its build.gradle file so that we can use a precompiled script plugin.

build-logic/build.gradle

plugins {
    id 'groovy-gradle-plugin'
}

Now let’s define our extension, which is simply about declaring an interface:

build-logic/src/main/groovy/my/plugin/MicronautExtension.groovy

package my.plugin

import org.gradle.api.provider.Property

interface MicronautExtension {
    Property<String> getVersion()
}

It’s now time to create our plugin: precompiled script plugins are a very easy way to create a plugin, simply by declaring a file in build-logic/src/main/groovy which name ends with .gradle:

build-logic/src/main/groovy/my.plugin.gradle

import my.plugin.MicronautExtension

def micronautExtension = extensions.create("micronaut", MicronautExtension) (1)
micronautExtension.version.convention("3.3.0")                              (2)

Create our extension, named "micronaut"
Assign a default value to the "version" property

By convention, our plugin id will be my.plugin (it’s derived from the file name). Our plugin is responsible for creating the extension, and it assigns a convention value to the version property: this is the value which is going to be used if the user doesn’t declare anything explicitly.

Then we can use the plugin in our main build, that is, in the lib project:

lib/build.gradle

plugins {
    // Apply the java-library plugin for API and implementation separation.
    id 'java-library'
    // And now apply our plugin
    id 'my-plugin'
}

micronaut {
   // empty for now
}

If we look at the lib compile classpath, it will not include any Micronaut dependency for now:

$ ./gradlew lib:dependencies --configuration compileClasspath

------------------------------------------------------------
Project ':lib'
------------------------------------------------------------

compileClasspath - Compile classpath for source set 'main'.
+--- org.apache.commons:commons-math3:3.6.1
\--- com.google.guava:guava:30.1.1-jre
     +--- com.google.guava:failureaccess:1.0.1
     +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
     +--- com.google.code.findbugs:jsr305:3.0.2
     +--- org.checkerframework:checker-qual:3.8.0
     +--- com.google.errorprone:error_prone_annotations:2.5.1
     \--- com.google.j2objc:j2objc-annotations:1.3

Our goal is to add a dependency which is derived from the version defined in our Micronaut extension, so let’s do this. Edit our build-logic plugin:

build-logic/src/main/groovy/my.plugin.gradle

import my.plugin.MicronautExtension

def micronautExtension = extensions.create("micronaut", MicronautExtension)
micronautExtension.version.convention("3.3.0")

dependencies {
    implementation micronautExtension.version.map {
        v -> "io.micronaut:micronaut-core:$v"
    }
}

Now let’s run our dependencies report again:

$ ./gradlew lib:dependencies --configuration compileClasspath

> Task :lib:dependencies

------------------------------------------------------------
Project ':lib'
------------------------------------------------------------

compileClasspath - Compile classpath for source set 'main'.
+--- org.apache.commons:commons-math3:3.6.1
+--- io.micronaut:micronaut-core:3.3.0
|    \--- org.slf4j:slf4j-api:1.7.29
\--- com.google.guava:guava:30.1.1-jre
     +--- com.google.guava:failureaccess:1.0.1
     +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
     +--- com.google.code.findbugs:jsr305:3.0.2
     +--- org.checkerframework:checker-qual:3.8.0
     +--- com.google.errorprone:error_prone_annotations:2.5.1
     \--- com.google.j2objc:j2objc-annotations:1.3

Victory! Now we can see our micronaut-core dependency. How did we do this?

Note that instead of using afterEvaluate, what we did is adding a dependency, but instead of using the traditional dependency notation, we used a provider: the actual dependency string is computed only when we need it. We can check that we can actually configure the version via our extension by editing our build file:

lib/build.gradle

micronaut {
   version = "3.3.1" // override the convention
}

$ ./gradlew lib:dependencies --configuration compileClasspath

> Task :lib:dependencies

------------------------------------------------------------
Project ':lib'
------------------------------------------------------------

compileClasspath - Compile classpath for source set 'main'.
+--- org.apache.commons:commons-math3:3.6.1
+--- io.micronaut:micronaut-core:3.3.1
|    \--- org.slf4j:slf4j-api:1.7.29
\--- com.google.guava:guava:30.1.1-jre
     +--- com.google.guava:failureaccess:1.0.1
     +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
     +--- com.google.code.findbugs:jsr305:3.0.2
     +--- org.checkerframework:checker-qual:3.8.0
     +--- com.google.errorprone:error_prone_annotations:2.5.1
     \--- com.google.j2objc:j2objc-annotations:1.3

Maybe add, maybe not!

In the previous example, we have systematically added a dependency, based on the version defined in the extension. What if we want to add a dependency if a property is set to a particular value? For this purpose, let’s say that we define a runtime property which will tell what runtime to use. Let’s add this property to our extension:

build-logic/src/main/groovy/my/plugin/MicronautExtension.groovy

package my.plugin

import org.gradle.api.provider.Property

interface MicronautExtension {
    Property<String> getVersion()
    Property<String> getRuntime()
}

Now let’s update our plugin to use that property, and add a dependency based on the value of the runtime property:

build-logic/src/main/groovy/my.plugin.gradle

import my.plugin.MicronautExtension

def micronautExtension = extensions.create("micronaut", MicronautExtension)
micronautExtension.version.convention("3.3.0")

dependencies {
    implementation micronautExtension.version.map { v ->
        "io.micronaut:micronaut-core:$v"
    }

    implementation micronautExtension.runtime.map { r ->
        switch(r) {
            case 'netty':                                                   (1)
                return "io.netty:netty-buffer:4.1.75.Final"
            case 'tomcat':
                return "org.apache.tomcat.embed:tomcat-embed-core:10.0.18"  (2)
            default:
                return null                                                 (3)
        }
    }
}

Add a dependency if the runtime is set to netty
Add a dependency if the runtime is set to tomcat
But do nothing if the runtime isn’t set

The trick, therefore, is to return null in the provider in case no dependency needs to be added. So let’s check first that without declaring anything, we don’t have any dependency added:

$ ./gradlew lib:dependencies --configuration compileClasspath

> Task :lib:dependencies

------------------------------------------------------------
Project ':lib'
------------------------------------------------------------

compileClasspath - Compile classpath for source set 'main'.
+--- org.apache.commons:commons-math3:3.6.1
+--- io.micronaut:micronaut-core:3.3.1
|    \--- org.slf4j:slf4j-api:1.7.29
\--- com.google.guava:guava:30.1.1-jre
     +--- com.google.guava:failureaccess:1.0.1
     +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
     +--- com.google.code.findbugs:jsr305:3.0.2
     +--- org.checkerframework:checker-qual:3.8.0
     +--- com.google.errorprone:error_prone_annotations:2.5.1
     \--- com.google.j2objc:j2objc-annotations:1.3

Now let’s switch to use tomcat:

lib/build.gradle

micronaut {
   version = "3.3.1"
   runtime = "tomcat"
}

$ ./gradlew lib:dependencies --configuration compileClasspath

> Task :lib:dependencies

------------------------------------------------------------
Project ':lib'
------------------------------------------------------------

compileClasspath - Compile classpath for source set 'main'.
+--- org.apache.commons:commons-math3:3.6.1
+--- io.micronaut:micronaut-core:3.3.1
|    \--- org.slf4j:slf4j-api:1.7.29
+--- org.apache.tomcat.embed:tomcat-embed-core:10.0.18
|    \--- org.apache.tomcat:tomcat-annotations-api:10.0.18
\--- com.google.guava:guava:30.1.1-jre
     +--- com.google.guava:failureaccess:1.0.1
     +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
     +--- com.google.code.findbugs:jsr305:3.0.2
     +--- org.checkerframework:checker-qual:3.8.0
     +--- com.google.errorprone:error_prone_annotations:2.5.1
     \--- com.google.j2objc:j2objc-annotations:1.3

Note how the dependency on Tomcat is added!

More complex use cases are supported!

We’ve shown how to add a dependency and derive the dependency notation from the version defined in our extension. We’ve then seen how we could add a dependency, or not, based on the value of an extension: either return a supported dependency notation, or null if nothing needs to be added.

Gradle actually supports more complex cases, that I will let as an exercise to the reader. For example:

adding a dependency provider and configure its rich version (see DependencyHandler#addProvider).
adding a list of dependencies, instead of a single dependence (see Configuration#getDependencies and DependencySet#addAllLater).
computing a dependency from two or more providers (see Provider#zip).

Conclusion

In this post, we’ve seen how to leverage Gradle’s provider API to properly implement plugins which need to add dependencies conditionally. This can either mean that they need to add dependencies which version depend on some user configuration, or even full dependency notations which depend on configuration. The interest of using the provider API again lies in the fact that it is lazy and therefore is (largely) immune to ordering issues: instead of relying on hooks like afterEvaluate which come with a number of drawbacks (reliability, ordering, interaction with other plugins), we rely on the fact that it’s only when a value is needed that it is computed. At this moment, we know that the configuration is complete, so we can guarantee that our dependencies will be correct.

Using the Micronaut Version Catalog

Tue, 8 Feb 2022 00:00:00 +0100

Introduction

With the release of Gradle 7.4, Micronaut users now have an interesting option to manage their dependencies: using Gradle’s version catalogs. Indeed, for a few releases already, Micronaut has shipped its own version catalog alongside its BOM.

Let’s explore how to use it and what’s the benefit.

What is a version catalog?

In a nutshell, a version catalog allows centralizing dependency versions in a single place. Instead a build script, a typical dependency declaration looks like this:

dependencies {
    implementation("org.apache.slf4j:slf4j-api:1.7.25")
}

With a version catalog, the declaration looks like this:

dependencies {
    implementation(libs.slf4j)
}

And the dependency coordinates are defined in the gradle/libs.versions.toml file:

[versions]
slf4j = "1.7.25"

[libraries]
slf4j = { module = "org.apache.slf4j", version.ref = "slf4j" }

There are a couple of advantages in doing so:

dependency versions are centralized in this TOML file
the catalogs create "type safe accessors" which are completed by the IDE (although to my knowledge completion is only supported by IntelliJ IDEA with the Kotlin DSL)

You can read a more complete description about version catalogs in this blog post I wrote a few months ago.

The Micronaut version catalog

In addition, frameworks like Micronaut can publish version catalogs, which are then usable in your projects. You can then think of the Micronaut version catalog as a list of dependencies to pick up from: you don’t have to think about a version to choose, you can simply use the "recommendation" from Micronaut, but you don’t have to remember the dependency coordinates either.

Importing the Micronaut version catalog

Let’s start with a project that you can generate using the Micronaut CLI:

mn create-app catalog

(alternatively, download the project using Micronaut Launch)

Open the generated project and update the Gradle version by changing the gradle/gradle-wrapper.properties file:

distributionUrl=https\://services.gradle.org/distributions/gradle-7.4-bin.zip

Now, in order to import the Micronaut version catalog, add this to your settings.gradle file:

settings.gradle

dependencyResolutionManagement {
    repositories {
        mavenCentral()
    }
    versionCatalogs {
        create("mn") {
            from("io.micronaut:micronaut-bom:${micronautVersion}")
        }
    }
}

Here, we’re creating a new version catalog called mn. Internally, Gradle will automatically download the catalog which is published at the same GAV coordinates as its BOM as a TOML file and expose it to your build scripts.

Let’s open our build.gradle file. By default it defines the following dependencies:

dependencies {
    annotationProcessor("io.micronaut:micronaut-http-validation")
    implementation("io.micronaut:micronaut-http-client")
    implementation("io.micronaut:micronaut-jackson-databind")
    implementation("io.micronaut:micronaut-runtime")
    implementation("jakarta.annotation:jakarta.annotation-api")
    runtimeOnly("ch.qos.logback:logback-classic")
    implementation("io.micronaut:micronaut-validation")
}

Now, we can replace this with the following:

dependencies {
    annotationProcessor(mn.micronaut.http.validation)
    implementation(mn.micronaut.http.client)
    implementation(mn.micronaut.jackson.databind)
    implementation(mn.micronaut.runtime)
    implementation(mn.jakarta.annotation.api)
    runtimeOnly(mn.logback)
    implementation(mn.micronaut.validation)
}

What happened here? Basically, we replaced hardcoded dependency coordinates with references to the mn version catalog. It’s particularly interesting if you are using the Kotlin DSL as I mentioned earlier, because in this case, the dependency notations are type-safe: you can’t make a typo in dependency coordinates, and you get completion:

Nice!

Future work

Version catalogs will probably be enabled by default in future releases of Micronaut, which means that projects created via Micronaut Launch or the CLI tool would automatically use the catalog, so you don’t have to do the conversion described in this blog post. Stay tuned!

Understanding Gradle plugins: the provider API

Mon, 24 Jan 2022 00:00:00 +0100

Introduction

Last week, a colleague of mine pinged me about a problem he was facing in a new Micronaut module, with Jetbrains' Grammar-Kit plugin, a plugin which integrates with JFlex. We discovered a couple of issues:

the plugin is adding repositories transparently to the build, in particular Jitpack.io and an internal Jetbrains mirror, which is a very bad practice as it is introducing a security risk, and could also lead to build reproducibility problems
the plugin isn’t compatible with Gradle’s lazy configuration API, making the build slower than it could be, by forcing the creation of tasks which do not necessarily need to be called

Last but not least, we also discovered that the JFlex API itself has problems: it’s using shared mutable state (here static state) for configuration, making it inherently not thread safe. Because what we had to do was pretty simple (take a jflex file and generate a lexer from it), we took advantage of this to write our own Gradle plugin to handle calls to JFlex.

Therefore, the question I got from my colleague was legitimate:

How do I link the task to run in the right phase, create a proper output dir, and add it to the java compile source set? Is there a good example plugin that does source generation that I could look at?

In this blog post we’re going to answer those questions and show how Gradle elegantly solves all the above problems.

Anatomy of a plugin

In Gradle, a plugin essentially consists in registering a number of tasks (things which execute actions, like compiling sources), or so-called extensions (exposed to the user in the form of a DSL for configuring the build). There isn’t a difference between what a plugin can do and what you can do in a build script, however, as soon as you have things which go beyond configuration, it’s a good idea to move things into a plugin. The nice thing is that creating a plugin in Gradle is quite straightforward, and doesn’t even require you to publish the plugin on Maven Central or the Gradle Plugin Repository: everything can be local to your project.

In our case, we want to create a task which is going to invoke the JFlex library to parse some .jflex files and generate .java files, so let’s do it!

First of all, we’re going to create a directory for our plugin, a jflex-plugin directory at the root of our existing project. We’re also going to create 2 files in that directory:

flex-plugin/settings.gradle

rootProject.name = 'jflex-plugin'

and

flex-plugin/build.gradle

plugins {
    id 'java-gradle-plugin'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'de.jflex:jflex:1.8.2'
}

gradlePlugin {
    plugins {
        jflex {
            id = 'io.micronaut.internal.jflex'
            implementationClass = 'io.micronaut.internal.jflex.JFlexPlugin'
        }
    }
}

What does this already tell us? First of all, that our plugin is an independent Gradle project: it lives in the same repository as our main project, but it’s really an independent build. Second, it defines a java-gradle-plugin, which is the Gradle way of saying "this is a plugin for Gradle, written in Java".

This plugin has an implementation dependency on jflex, and it declares the id of the plugin, as well as its implementation class. This is all the boilerplate you have to write to create a plugin, really.

Because our main build is going to use this plugin, we also have to edit the main build settings.gradle file to include that plugin:

settings.gradle

includeBuild "jflex-plugin"

This makes it possible to apply our JFlex plugin in our project:

build.gradle

plugins {
    id 'java-library'
    id 'io.micronaut.internal.jflex'
}

We use the plugin id that we have defined in our plugin descriptor. This is how Gradle knows how to wire things together: by using this plugin request, it will automatically trigger the jflex-plugin build.

Note	It wouldn’t bother building the plugin if it wasn’t used at all, which would be the case, for example, if we had multiple subprojects and that only one of them uses the plugin: depending on what we build, we may need to build the plugin or not.

The provider API

For the sake of learning, we’ll start our implementation with what is most "natural" to developers: if you are familiar with Maven, you’ll think of a Mojo. If you are familiar with Ant, it’s a task. Similarly, in Gradle, the unit which is responsible for executing an action is called a task. In our case, we want a task which is going to read JFlex files and generate sources. Here’s its skeleton:

jflex-plugin/src/main/java/io/micronaut/internal/jflex/JFlexTask.java

@CacheableTask                                                  // (1)
public abstract class JFlexTask extends DefaultTask {           // (2)

    @InputDirectory                                             // (3)
    @PathSensitive(PathSensitivity.RELATIVE)
    public abstract DirectoryProperty getSourceDirectory();

    @OutputDirectory                                            // (4)
    public abstract DirectoryProperty getOutputDirectory();

    @TaskAction                                                 // (5)
    public void generateSources() {
      // call JFlex library
    }
}

This annotation tells Gradle that the result of executing this task can be cached
every Gradle task needs to extend the DefaultTask type
the input of our task is a directory containing JFlex files
the output of our task is going to be a directory containing generated Java files
this is the main task action, which is going to call JFlex

Let’s explore a bit how this task is defined.

First, note how our task is defined abstract: this is because we will let Gradle generate boilerplate code for us, in particular how to inject the input and output properties. We’ll see later more reasons why it’s interesting to let Gradle to this for you, but for now the obvious reason is that it reduces the amount of code you have to write: you don’t need to know how to create a DirectoryProperty: Gradle will do it for you.

Second, note how we are using DirectoryProperty as the type for our input and output properties. This is a very important Gradle type, which belongs to the so-called "provider API" or, as you can sometimes read, the "lazy API". Most of the ordering problems that earlier versions of Gradle had are fixed by this API, so use it!

One thing we can notice is that it’s strongly typed: to declare an input directory, we don’t define the property as a File or Path: it’s a directory, which helps both Gradle and users understand what you are supposed to give as an input: if the property is set to a regular file, then Gradle can provide a reasonable error message explaining that it expected a directory instead.

It’s time to introduce how you could use this type in a build script:

build.gradle

tasks.register("generateLexer", JFlexTask) {
    sourceDirectory.set(layout.projectDirectory.dir('src/main/jflex')) // (1)
    outputDirectory.set(layout.buildDirectory.dir('generated/jflex'))  // (2)
}

set the input directory to src/main/jflex
set the output directory to build/generated/jflex

It may sound a bit complicated to declare, especially if you were used to the following syntax:

build.gradle

tasks.register("generateLexer", JFlexTask) {
    sourceDirectory = file("src/main/jflex")
    outputDirectory = file("build/generated/jflex')
}

Note	`register` is the new `create`: you should never use create anymore, as it eagerly creates tasks, which means configuring them even if they won’t participate in the task graph, while register is lazy: if a task needs to be executed, and only if, it’s going to be configured.

Interestingly, this syntax outputDirectory = file("build/generated/jflex') is still valid with our properties and would lead to the same result if executed. It’s simpler, so why should you bother with the more complex syntax? To understand this, let’s focus on the output directory, which makes it more obvious what is going on: compare build/generated/jflex with layout.buildDirectory.dir('generated/jflex').

In the 1st case, the output directory is hardcoded to the build/generated/jflex directory. In the 2nd case, the output directory is derived from the location of the build directory. It means that if, for some reason, your build is configured to use a different output directory than the conventional build directory, say target (as in Maven). In the 1st case, the output directory of the task would be build/generated/jflex, so it would be writing to the wrong directory. In the 2nd case, the output would be correctly wired to target/generated/jflex.

Some smart Gradle users might think they could workaround the problem by using file("$buildDir/generated/jflex") instead. That’s better, but not sufficient, because the result depends on when this is called: if the build directory is changed after the task is configured, then you’d get the wrong result, which is why lots of users start to randomly add afterEvaluate to workaround such problems.

Outcome #1: The provider API solves ordering issues and avoids spurious calls to afterEvaluate.

Convention plugins

In the beginning of this blog post, I mentioned that what we want to avoid users to create tasks directly in their build scripts: this is a sign that the code should be moved to a plugin. This is exactly what we’re going to do, so instead of asking the user to declare the task, we’re going to do it for them. It’s time to create our plugin class:

jflex-plugin/src/main/java/io/micronaut/internal/jflex/JFlexPlugin.java

public class JFlexPlugin implements Plugin<Project> {                                  // (1)
    @Override
    public void apply(Project project) {
        project.getPluginManager().apply(JavaPlugin.class);                            // (2)
        JavaPluginExtension javaExt = project.getExtensions()
           .getByType(JavaPluginExtension.class);                                      // (3)
        TaskProvider<JFlexTask> generateLexer = project.getTasks()
            .register("generateLexer", JFlexTask.class, task -> {                      // (4)
               task.setGroup(LifecycleBasePlugin.BUILD_GROUP);
               task.setDescription("Generates lexer files from JFlex grammar files.");
               task.getSourceDirectory().convention(
                    project.getLayout().getProjectDirectory().dir("src/main/jflex")    // (5)
               );
               task.getOutputDirectory().convention(
                    project.getLayout().getBuildDirectory().dir("generated/jflex")     // (6)
               );
        });
        // Register the output of the JFlex task as generated sources
        javaExt.getSourceSets()
                .getByName(SourceSet.MAIN_SOURCE_SET_NAME)
                .getJava()
                .srcDir(generateLexer);                                                // (7)
    }
}

Declare a project scoped plugin
This plugin will contribute Java sources, so it depends on the Java plugin, let’s apply it
The Java plugin defines a Java extension that we’re going to need
Registers our generateLexer task
Defines the conventional (default) location of JFlex source files
Defines the conventional (default) location of generated sources
Defines that the output of the task are Java files which need to be compiled

I recommend writing plugins in plain Java, but you could use Groovy or Kotlin. It makes things a bit more verbose, but they are clearer and "DSL magic" free. Let’s explore what the plugin is doing. First of all, it’s a project plugin, which basically means it’s a plugin which is supposed to be applied on a project build file, so typically a build.gradle file. There are other kinds of plugins in Gradle, which I won’t cover in this post.

For the most part, the plugin does exactly what we had in the build script: it registers a task, gives it a description, but more importantly, it sets the conventional values of inputs and outputs. Note how I used the convention method to set the input directory, instead of the set method that we used in the build script: while using both would work, there’s a semantic difference between the two: in a plugin, you most likely want to set the convention value, which is the value which is used by default, if the user says nothing.

Our plugin does one more thing, that we didn’t cover yet: wiring the task in the "lifecycle", as my colleague asked. The notion of "lifecyle" doesn’t really make sense in Gradle, and most likely comes from the Maven mindset, where things are defined via a "lifecyle". I already covered this topic in this blog post, but here’s the major difference: in Gradle, everything declares its inputs, and the tool is responsible for wiring things properly, so that you don’t have to execute redundant work.

Here, the legitimate question is: my plugin generates some sources, but they need to be compiled, and therefore available to my production code, how can I do that? This is where the JavaPluginExtension comes into play. In fact, our plugin doesn’t work independently: it assumes that we’re programming Java, and it assumes that we can compile Java sources. For this, we can actually make the assumption explicit, by requiring that the JavaPlugin is applied. When this plugin is applied, it defines a JavaPluginExtension, which declares source sets. In particular, it defines the Java source sets (main and test), which are the sources which are compiled.

The shift in mindset is, therefore, not to wonder how to compile the generated sources and put them "on the classpath", like you’d do in Maven, but simply explain that there’s another directory of sources to consider.

This is exactly what our plugin is doing:

javaExt.getSourceSets()
    .getByName(SourceSet.MAIN_SOURCE_SET_NAME)
    .getJava()
    .srcDir(generateLexer);

This says "please add the output of the generateLexer task as a source directory". Which is semantically much more powerful. The magic is that because the generateLexer task defines an output directory, now we just said that this output directory contains Java classes. And any task which requires Java sources will automatically trigger the execution of our generateLexer task: we don’t have to define the relationship explicitly!

Outcome #2: Gradle models relationships using domain objects which are shared between plugins. It can infer dependencies thanks to those objects.

In other words, because the input of the Java compilation task is a source set, and that source set defines that as an input, it has a directory which is generated by the generateLexer task, Gradle knows that before compiling, it needs to call that generateLexer task. Any other task using the source set as an input will do the same: it avoids duplication of code and hard wiring!

Using the worker API

At this stage, we’re pretty much done with the wiring, but we still miss the actual implementation of the task. This could be left as an exercise to the reader, but there’s actually an interesting aspect of the Gradle APIs to cover.

If you remember, I mentioned in the introduction of the blog post that the JFlex API uses a mix of static state and instance state to configure itself. This isn’t nice, as it basically means that the API is not thread-safe: if, for some reason, we have multiple tasks generating sources (for example if we have multiple jflex directories, or different projects having jflex sources), then we can’t safely generate sources in parallel!

This is quite problematic, but Gradle provides a simple workaround for this: the worker API. The worker API allows a number of things, but in particular it permits executing code in a different process, or, more lightweight, in an isolated classloader. The second option is good for us, because static state in Java is only as static as it is in a given classloader: if 2 "identical" classes are loaded in 2 different classloaders, then they both have their independent static state. We’re going to use this to properly isolate execution of our code.

As a consequence, executing JFlex will be slightly more complicated, but as usual in programming, it’s only one level of indirection. Instead of having our task directly invoke JFlex, we need to create a class which is going to invoke JFlex.

To use the worker API, we need to inject the so-called WorkerExecuter in our task:

jflex-plugin/src/main/java/io/micronaut/internal/jflex/JFlexTask.java

@CacheableTask
public abstract class JFlexTask extends DefaultTask {

    @InputDirectory
    @PathSensitive(PathSensitivity.RELATIVE)
    public abstract DirectoryProperty getSourceDirectory();

    @OutputDirectory
    public abstract DirectoryProperty getOutputDirectory();

    @Inject
    protected abstract WorkerExecutor getWorkerExecutor();

    @TaskAction
    public void generateSources() {
        // We're using classloader isolation, because the JFlex API
        // uses static state!
        getWorkerExecutor()
                .classLoaderIsolation()
                .submit(JFlexAction.class, params -> {
                    params.getSourceDirectory().set(getSourceDirectory());
                    params.getSourceFiles().from(getSourceDirectory());
                    params.getOutputDirectory().set(getOutputDirectory());
                });
    }
}

Note again how you don’t need to care how to get a WorkerExecuter: just tell Gradle you need it and voilà! When using the worker API, the task action basically becomes an empty shell, which just configures how actual execution should happen. In this case, we declare classloader isolation, as well as the inputs of the action, which is going to be executed in isolation.

The action class basically consists of calling the JFlex API:

jflex-plugin/src/main/java/io/micronaut/internal/jflex/JFlexAction.java

public abstract class JFlexAction implements WorkAction<JFlexAction.Parameters> {
    public interface Parameters extends WorkParameters {
        DirectoryProperty getSourceDirectory();
        ConfigurableFileCollection getSourceFiles();
        DirectoryProperty getOutputDirectory();
    }

    @Override
    public void execute() {
        OptionUtils.setDefaultOptions();
        Path sourcePath = getParameters().getSourceDirectory().getAsFile().get().toPath();
        File outputDirectory = getParameters().getOutputDirectory().getAsFile().get();
        OptionUtils.setDir(outputDirectory);
        Options.dump = false;
        Options.encoding = StandardCharsets.UTF_8;
        Options.no_backup = true;
        getParameters().getSourceFiles()
                .getAsFileTree()
                .getFiles()
                .forEach(jflexFile -> generateSourceFileFor(jflexFile, outputDirectory, sourcePath));
    }

    private void generateSourceFileFor(File jflexFile, File outputDirectory, Path sourcePath) {
        String relativePath = sourcePath.relativize(jflexFile.getParentFile().toPath()).toString();
        OptionUtils.setDir(new File(outputDirectory, relativePath));
        new LexGenerator(jflexFile).generate();
    }
}

The action declares its inputs with the WorkParameters interface and the code which is going to be executed in an isolated classloader lives in the execute method. You can see how it uses static state (OptionsUtils.setDefaultOptions(), Options.dump, …). The worker API lets us workaround what should probably be considered as a bug in JFlex!

Outcome #3: The Gradle Worker API lets you isolate your task code in classloaders or even separate worker processes.

More about the provider API

Before closing this blog post, I want to give you a bit more insights about the provider API. I already mentioned that one of the main advantages is that it solves ordering issues, by being fully lazy.

One of the most interesting aspects of this API is value derivation. To understand this concept, let’s imagine a Greeter task which is responsible for saying hello:

abstract class Greeter extends DefaultTask {
    @Input
    public abstract Property<String> getUser()

    @Input
    public abstract Property<String> getIntro()

    @Input
    public abstract Property<String> getOutro()

    @TaskAction
    public void sayHello() {
        String user = getUser().get();
        String intro = getIntro().get();
        String outro = getOutro().get();
        System.out.println(intro + user + outro);
    }
}

we can register a task which says hello in English by doing this:

tasks.register("sayHello", Greeter) {
   intro = "Hello, "
   user = "Cédric"
   outro = "!"
}

And another one which says hello in French:

tasks.register("direBonjour", Greeter) {
   intro = "Bonjour "
   user = "Cédric"
   outro = " !"
}

It’s a bit annoying that we have to repeat the user declaration in both tasks, and the rule "in french, exclamation mark must be preceeded with a space" doesn’t need to be known to the user. To avoid this redundancy, we’re going to write a plugin which makes all this more convenient.

First, we’re going to create an extension, which is going to hold what is relevant for user configuration: the name of the person to greet and what outro we want to use.

interface GreetingExtension {
    Property<String> getUser()
    Property<String> getOutro()
}

Again we don’t have to provide an implementation for this, Gradle knows how to create a Property<String>. This extension simply needs to be created by our plugin:

GreetingExtension extension = project.getExtensions().create("greeting", GreetingExtension.class);
extension.getOutro().convention("!");

It’s interesting to see that our DSL will only expose "user" and "outro", but not the intro, which is actually dependent on the language. We can also set a conventional value on the extension itself. The plugin can then register both tasks for us:

tasks.register("sayHello", Greeter.class, task -> {
   task.getIntro().convention("Hello, ");
   task.getUser().convention(extension.getUser());
   task.getOutro().convention(extension.getOutro());
});
tasks.register("direBonjour", Greeter.class, task -> {
   task.getIntro().convention("Bonjour ");
   task.getUser().convention(extension.getUser());
   task.getOutro().convention(extension.getOutro().map(o -> " " + o));
});

Now you see the interest of using the provider API: for the english case, the task is going to use the outro value directly, while for the french version, by default, it’s going to compute a derived value.

The user will configure the tasks via the extension:

greeting {
    user = "Cédric"
}

Calling sayHello will output:

Hello, Cédric!

While calling direBonjour will output:

Bonjour Cédric !

Should the user configure a different outro, the outputs would be different:

greeting {
    user = "Cédric"
    outro = "!!!"
}

results in this english version:

Hello, Cédric!!!

While the french one is:

Bonjour Cédric !!!

BUT, because we defined the convention value of outro of the french task as a derived value, it is still possible for the user to override it completely:

greeting {
    user = "Cédric"
    outro = "!!!"
}
tasks.named("direBonjour") {
    outro = " !"
}

Then executing direBonjour would print:

Bonjour Cédric !

Outcome #4: The provider API lets precisely define how to compute a value from another property, in a lazy manner, and provides an elegant way to supply default, or conventional values.

You can read more about lazy configuration and the provider API in the Gradle documentation, but in a nutshell, the derivation logic is exactly what the layout.buildDirectory.dir("…") is doing: it defines a directory which is derived from the existing build directory value.

Conclusion

In this blog post, we’ve leveraged a real world use case, integrating lexer generation via JFlex, to explain how to properly write a Gradle plugin which:

uses the lazy provider API, making it immune to configuration ordering problems
explains how Gradle’s "task dependencies" are implicit, avoiding hardcoding relationships between tasks, and making it much more robust to arbitrary configuration changes
doesn’t rely on arbitrary ordering (e.g, like in Maven, "all sources of all generators must be generated before you can compile anything") but instead knows that only if you need to compile the main source set, then you need to generate JFlex sources
uses the worker API, letting us working around a bug in the JFlex library regarding shared mutable state

In addition, we’ve seen the basics of the provider API, which allows plugins to define default values as well as computing derived values for inputs or outputs in a lazy manner. We’ve also hinted at how plugins can expose configuration mechanisms which reduce the API surface, while making it convenient to refactor, therefore dramatically reducing the cost of maintenance of builds.

Eventually, the user facing code of using our JFlex plugin is a single line in a build script:

plugins {
    id "io.micronaut.internal.jflex"
}

There is no configuration (because we use the convention values) and no imperative code (because the build logic of creating tasks is deferred to a plugin). As a bonus, because we used a separate build and composite builds, if we want to publish this plugin to the Gradle plugin portal later, it would be just about adding some configuration to the jflex-plugin module. There is effectively no need to publish a plugin, either to a local repository, or a remote one, to be able to use plugins in Gradle!

The full code is actually available in the upcoming Micronaut TOML module.

Composition over inheritance: Gradle vs Maven

Wed, 1 Dec 2021 00:00:00 +0100

Introduction

In general, when I read comments about Maven vs Gradle, I realize that people focus on the cosmetics (XML vs Groovy/Kotlin) when it’s from my point of view the least interesting aspect of the comparison. In this article, I want to focus on one particular aspect which differentiates the 2 build tools: the famous composition over inheritance paradigm. In different aspects (POM files, lifecycle), Apache Maven is using inheritance, while Gradle is using composition. It is a particularly important difference which completely changes the way we think about building software.

Inheritance in Maven builds

A typical Maven project is built with a pom.xml file, which declares everything the module needs: - the dependencies - the build plugins and their configuration

Very quickly, it turns out that there are common things that you want to share between modules:

they would use the same compiler options
they would use the same plugins and configuration
they would apply a number of common dependencies
etc.

Let’s imagine that we have a project which consists of 3 modules: - a library module, pure Java - an application module which uses the library and the Micronaut Framework - and a documentation module which provides a user manual for the application using Asciidoctor.

The idiomatic way to solve the problem of sharing the configuration of the library and application modules (which are both Java) in Maven is to define a so-called "parent POM" which declares all of these common things, for example:

<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>example-parent</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <name>Common Config</name>

    <properties>
        <java.version>1.8</java.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.jupiter.version>5.8.1</junit.jupiter.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

To simplify things, we could call this a "convention": by convention, all modules which will use this parent POM will apply all those plugins and dependencies (note, there are subtleties if you use <pluginManagement> or <dependencyManagement>).

A "child POM" like our application pom only has to declare the parent to "inherit" from it:

<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.example</groupId>
        <artifactId>example-parent</artifactId>
        <version>1.0.0-SNAPSHOT</version>
    </parent>

    <artifactId>application</artifactId>

    <dependencies>
        <dependency>
            <groupId>org.example</groupId>
            <artifactId>library</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
    </dependencies>

</project>

This model works really well when all modules have a lot in common. The inheritance model also makes it simple to override things (child values override parent values). In the example above, we don’t have to specify the groupId and version of our module because it will be inherited from the parent.

However, this model comes with a number of drawbacks:

as soon as you have different modules which share different set of dependencies, or use different sets of plugins, you have to create different parents and have an inheritance model between parents. Unfortunately this is the case here, since only our library and application modules have something in common. It won’t be a surprise for many that you have to exclude dependencies just because they came through parent poms…
you can only have a single parent, meaning that you cannot inherit from a framework parent POM and from your own conventions.
it’s not great for performance, because you end up configuring a lot of things which will never be necessary for your particular "child" module.
overriding values is sometimes much more complicated and you have to start relying on obscure syntaxes like combine.children="append" (see this excellent blog post for details).

Those limitations are quickly reached when you are using a framework like Micronaut or Spring Boot. Because those frameworks are built with developer productivity in mind, they come with their own "parent POMs" which makes the lives of developers easier by avoiding copy and paste of hundreds of lines of XML. They also need to provide this parent POM because they would come with their own Maven plugin which works around the limitations of the lifecycle model.

But then, we have a problem: on one side, you have this "parent POM" which is provided by the framework, and on the other side, you have your own "parent POM" which is providing, say, the company-specific conventions (like checkstyle configuration, coordinates of Maven repositories for publication, etc.).

In order to be able to use both conventions, you have to create a new parent POM, and you have no choice but writing your company convention parent POM inheriting from the framework POM: obviously you can’t change the framework POM itself! This is problematic, because it means that for every release of the framework, you have to update your company convention parent POM. This is also problematic for another aspect: not all the modules of your multi-project build are "Spring Boot" or "Micronaut" applications. Some of them may be simple Java libraries which are used by your app, but do not require the framework. As a consequence, you have to create multiple parents, and duplicate the configuration in each of those POM files.

This inheritance problem surfaces in different places in Maven. Another one is, as I mentioned, the "lifecycle" which works in phases. Basically, in Maven everything is executed linearly: if you want to do install, then you have to execute everything which is before that phase, which includes, for example, test. This may sound reasonable, but this model completely falls apart: this is no surprise that every single plugin has to implement their own -DskipTest variant, in order to avoid doing work which shouldn’t be done. I had users@maven.apache.org:2021-9">an interesting use case when implementing the GraalVM native Maven plugin, which requires to configure the surefire plugin to pass extra arguments. Long story short: this isn’t possible with Maven. Consequence: the only workaround is the multiplication of Maven profiles, which a user has to understand, maintain, and remember.

Composition in Gradle builds

Gradle builds use a very different model: composition. In a nutshell, in a Gradle project you don’t explain how to build, but what you build: that is, you would say "this is a library", or "this is a CLI application" or "this is a documentation module". Because a library exposes an API and an application doesn’t, those are different things, so their conventions, and capabilities, are different.

The way you "say" this is in a Gradle build is by applying plugins.

A typical Java library would apply the java-library plugin, while an application would apply the application plugin and a documentation project would apply, say, the asciidoctor plugin. What do a Java library project and a documentation project have in common? Barely nothing. A Java Library has Java sources, a number of dependencies, code quality plugins applied, etc. The documentation module, on its side, is a set of markdown or asciidoc files, and resources. The layout of the projects is different, the conventions are different, and the set of plugins are different. Java projects may share the same conventions for source layout, but they are obviously different for the docs. In addition, there’s no reason to let the user declare "implementation" dependencies on the documentation project: it doesn’t make sense so it should be an error to do so.

On the other hand all those modules may share a number of things:

they are all published to a Maven repository
they need to use the same Java toolchain
they need to comply to security policies of your company

The way Gradle solves this problem is by composing plugins:

a plugin can "apply" another plugin
each plugin is guaranteed to be applied only once, even if several plugins use it
a plugin can "react" to the application of other plugins, allowing fine-grained customizations

So in the example above, the application use case can be easily solved: first, you’d have your own "convention plugin" which defines your company conventions (e.g apply the checkstyle plugin with a number of rules). Then, you’d have the Micronaut application plugin which is already written for you. Finally, your application module would simply apply both plugins:

plugins {
   id 'com.mycompany.conventions' version '1.0.0'
   id 'io.micronaut.application' version '3.0.0'
}

micronaut {
    version '3.2.0'
}

What becomes more interesting is that you can (and you actually should) create your own "component types" which apply a number of plugins. In the example above, we could replace the use of the 2 plugins with a single one:

plugins {
   id 'com.mycompany.micronaut-application' version '3.0.0'
}

Note how we moved the configuration of the micronaut version to our convention plugin. I’m not going to explain how to write a custom Gradle plugin in this blog post, but the code of this plugin would very much look like this:

plugins {
    id 'com.mycompany.conventions' version '1.0.0'
    id 'io.micronaut.application' version '3.0.0'
}

micronaut {
    version '3.2.0'
}

Does it look familiar? Yes it does, this is exactly what we had in the beginning: composition is slowly happening! I encourage you to take a look at this documentation for further details about writing your own convention plugins.

Interestingly, as I said, Gradle plugins are allowed to react to the presence of other plugins. This makes it particularly neat for defining dynamically more tasks depending on the context. For example, a plugin can do:

pluginManager.withPlugin('io.micronaut.application') {
    // configure the Micronaut application plugin
}
pluginManager.withPlugin('io.micronaut.library') {
    // configure the Micronaut library plugin
}
pluginManager.withPlugin('io.spring.boot') {
    // configure the Spring Boot plugin
}

Which is very resilient to the fact that the plugins may be applied in any order and that they can combine with each other to provide higher level constructs. It also makes it possible to give choice to users regarding their preferences: you provide a single convention plugin which is aware of what to do if the user prefers to use Spring Boot over Micronaut.

In the end, com.mycompany.micronaut-application is defined as a combination of the io.micronaut.application, your.company.conventions plugins. Instead of declaring how to build your company application, you simply described what it is.

This is only touching the surface of the Gradle world here, but when I read that Gradle is "just Ant on steroids", nothing could be more wrong. Gradle in this case is much superior, because it focuses on convention over configuration, while providing better constructs than Maven does for it.

But let’s come back to our multi-project example: each of the modules would apply a different convention plugin (which is also why it’s important that the allprojects pattern dies):

library would apply the com.mycompany.library plugin
application would apply the com.mycompany.application plugin
docs would apply the com.mycompany.docs plugin

The com.mycompany.library plugin would, for example, apply the java-library and com.mycompany.java-conventions plugin. The com.mycompany.application plugin would, for example, apply the io.micronaut.application and com.mycompany.java-conventions plugin (knowing that the io.micronaut.application plugin applied the application plugin and more, such as the GraalVM plugin) The com.mycompany.docs plugin would, for example, apply the org.asciidoctor.jvm.convert plugin and the com.mycompany.docs plugin.

You’ll notice how those actually combine together, making it easier to maintain and upgrade builds: should you change the company conventions, all you have to do is release a new version of the convention plugin.

Conclusion

In this quickie, I have explained a major difference in how Maven and Gradle envision build configuration. While both of them are designed with convention over configuration in mind, the inheritance model of Maven makes it difficult to build conventions on top of each other without duplication. On the other hand, Gradle uses a composition model which makes it possible to design your own conventions while being aware of other plugins being applied by the user: Gradle builds are more lenient and more maintainable.

As a complement, you might be interested in:

reading the structuring large projects documentation from the Gradle docs
watching my 10 mins video about authoring Gradle builds
following my former team mate Jendrik who created a series of videos about modern Gradle

Multi-repository development made easy

Thu, 4 Nov 2021 00:00:00 +0100

Are you working in a multi-repository setup?

In general, things start getting messy as soon as you have a feature which requires changes to more than one repository. For example, you may have a core repository, and a module repository, and the feature that you’re working on for module requires API changes in module.

If so, it’s likely that you’ve been annoyed by the fact that to be able to test the changes to module, you minimally had to publish a local snapshot to your local Maven repository. While this can kind of work locally, it’s easy to miss publishing from time to time, and therefore thinking that a change works when it actually relies on an outdated dependency.

Things get more complicated as soon as CI is involved, or that you want to share the results of work in progress, for example for review, with your colleagues:

did you ever had to explain that they had to checkout core/some-branch, publish to Maven local, then checkout module/some-feature-branch and test it?
did you ever realize late that you forgot to push changes to master so that they could try?
did you ever complain that to make this happen on CI, you actually had to eagerly merge your feature branch to core, just so that the other repository, on a feature branch, could see it?
did you ever want to see if your modules simply do not break with latest master, without having to change anything to your build scripts?

If you answered yes to any of those questions, then I’m glad to say there’s a solution!

The underlying problem is that using Maven SNAPSHOTs to deal with multi-repository development is not a good enough. It cannot model the complexity of multi-repository development, with features being developed concurrently on different branches. Using SNAPSHOTs (binary dependencies) to coordinate projects leads to hard to diagnose bugs, broken integration processes. You typically have to eagerly push changes, or wait for snapshots to be published on a shared repository, just so that you can actually verify that integration with other modules work. Those problems do not happen in a a single repository world, because all changes are integrated at once.

I faced this very same problem with Micronaut:I’m currently working on a feature which involves changes to multiple repositories at once:

Micronaut Core, with additional public APIs
Micronaut AOT, the new module I’m actively working on
the Micronaut Gradle Plugin, which integrates the AOT module
and a local test project which uses the modified Gradle plugin

That’s, minimally 4 different projects, and a change to any of them is a pain to deal with. With my experience with Gradle, I knew there was a better way.

A plugin to make it easier!

Today, I’m happy to announce a new Gradle plugin which aims at making multi-repository development a breeze: Included Git repositories plugin.

This plugin lets you import Git repositories as source dependencies, without having to change your dependency declarations. What does that mean? In the example above, it means that I can explain, when I’m working on module, that it needs to build against core/some-branch: Gradle will then automatically checkout the project, build the branch and substitute any binary dependency corresponding to core with the source dependency.

In a nutshell, the configuration would look like this:

gitRepositories {
	include('core') {
		uri = 'git@github.com:mycompany/core.git'
		branch = 'some-branch'
	}
}

That’s it! No need to change your build scripts to update dependency coordinates, Gradle will do the magic!

It completely changes the way of thinking about multi-repository development, because CI, or colleagues, would not have to care about instructions about how to build your particular branch: everything is known upfront.

Of course, you’re going to tell that well, that’s cool but it still requires you to push your changes to the remote repository so that you can test things locally. Well, a good multi-repository development story must integrate both the local and remote experience. This is why this plugin actually makes it a breeze to support this pattern.

There are actually 2 ways you can handle this. The first one is to explain to Gradle that instead of checking out the sources, it can simply use a local copy instead. In this case, the plugin will simply ignore whatever you declared in the gitRepositories block for the repository, and use whatever is available locally. For this you’d set a local.git.<repoName> Gradle property (in your gradle.properties file) pointing to your local copy. In the example above, I would for example add a local.git.core property pointing to my local copy of core.

Alternatively, if you keep things organized into checkout directories like I do, it’s likely that you have all your micronaut related projects in a single micronaut-projects directory. In this case, by setting the auto.include.git.dirs Gradle property to the micronaut-projects directory, the plugin will automatically map directory names in that micronaut-projects directory to included Git repository names. So if I have:

gitRepositories {
	include('micronaut-core') {
		uri = 'git@github.com:mycompany/core.git'
		branch = 'some-branch'
	}
}

and a micronaut-core directory under micronaut-projects, then it will automatically be used instead of cloned from remote.

Those options make it extremely convenient to develop locally, and only push changes when ready. On CI, builds would checkout the dependents automatically, and you’d have nothing to configure.

More complex use cases

The very same mechanism can be used to create "integration" builds on CI. For example, it makes it very simple to have builds which would automatically build against the latest state of master, instead of having to wait for SNAPSHOT to be published, and more importantly, without having to change any build file. As a bonus, it also works for transitive dependencies: for example if you have A --depends on-→ B --depends on-→ C, then you may want to make sure that if C is changed, A still works. How do you do this with snapshots, if there’s no direct dependency between A and C? This plugin makes it very simple to test: just declare a Git repository for C and you’re done!

Need your help!

I think this plugin has potential to dramatically change how we develop in the multi-repository world. The plugin is in very early stages, and I will need your help: report bugs, improve the documentation, improve testing, etc. It will also be interesting to get your user stories so that we, collectively, can improve it to support more scenarios.

Plugin documentation: https://melix.github.io/includegit-gradle-plugin
Plugin development home: https://github.com/melix/includegit-gradle-plugin
Plugin on Gradle plugin portal: https://plugins.gradle.org/plugin/me.champeau.includegit

A Gradle quickie: properly using dependsOn

Wed, 6 Oct 2021 00:00:00 +0200

Today I’d like to share a small example of what not to do with Gradle. Some of you may already know that I recently joined the Micronaut team at Oracle, and part of my job is to improve the build experience, be it for Micronaut itself or Micronaut users. Today I’m going to focus on an example I found in the Micronaut build itself.

TL/DR: If you use dependsOn, you’re likely doing it wrong.

When should you use dependsOn?

In a nutshell, Gradle works by computing a graph of task dependencies. Say that you want to build a JAR file: you’re going to call the jar task, and Gradle is going to determine that to build the jar, it needs to compile the classes, process the resources, etc… Determining the task dependencies, that is to say what other tasks need to be executed, is done by looking up at 3 different things:

the task dependsOn dependencies. For example, assemble.dependsOn(jar) means that if you run assemble, then the jar task must be executed before
the task transitive dependencies, in which case we’re not talking about tasks, but "publications". For example, when you need to compile project A, you need on classpath project B, which implies running some tasks of B.
and last but not least, the task inputs, that is to say, what it needs to execute its work

In practice, it’s worth noting that 2. is a subset of 3. but I added it for clarity.

Now let’s look at this snippet:

task docFilesJar(type: Jar, description: 'Package up files used for generating documentation.') {
    archiveVersion = null
    archiveFileName = "grails-doc-files.jar"
    from "src/main/template"
    doLast {
        copy {
            from docFilesJar.archivePath
            into "${buildDir}/classes/groovy/main"
        }
    }
}

jar.dependsOn docFilesJar

First, let’s realize that this snippet is years old. I mean, very years old, copied from Grails, which was using early releases of Gradle. Yet, there’s something interesting in what it does, which is a typical mistake I see in all builds I modernize.

It’s tempting, especially when you’re not used to Gradle, to think the same way as other build tools do, like Maven or Ant. You’re thinking "there’s a task, jar, which basically packages everything it finds in classes/groovy/main, so if I want to add more stuff to the jar task, let’s put more stuff in `classes/groovy/main`".

This is wrong!

This is wrong for different reasons, most notably:

when the docsFilesJar task is going to be executed, it will contribute more files to the "classes" directory, but, wait, those are not classes that we’re putting in there, right? It’s just a jar, resources. Shouldn’t we use resources/groovy/main instead? Or is it classes/groovy/resources? Or what? Well, you shoudn’t care because it’s not your concern where the Java compile task is going to put its output!
it breaks cacheability: Gradle has a build cache, and multiple tasks contributing to the same output directory is the typical example of what would break caching. In fact, it breaks all kinds of up-to-date checking, that is to say the ability for Gradle to understand that it doesn’t need to execute a task when nothing changed.
it’s opaque to Gradle: the code above executes a copy in a doLast block. Nothing tells Gradle that the "classes" have additional output.
imagine another task which needs the classes only. Depending on when it executes, it may or may not, include the docsFileJar that it doesn’t care about. This makes builds non-reproducible (note that this is exactly the reason why Maven build cannot be trusted and that you need to run clean, because any "goal" can write to any directory at any time, making it impossible to infer who contributed what).
it requires to declare an explicit dependency between the jar task and the docsFileJar task, to make sure that if we execute jar, our "docs jar" file is present
it doesn’t tell why there’s a dependency: is it because you want to order things, or is it because you require an artifact produced by the dependent task? Something else?
it’s easy to forget about those: because you may run build often, you might think that your build works, because jar is part of the task graph, and by accident, the docsFileJar would be executed before
it creates accidental extra work: most often a dependsOn will trigger too much work. Gradle is a smart build tool which can compute precisely what it needs to execute for each specific task. By using dependsOn, you’re a bit using a hammer and forcing it to integrate something in the graph which wasn’t necessarily needed. In short: you’re doing too much work.
it’s difficult to get rid of them: when you see a dependsOn, because it doesn’t tell why it’s needed, it’s often hard to get rid of such dependencies when optimizing builds

Use implicit dependencies instead!

The answer to our problem is actually simpler to reason about: reverse the logic. Instead of thinking "where should I put those things so that it’s picked up by jar", think "let’s tell the jar task that it also needs to pick up my resources".

All in all, it’s about properly declaring your task inputs.

Instead of patching up the output of another task (seriously, forget about this!), every single task must be thought as a function which takes inputs and produces an output: it’s isolated. So, what are the inputs of our docsFileJar? The resources we want to package. What are its outputs? The jar itself. There’s nothing about where we should put the jar, we let Gradle pick a reasonable place for us.

Then what are the inputs of the jar task itself? Well, it’s regular inputs plus our jar. It’s easier to reason about, and as bonus, it’s even shorter to write!

So let’s rewrite the code above to:

task docFilesJar(type: Jar, description: 'Package up files used for generating documentation.') {
    archiveVersion = null
    archiveFileName = "grails-doc-files.jar"
    from "src/main/template"
}

jar {
    from docFilesJar
}

Can you spot the difference? We got rid of the copy in the docFilesJar task, we don’t want to do this. What we want, instead, is to say "when you build the jar, also pick this docsFileJar. And that’s what we’re doing by telling from docsFileJar. Gradle is smart enough to know that when it will need to execute the jar task, first, it will need to build the docsFilesJar.

There are several advantages to this:

the dependency becomes implicit: if we don’t want to include the jar anymore, we just have to remove it from the specification of the inputs.
it doesn’t pollute the outputs of other tasks
you can execute the docsFileJar independently of jar

All in all, it’s about isolating things from each other and reducing the risks of breaking a build accidentally!

All things lazy!

The modified code isn’t 2021 compliant. The code above works, but it has one drawback: the docFilesJar and jar tasks are going to be configured (instantitated) even if we call something that doesn’t need it. For example, imagine that you call gradle compileJava: there’s no reason to configure the jar tasks there because we won’t execute them.

For this purpose, to make builds faster, Gradle provides a lazy API instead:

tasks.register('docFilesJar', Jar) {
    description = 'Package up files used for generating documentation.'
    archiveVersion = null
    archiveFileName = "grails-doc-files.jar"
    from "src/main/template"
}

tasks.named('jar', Jar) {
    from docFilesJar
}

Conclusion

As a conclusion:

avoid using explicit dependsOn as much as you can
I tend to say that the only reasonable use case for dependsOn is for lifecycle tasks (lifecycle tasks are tasks which goal is only there to "organize the build", for example build, assemble, check: they don’t do anything by themselves, they just bind a number of dependents together)
if you find use cases which are not lifecycle tasks and cannot be expressed by implicit task dependencies (e.g declaring inputs instead of dependsOn), then report it to the Gradle team

Frequently asked questions about version catalogs

Sun, 11 Apr 2021 00:00:00 +0200

This post is a follow up to my version catalogs intro blog post and answers some frequently asked questions. Ideally, this should be part of the Gradle docs itself but we don’t have a good place for this kind of docs yet (I’m working on it!), so in the meantime, here you go! This blog post will be updated as I’m seeing more questions.

Table of Contents

Version catalogs FAQ

Version catalogs FAQ

Can I use a version catalog to declare plugin versions?

No.The initial implementation of the version catalogs had, in TOML files, a dedicated section for plugins:

[plugins]
id.of.my.awesome.plugin="1.2.3"

However, after community feedback and for consistency reasons, we removed this feature from the initial release. This means that currently, you have to use the pluginManagement section of the settings file to deal with your plugin versions, and this cannot use, in particular, the TOML file to declare plugin versions:

settings.gradle

pluginManagement {
    plugins {
        id("me.champeau.jmh") version("0.6.3")
    }
}

It may look surprising that you can’t use version(libs.plugins.jmh) for example in the pluginManagement block, but it’s a chicken and egg problem: the pluginManagement block has to be evaluated before the catalogs are defined, because settings plugins may contribute more catalogs or enhance the existing catalogs. Therefore, the libs extension doesn’t exist when this block is evaluated.

The limitation of not being able to deal with plugin versions in catalogs will be lifted in one way or another in the future.

Can I use the version catalog in buildSrc?

Yes you can. Not only in buildSrc, but basically in any included build too. You have several options, but the easiest is to include the TOML catalog in your buildSrc/settings.gradle(.kts) file:

buildSrc/settings.gradle

dependencyResolutionManagement {
    versionCatalogs {
        lib {
            from(files("../gradle/libs.versions.toml"))
        }
    }
}

But how can I use the catalog in plugins defined in `buildSrc`?

The solution above lets you use the catalogs in the build scripts of buildSrc itself, but what if you want to use the catalog(s) in the plugins that buildSrc defines, or precompiled script plugins? Long story short, currently, you can do it using a type unsafe API only.

First you need to access the version catalogs extension to your plugin/build script, for example in Groovy:

def catalogs = project.extensions.getByType(VersionCatalogsExtension)

or in Kotlin:

val catalogs = extensions.getByType<VersionCatalogsExtension>()

then you can access the version catalogs in your script, for example writing:

pluginManager.withPlugin("java") {
    val libs = catalogs.named("libs")
    dependencies.addProvider("implementation", libs.findDependency("lib").get())
}

Note that this API doesn’t provide any static accessor but is nevertheless safe, using the Optional API. There’s a reason why you cannot access type-safe accessors in plugins/precompiled script plugins, you will find more details on this issue. In a nutshell, that’s because buildSrc plugins (precompiled or not) are plugins which can be applied to any kind of project and we don’t know what the target project catalogs will be: there’s no inherent reason why they would be the same. In the future we will probably provide a way to explain that, at your own risk, expect the target catalog model to be the same.

Can I use version catalogs in production code?

No, you can’t. Version catalogs are only accessible to build scripts/plugins, not your production code.

Should I use a platform or a catalog?

You should probably use both, look at our docs for a complete explanation.

Why did you choose TOML and not YAML?

or XML (or pick your favorite format). The rationale is described in the design document.

My IDE is red everywhere, MISSING_DEPENDENCY_CLASS error

If you are seeing this error:

upgrade to the latest IntelliJ IDEA 2021.1, which fixes this problem.

Why can’t I have nested aliases with the same prefix?

Imagine that you want to have 2 aliases, say junit and junit-jupiter and that both represent distinct dependencies: Gradle won’t let you do this and you will have to rename your aliases to, say junit-core and junit-jupiter. That’s because Gradle will map those aliases to accessors, that is to say libs.getJunit() and libs.getJUnit().getJupiter(). The problem is that you can’t have an accessor which is both a leaf (represents a dependency notation) and a node (that is to say an intermediate node to access a real dependency). The reason we can’t do this is that we’re using lazy accessors of type Provider<MinimalExternalDependency> for leaves and that type cannot be extended to provide accessors for "children" dependencies. In other words, the type which represents a node with children provides accessors which return Provider<…> for dependencies, but a provider itself cannot have children. A potential workaround for this would be to support, in the future, an explicit call to say "I’m stopping here, that’s the dependency I need", for example:

dependencies {
    testImplementation(libs.junit.get())
    // or
    testImplemementation(libs.junit.peek()) // because `get()` might be confusing as it would return a `Provider` on which you can call `get()` itself
}

For now the team has decided to restrict what you can do by preventing having aliases which have "name clashes".

Why can’t I use an alias with dots directly?

You will have noticed that if you declare an alias like this:

[libraries]
junit-jupiter = "..."

then Gradle will generate the following accessor: libs.junit.jupiter (basically the dashes are transformed to dots). The question is, why can’t we just write:

[libraries]
junit.jupiter = "..."

And the reason is: tooling support. The previous declaration is actually equivalent to writing:

[libraries]
   [junit]
   jupiter = "..."

but technically, it’s undecidable where the "nesting hierarchy" stops, which would prevent tools from providing good completion (for example, where you can use { module = "…"}. It also makes it harder for tooling to automatically patch the file since they wouldn’t know where to look for.

As a consequence, we’ve decided to keep the format simple and implement this mapping strategy.

Should I use `commons-lang3` as an alias or `commonsLang3`?

Problably neither one nor the other :) By choosing commons-lang3, you’re implicitly creating a group of dependencies called commons, which will include a number of dependencies, including lang3. The question then is, does that commons group make sense? It’s rather abstract, no? Does it actually say it’s "Apache Commons"?

A better solution would therefore be to use commonsLang3 as the alias, but then you’d realize that you have chosen a version in the alias name, so why not commonsLang directly?

Therefore:

[libraries]
commonsLang = { module="org.apache.commons:commons-lang3:3.3.1" }

This means that the dashes should be limited to grouping of dependencies, so that they are organized in "folders". This can make it practical when you have lots of dependencies, but it also makes them less discoverable by completion, since you’d have to know in which subtree to look at. Proper guidance on what to use will be discussed later, based on your feedback and practices.

Should I use the settings API or the TOML file?

Gradle comes with both a settings API to declare the catalog, or a convenience TOML file. I would personally say that most people should only care about the TOML file as it covers 80% of use cases. The settings API is great as soon as you want to implement settings plugins or, for example, if you want to use your own, existing format to declare a catalog, instead of using the TOML format.

Why can’t I use excludes or classifiers?

By design, version catalogs talk about dependency coordinates only. The choice of applying excludes is on the consumer side: for example, for a specific project, you might need to exclude a transitive dependency because you don’t use the code path which exercises this dependency, but this might not be the case for all places. Similarly, a classifier falls into the category of variant selectors (see the variant model): for the same dependency coordinates, one might want classifier X, another classifier Y, and it’s not necessarily allowed to have both in the same graph. Therefore, classifiers need to be declared on the dependency declaration site:

dependencies {
   implementation(variantOf(libs.myLib) { classifier('test-fixtures') })
}

The rationale being this limitation is that the use of classifiers is an artifact of the poor pom.xml modeling, which doesn’t assign semantics to classifiers (we don’t know what they represent), contrary to Gradle Module Metadata. Therefore, a consumer should only care about the dependency coordinates, and the right variant (e.g classifier) should be selected automatically by the dependency resolution engine. We want to encourage this model, rather than supporting adhoc classifiers which will eventually require more work for all consumers.

How do I tell Gradle to use a specific artifact?

Similarly to classifiers or excludes, artifact selectors belong to the dependency declaration site. You need to write:

dependencies {
    implementation(libs.myLib) {
        artifact {
            name = 'my-lib' // note that ideally this will go away, see https://github.com/gradle/gradle/issues/16768
            type = 'aar'
        }
    }
}

Where should I report bugs or feature requests?

As usual, on our issue tracker. There’s also the dedicated epic where you will find the initial specification linked, which explains a lot of the design process.

Simplified version management with Gradle 7

Wed, 24 Mar 2021 00:00:00 +0100

Gradle 7 introduces the concept of version catalogs, which I’ve been working on for several months already. Long story short, I’m extremely excited by this new feature which should make dependency management easier with Gradle. Let’s get started!

Please also read my Version catalogs FAQ follow up post if you have more questions!

Sharing dependencies between projects

One of the most frequent questions raised by Gradle users is how to properly share dependency versions between projects. For example, let’s imagine that you have a multi-project build with this layout:

root
 |---- client
 |---- server

Because they live in the same "multi-project", it is expected that both client and server would require the same dependencies. For example, both of them would need Guava as an implementation detail and JUnit 5 for testing:

build.gradle

dependencies {
    implementation("com.google.guava:guava:30.0-jre")
    testImplementation("org.junit.jupiter:junit-jupiter-api:5.7.1")
    testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine")
}

Without any sharing mechanism, both projects would replicate the dependency declarations, which is subject to a number of drawbacks:

upgrading a library requires updating all build files which use it
you have to remember about the dependency coordinates (group, artifact, version) of all dependencies
you might accidentally use different versions in different projects
some dependencies are always used together but you have to duplicate entries in build files

Existing patterns

For these reasons, users have invented over the years different patterns for dealing with dependency versions over time. For example:

Versions in properties files:

gradle.properties

guavaVersion=30.0-jre

then in a build file:

dependencies {
    implementation("com.google.guava:guava:${guavaVersion}")
}

Or versions in "extra properties" in the root project:

extra properties

ext {
   guavaVersion = '30.0-jre'
}

// ...

dependencies {
    implementation("com.google.guava:guava:${guavaVersion}")
}

Sometimes you even find full coordinates in dependencies.gradle files.

And since the rise of the Kotlin DSL, another pattern became extremely popular in the Android world: declaring libraries in buildSrc then using type-safe accessors to declare dependencies in build scripts:

buildSrc/src/main/kotlin/Libs.kt

object Libs {
   val guava = "com.google.guava:guava:30.0-jre"
}

and in a build script:

build.gradle

dependencies {
    implementation(Libs.guava)
}

This last example is interesting because it goes into the direction of having more type-safety, more compile-time errors (as opposed to runtime errors). But it has a major drawback: any change to any dependency will trigger recompilation of build scripts and invalidate the build script classpath, causing up-to-date checkness to fail and in the end, rebuilding a lot more than what you should do for a single version change.

Introducing version catalogs

A version catalog is basically a replacement for all the previous patterns, supported by Gradle, without the drawbacks of the previous approaches. To add support for version catalogs, you need to enable the experimental feature in your settings file:

settings.gradle

enableFeaturePreview("VERSION_CATALOGS")

In its simplest form, a catalog is a file found in a conventional location and uses the TOML configuration format:

gradle/libs.versions.toml

[libraries]
guava = "com.google.guava:guava:30.0-jre"
junit-jupiter = "org.junit.jupiter:junit-jupiter-api:5.7.1"
junit-engine = { module="org.junit.jupiter:junit-jupiter-engine" }

[bundles]
testDependencies = ["junit-jupiter", "junit-engine"]

This declares the dependency coordinates which will be used in build scripts. You still have to declare your dependencies, but this now can be done using a typesafe API:

build.gradle

dependencies {
    implementation(libs.guava)
    testImplementation(libs.testDependencies)
}

The benefit of type-safe APIs is immediately visible in the IDE:

In the catalog file above, we inlined dependency versions directly in the coordinates. However, it’s possible to externalize them so that you can share a dependency version between dependencies. For example:

gradle/libs.versions.toml

[versions]
groovy = "2.5.14"
guava = "30.0-jre"
jupiter = "5.7.1"

[libraries]
guava = { module="com.google.guava:guava", version.ref="guava" }
junit-jupiter = { module="org.junit.jupiter:junit-jupiter-api", version.ref="jupiter" }
junit-engine = { module="org.junit.jupiter:junit-jupiter-engine" }

groovy-core = { module="org.codehaus.groovy:groovy", version.ref="groovy" }
groovy-json = { module="org.codehaus.groovy:groovy-json", version.ref="groovy" }

[bundles]
testDependencies = ["junit-jupiter", "junit-engine"]

This new feature makes it trivial to update a dependency version: you have a single place where to look at.

This comes with other benenefits like the fact that updating the GAV coordinates (group, artifact or version) of a dependency doesn’t trigger recompilation of build scripts. The TOML format also provides us with the ability to declare rich versions.

Under the hood

Under the hood, Gradle provides an API to declare catalogs. This API is found on the Settings, which means that plugin authors can contribute catalogs, for example via convention plugins applied to the settings.gradle(.kts) file.

This API is more verbose than when you use the TOML file, but is designed for type-safety. The equivalent of the catalog above would be this:

settings.gradle

dependencyResolutionManagement {
   versionCatalogs {
      libs {
           alias("guava").to("com.google.guava", "guava").versionRef("guava")
           alias("junit-jupiter").to("org.junit.jupiter", "junit-jupiter-api").versionRef("jupiter")
           alias("junit-engine").to("org.junit.jupiter", "junit-jupiter-engine").withoutVersion()
           alias("groovy-core").to("org.codehaus.groovy", "groovy").versionRef("groovy")
           alias("groovy-json").to("org.codehaus.groovy", "groovy-json").versionRef="groovy")

           version("groovy", "2.5.14")
           version("guava", "30.0-jre")
           version("jupiter", "5.7.1")
      }
   }
}

This API actually must be used if you are consuming an external catalog. That’s one of the big selling points of this feature: it allows teams (or framework authors) to publish catalogs, so that users can get recommendations. For example, let’s imagine that the Spring Boot team publishes a catalog of recommendations (they do something similar today with a BOM, but BOMs will have an impact on your transitive dependencies that you might not want).

Consuming this catalog it in a Gradle build would look like this:

settings.gradle

dependencyResolutionManagement {
   versionCatalogs {
       spring {
           from("org.springframework:spring-catalog:1.0')
       }
   }
}

This would make a catalog available under the spring namespace in your build scripts. Therefore, you’d be able to use whatever version of SLF4J the Spring team recommends by declaring this dependency:

build.gradle

dependencies {
    implementation(spring.slf4j)
}

Such a catalog would be published on a regular Maven repository, as a TOML file. Thanks to Gradle’s advanced dependency resolution engine, it’s totally transparent to the user that the actual dependency is a catalog.

What version catalogs are not

At this stage, it becomes important to state what version catalogs are not:

they are not the "single source of truth" for your dependencies: it’s not because you have a catalog that you can’t directly declare dependencies using the "old" notation in build scripts. Nor does it prevent plugins from adding dependencies. Long story short: the presence of a catalog makes discoverability and maintenance easier, but it doesn’t remove any of the flexibility that Gradle offers. We’re thinking about ways to enforce that all direct dependencies are declared via a catalog in the future.
the version declared in a catalog is not necessarily the one which is going to be resolved: a catalog only talks about direct dependencies (not transitives) and the version that you use is the one used as an input to dependency resolution. With transitive dependencies, it’s typically possible that a version gets upgraded, for example.
while it makes it possible for third-party tooling to "update automatically" versions, this wasn’t a goal of this work. If you relate to the previous point, it all makes sense: as long as you rely on the input (what is written) to assume what is going to be resolved, you’re only wishing that it is what is going to be resolved. It may be enough for some cases, though. Please refer to my blog post about Dependabot for more insights on this topic. Again, future work we have in mind is adding some linting to make sure that the first level dependencies you declare match whatever you resolved, because in general, having a difference there is a sign that something is wrong in the setup. I’m going to repeat myself, but don’t assume that the version you see in a config file is the one you will get.

Please take a look at the documentation for further details, and give us your feedback!

Using Java 16 with Gradle

Wed, 17 Mar 2021 00:00:00 +0100

Java 16 is out and I’m seeing a number of folks trying to figure out how to use Java 16 with Gradle. Often they would try to run Gradle with JDK 16 and see it fail. There’s a ticket about Java 16 support in Gradle but in most cases you can already work with JDK 16 without waiting for official support.

The happy path

Gradle 7, which is due soon, will provide official support for Java 16. If you have an existing build that you want to try on Java 16, you can update the wrapper to use the latest Gradle 7.0 milestone release:

gradle/wrapper/gradle-wrapper.properties

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-7.0-milestone-3-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

If you are lucky this is all you need to do.

However, Gradle 7 is a major release, and as such it brings a number of changes which may break your build (deprecated methods being removed, or, in particular for the Java 16 support, upgrading to Groovy 3 internally). It may be a bit involved to migrate to Gradle 7 just to try Java 16.

Decouple the Java version used for Gradle itself from the version you need!

It’s actually better to decouple the version of Java required to run Gradle from the version of Java your application requires. In general, it’s actually considered the best practice to use whatever version of the JDK Gradle officially supports to run Gradle itself, and configure the build to use a different JDK.

Configuring Java toolchains

In Gradle terminology, this is called activating Java Toolchains.

Let’s get started with a sample project running on latest stable Gradle, which is 6.8.3. Make sure that you have Gradle 6.8.3 on your PATH to get started. I’m personally recommending to use sdkman! to install Gradle:

$ sdk install gradle 6.8.3

At the same time, we want to make sure we run Gradle with a supported version, which is anything between Java 8 and 15:

$ java -version
java -version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment 18.9 (build 11.0.9.1+1)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.9.1+1, mixed mode)

If it outputs something else than 8 to 15, please make sure to update your PATH to point to such a JDK. Again you can do this with sdkman!:

$ sdk install java 11.0.9.open

Demo application

Now, let’s create a sample Gradle project:

$ mkdir demo-app
$ cd demo-app
$ gradle init

Then select:

Select type of project to generate:
  1: basic
  2: application
  3: library
  4: Gradle plugin
Enter selection (default: basic) [1..4] 2

Select implementation language:
  1: C++
  2: Groovy
  3: Java
  4: Kotlin
  5: Scala
  6: Swift
Enter selection (default: Java) [1..6] 3

Split functionality across multiple subprojects?:
  1: no - only one application project
  2: yes - application and library projects
Enter selection (default: no - only one application project) [1..2] 1

Select build script DSL:
  1: Groovy
  2: Kotlin
Enter selection (default: Groovy) [1..2] 1

Select test framework:
  1: JUnit 4
  2: TestNG
  3: Spock
  4: JUnit Jupiter
Enter selection (default: JUnit 4) [1..4] 4

Project name (default: demo-app):
Source package (default: demo.app):

and confirm the default name and packages.

Then let’s run our app:

$ ./gradlew run

> Task :app:run
Hello World!

BUILD SUCCESSFUL in 4s
2 actionable tasks: 2 executed

Migrating the application to Java 16

All good! Now let’s configure Gradle to use Java 16 to build and run our app instead. Let’s open the build script, found under app:

app/build.gradle

plugins {
    // Apply the application plugin to add support for building a CLI application in Java.
    id 'application'
}

// Add this under the `plugins` section:
java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(16)
    }
}

The java.toolchain block lets us configure the toolchain that Gradle is going to use to build and run your application. We’re setting 16, which means that we’re going to compile the main and test sources as well as execute the with a Java 16 JDK. Gradle will automatically try to find if you have a Java 16 installation in a conventional location. If it cannot find one you will see something like this happening:

Provisioning toolchain adoptopenjdk-16-x64-linux.tar.gz > adoptopenjdk-16-x64-linux.tar.gz > 66 MiB/195.8 MiB

which means that Gradle is downloading the JDK for you!

Let’s check:

./gradlew run

Dang! The build fails!. To some extent, it’s good news, it means that Gradle is really using Java 16, but why is it failing?

Disabling incremental compilation

Well, you’re facing one of the bugs we fixed in 7, which is that our incremental compiler isn’t compatible with Java 16 because we’re using classes which have been made "hidden" by the module system in Java 16.

There’s an easy fix: let’s disable incremental compilation!

Again, let’s open our app/build.gradle file and add this:

tasks.withType(JavaCompile).configureEach {
	// disable incremental compilation
    options.incremental = false
}

And let’s run the build again:

./gradlew run

Yay! This time the build passed! Congrats, you have your first Java 16 app running!

Alternatively to disabling incremental compilation, you might just want to let Gradle access the JDK internals. This solution is better for performance, even if a bit "hackish":

tasks.withType(JavaCompile).configureEach {
    options.forkOptions.jvmArgs.addAll( ['--add-opens', 'jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED'] )
}

In case you want to use one of the experimental features that Java 16 provides, the setup I’ve described in a previous post about Java Feature Previews still hold and is a good follow-up to this post!

Gradle + WebAssembly

Sun, 21 Feb 2021 00:00:00 +0100

A couple weeks ago, I listened to an interview (in French) of Geoffroy Couprie and Ivan Enderlin who gave a very nice overview of what WebAssembly is and what it’s good for. To be honest it wasn’t clear to me what the advantages of WebAssembly were, compared to a good old JVM: all in all, both are virtual machines, and both promise the same thing, that is to say write once run everywhere. Also, in my mind, WebAssembly was some kind of restricted version of JavaScript like asm.js. I was wrong: it’s more than that.

Why WebAssembly?

One of the aspects which were discussed during this interview was basically sandboxing: with WebAssembly, it’s the responsibility of the embedder to give access, for example, to I/O to the wasm program. In practice, it means that wasm binaries are by default very restricted. In fact, by design, they are restricted to pure computations, which makes them very suitable for isolation of work. Extensions from embedders are responsible for giving access to the host resources (for example the GPU, or the file system).

Another promise of WebAssembly is running at close-to-native speeds.

Therefore, a crazy idea came to me: what if I could use a wasm binary as a task implemention in Gradle? This, for example, would let us use whatever language compiling to WASM as an implementation language for Gradle tasks. mmmmmmm

The result of this experience is available at this repository. The results are quite promising.

Isolating task inputs from the task execution

In a nutshell, a Gradle task can be seen as a function which takes a number of inputs and returns a value. Cacheability is derived from the "pureness" of the function: for the same inputs, the output should be the same.

This makes it very suitable to the WebAssembly model where functions are exported to an embedder.

In reality, it’s more complicated: Gradle tasks most likely use files as inputs, and also produce files. This means that the WebAssembly runtime would need to provide I/O access for this to be really useful.

After some support from Ivan Enderlin, I quickly figured out that it would be difficult to make file access work, so I simplified the problem. In the prototype, my WASM tasks are not able to produce a file output and are limited to simply display the execution result on the console.

For tasks which actually have files as inputs, I’m reading the file contents from Java code into byte arrays which are "propagated" to the WASM runtime memory. This means, effectively, that the WASM functions I played with don’t have any kind of access to the file system and remain pure functions.

With those limitations in mind, here’s what I came up with.

Declaring the task I/O protocol

Gradle is evolving fast and nowadays the idiomatic way to declare inputs and outputs of a task is to use the lazy configuration API. However, this isn’t enforced and nobody prevents you from writing tasks which do not use this API. As an implementor of a new integration mechanism, I can set the rules and actually restrict the scope to tasks which use this API, which has a number of advantages.

Say that we want to write a task which computes the sum of two integers. With the lazy configuration API, you need two "properties" corresponding to the input numbers:

abstract class MyTask extends DefaultTask {
   abstract Property<Integer> getX(); // implementation is generated by Gradle
   abstract Property<Integer> getY();

The actual value is "lazy" in the sense that it can be set via a plugin, overriden in a build script, mapped from another property, etc. For example, that the value of the x property can be computed from the output of another task:

x.set(myOtherTask.outputFile.map { ... }

The advantage of this is that Gradle can track the inputs and outputs of tasks for you and that you don’t have to declare any explicit dependency between tasks, making the build less brittle.

However, once you’re about to execute a task, you don’t really care about the Property<…> wrappers anymore: what matters are the actual values, that you can get():

int x = getX().get();
int y = getY().get();

This often leads to some boilerplate code in task implementations. In our case, the x and y integers are actually the only thing we care about to call our WASM function: we don’t need to pass the richer Provider type to the functions.

Let’s imagine that we have this function written in Rust:

#[no_mangle]
pub extern fn sum(x: i32, y: i32) -> i32 {
    x + y
}

We can see that the inputs of this function are integers and that it returns another integer: it’s that simple.

So, what if we could actually simplify the implementation of the Gradle task itself? With all this in mind, I decided to prototype an annotation processor which would generate the boilerplate code for me.

To transform the Rust function above into a full Gradle task, all I have to write is a declaration of what I call its I/O protocol:

@WasmProtocol(
        taskName = "Sum",
        classpathBinary = "demo_lib",
        functionName = "sum"
)
public interface SumIO {
    @Input
    Property<Integer> getX();

    @Input
    Property<Integer> getY();
}

This will generate a task of type Sum, which will use a wasm binary found on classpath named demo_lib, and use the function sum from that binary as the task implementation. That’s it!

Note that this protocol isn’t declaring any output: that’s a limitation of the prototype right now, which is inherently caused by this whole "Gradle tasks are mostly generating files" problem. But we don’t really care for now.

A more realistic example

For my tests, I used 3 different functions: this sum function, a fib function (the Fibonacci suite function) and I wanted to try something more complicated like computing an md5 hash.

This, for example, is how I would define the protocol for the MD5 hashing function:

public interface HasherIO {
    @InputFile
    @PathSensitive(PathSensitivity.NAME_ONLY)
    RegularFileProperty getInputFile();

    @OutputFile
    RegularFileProperty getOutputFile();
}

Except that in this case, because the input is a file, I couldn’t use my code annotation processor code generator yet (but it’s planned). Instead I wrote code to read the file manually, allocate WebAssembly memory buffers, in order to call the function which is implemented in Rust:

#[no_mangle]
pub extern fn process(bytes: *const u8, len: usize) -> *const c_void {
    let data: &[u8] = unsafe { std::slice::from_raw_parts(bytes, len) };
    let mut hasher = Md5::new();
    hasher.update(data);
    let result = hasher.finalize_fixed();
    let pointer = result
        .to_vec()
        .as_ptr();
    mem::forget(pointer);

    pointer as *const c_void
}

Then all I need is to use this task in a build script. Let’s see how it performs…

The WebAssembly runtimes

I wrote 2 different implementations of the WASM integration runtime: the first one was pretty straightforward to write and makes use of wasmer-java. The second one took me significantly more time to implement and is using GraalVM.

Integrating Wasmer was easy for different reasons: 1. it’s just a library which you have to add to your classpath 2. it’s relatively well documented

GraalVM was more complicated because:

you actually need to run your program on GraalVM
you need to install the WASM support separately (it’s not downloadable as a regular Maven dependency, for example)
you still need to add the GraalVM polyglot API on classpath
it’s poorly documented at this stage (in particular, there’s no documentation whatsoever on how to share memory between the Java host and the WASM guest)

Anyway, I think (but I haven’t done it yet) that the GraalVM runtime will be easier to support I/O since it already offers the configuration options to let the WASM host access the host file system. Wasmer doesn’t support I/O yet.

Let’s talk about performance, now. Disclaimer: this isn’t proper benchmarking. The results you will see were obtained via functional testing of a plugin. There’s a lot of variance, but they were reproducible.

Measuring the wasmer runtime

In short, the wasmer runtime is very promising: it’s easy to setup and actually performs extremely well. The API is not very Java friendly, but the abstraction layer I wrote (which supports both wasmer and GraalVM) makes it significanly easier.

Here are some results for a memoized Fibonacci, which compares a version I wrote in Java vs the one I wrote in Rust:

Memoized Fibonacci on wasmer

Java fib(90) = 2880067194370816120
Took 3ms
Precompiled Rust fib(90) = 2880067194370816120
Took 366μs

The WASM version compiled from Rust is already faster than the Java version!

Let’s see how it performs when hashing a MD5 file (remember that for this use case, I’m actually passing a byte array to the WASM program, not a file):

Hashing a 4MB file on wasmer

hash from Java is 49DFDCEF6751973A236D35401B6CBFC8
Took 64ms
hash from Rust is 49DFDCEF6751973A236D35401B6CBFC8
Took 58ms

Again, the WASM version is still faster!

On both operations, the WASM binary performs better than the pure Java version. However, there’s a catch: in order to reach that level of performance, the WASM binary has to be precompiled to a native binary by wasmer. This, already takes time. If you include this in the whole picture, the numbers are different: 36ms for Fibonacci (compared to 3ms in Java, 10x slower). However, this is in practice not a big deal since those binaries can actually be cached, meaning that if we have to call them multiple times, or from different builds, we can actually fetch them from the cache.

All in all, it means that the wasmer runtime is very fast and integrates quite well with Java.

Measuring the GraalVM runtime

The WebAssembly support for GraalVM is still experimental. However, it has the advantage of taking advantage of the Truffle API, which promises better integration between languages and, eventually, better performance.

In my case, that wasn’t quite true. Again as usual don’t trust benchmarks, but here are the numbers:

Memoized Fibonacci on wasmer

Java fib(90) = 2880067194370816120
Took 3ms
Precompiled Rust fib(90) = 2880067194370816120
Took 21ms

This time, the WASM code is significantly slower. The explanation is probably that contrary to the wasmer runtime, the WASM binary has to be parsed and transformed into a model that the Truffle API can understand, and as far as I could tell, this is not cacheable. However, this isn’t the only explanation, as we can see with the hash example:

Hashing a 4MB file on GraalVM

hash from Java is 49DFDCEF6751973A236D35401B6CBFC8
Took 57ms
hash from Rust is 49DFDCEF6751973A236D35401B6CBFC8
Took 407ms

Again we can see that the performance is significantly worse with GraalVM. I must again say that maybe I’m not using the API properly. In particular, I have found no better way to pass the byte[] to the WASM memory model other than byte by byte!

What we’ve learnt

In this blog post, we’ve seen that we can use a wasm binary in Gradle as the implementation of a task. This binary can be written in any language which supports compiling to WebAssembly. In my [test project], I have written tasks in 2 different languages: Rust and AssemblyScript.

We’ve seen that we can integrate WASM binaries using 2 different "runtimes":

Wasmer, which is using JNI and compiles, as far as I understand, WebAssembly binaries to native code
GraalVM, which is a different Java Virtual Machine, which usually performs extremely well with Java, and provides a Polyglot runtime leveraging the Truffle API.

As of today, the Wasmer version performs significantly better and WASM functions can be executed even faster than Java code! The GraalVM version is still experimental and performs quite poorly compared to using native Java code. It’s also more painful to test because it’s not enough to install GraalVM: you also have to install components separately, which is not build friendly, nor CI friendly.

The next step for me is to try to integrate more directly with the file system: at the current stage, none of the approaches is suitable for Gradle as we need to read and write files.

Also, one has to keep in mind that it’s pretty rare that you’d like to integrate in a build arbitrary code like this: in general, you want to call external tools (javac, gcc, …). Nevertheless, this experiment is quite fun and I’m going to experiment more with this annotation processing API, which, I think, would be valuable in any case.

The problem with Gradle: really?

Wed, 6 Jan 2021 00:00:00 +0100

Disclaimer: I’m a former Groovy committer and a Gradle Inc. employee. However, I started using Gradle long before I joined Gradle, and I have quite some experience with Java build tools: I started with Ant, but I also used Maven quite a lot, then fell in love with Gradle, for many reasons.

A blog post from Bruce Eckel made quite some noise on Reddit (no wonder, Bruce is a guru!). It’s not the first time I read comments about Gradle, this article in itself is balanced and not that negative. Most of the criticism is on the title itself, which is misleading and looks like a clickbait. One can easily be retweeted without actually reading the details, and most importantly the conclusion. Also, it doesn’t compare with what you’d have to do with other build tools, which is important to draw good conclusions.

What makes Gradle different?

At the core of this blog post is the thing that the "big picture" of Gradle is unclear. There are several things to say here. First of all, Gradle is not a new, shiny toy to play with: it was born more than 10 years ago, and saw its v1.0 release on Jun 12, 2012. Second, there’s not a "single" problem with Gradle: there are different problems, some more important than the others, but there are also many, many benefits. As such, one should draw a complete picture, which is sometimes hard for build tools, something that developers tend to underestimate, in particular in terms of impact on developer productivity.

Gradle is designed to improve developer productivity which is achieved via incremental builds, incremental tasks, parallel execution, reproducibility and many other features. Let’s put it this way: if you look at Gradle from the DSL point of view, that is, the look and feel, instead of its capabilities, the engine, you will miss the most interesting parts. The beauty of Gradle is not in its "flexibility" offered because it uses configuration as code: it’s the engine which is designed to model software properly and optimizes building software.

Do you have to know everything?

Bruce states that with Gradle, "To do anything you have to know everything".

While the formula is elegant, this is a bold statement: this simply doesn’t align with the reality of the Gradle ecosystem, our community and larger user base.

What is true, however, is that the learning curve of Gradle is higher than with other build tools (say, Maven for example). I always said that the learning curve of Gradle is difficult, but once you get it, the benefit is just amazing: we have customers who tell stories about dramatic productivity boosts after migrating to Gradle, simplified maintenance and coordination of large development teams via corporate plugins.

One of the reasons is that Gradle is fundamentally different from other build tools: it focuses on modeling software components and their relationships, and as such it can be considered as a language for building any kind of application. It raises the concept of convention over configuration to another level.

That’s no surprise that Gradle has been successfully used to build software in various ecosystems: the JVM, where it’s born (with Java, Groovy, Scala in particular), but also native development (take a look at Nokee if you are interested), Go (Go Gradle), …

Some ecosystems actually build on top of unique features of Gradle, in particular in terms of dependency management: Android (with multiple flavors of libraries and applications) or Kotlin in particular (with Kotlin Multiplatform).

Why is that? Gradle is an execution engine optimized for build workflows, with intelligence about how to build things incrementally.

Nonetheless, improving the experience of first time users is something the Gradle team focuses on, as we will see below.

Tasks vs dependencies: a simplistic view

Bruce says that any build system is basically the combination of two essential ingredients: tasks and dependencies, where dependencies are seen as artifacts.

That is not quite correct in case for Gradle: Gradle is responsible for wiring units of work together, either through explicit or implicit dependencies. It’s important to understand that "dependency" here has a wider meaning: it’s something which is required to be able to execute a unit of work. There is, technically speaking, no difference between an external and an internal dependency. What matters is that we have units of work, which have inputs and outputs. The role of Gradle is to wire things so that everything is ordered properly and optimized for execution (parallelism, caching, …). It’s a mistake to think that there are only tasks and dependencies (files, mostly): there are work units, inputs and outputs. A task (say, JavaCompile) is just one kind of work unit, but that’s not the only one. Here are some other kind of work units: transforming artifacts (for example, transform a jar into something else before it is consumed), downloading a toolchain, uploading an artifact to the build cache, … A task itself can be splitted into several work units in order to maximize incrementality and parallelism.

A lot of the criticism we see on Gradle is because there’s a mismatch between the mental model of what Gradle is, and the surface, which is the language(s) it uses to configure the actual model.

A lot of the confusion comes from the fact that while Gradle has a vision, it should be better shared and explained. It’s also victim of its age which means success: the road towards fine grained execution, safe parallelism, configuration, … is hard and there are some APIs which are sub-optimal for this, and maintaining backwards compatibility is a day-to-day challenge.

Let’s explain a bit what Gradle has to do, when you say that you want to execute a task:

configure the task (execute its configuration phase), which means executing plugins which configure the default values, use your configuration, etc.
compute the dependencies of the task, which can either be explicit (typically the dependsOn clause) or implicit (because you configured a task input as the output of another task, typically): this is basically a directed graph resolution engine, which is very fine grained and where each node is a unit of work.
and finally executes the nodes of the graph in an optimized way

Historically, Gradle has separated the configuration phase from the execution phase, but it doesn’t have to be that way: as soon as the inputs of a task are ready, we should be able to execute it, and we shouldn’t have to wait for the configuration of other tasks to be ready to do it. Also Gradle made it easy, in the beginning, to inline the definition of tasks, including their execution phase, in a build script, but it is, for quite some time, considered a bad design principle: don’t write build logic in build scripts. The whole blog post from Bruce never mentions this term a single time: write plugins!

Despite our efforts, lots of resources on the web still focus on how to declare tasks, how flexible Gradle is, without taking time to leverage good engineering practices. In Java nobody writes a giant single class with all the logic and the data as code, right? So don’t do it in your build, that’s as simple as that.

Let’s be honest: it’s partly our fault, and for that I agree with Bruce: the documentation of Gradle is huge and things can be made better, especially for beginners. It’s also true that the docs contain outdated patterns and it’s often very difficult to realize that you have outdated info in thousands of pages of docs.

There is, however, a good amount of resources for beginners:

samples: https://docs.gradle.org/current/samples/index.html
guides (slowly replaced with the samples above): https://gradle.org/guides/
free trainings: https://gradle.com/training/
webcasts: https://gradle.com/blog/tag/webcast/

Deconstructing the myth

We are still in the early days of the “adding a build system atop an existing language” paradigm. Gradle is an experiment in this paradigm, so we expect some sub-optimal choices. However, by understanding its issues you might have less frustration learning Gradle than I did.

— Bruce Eckel
The Problem With Gradle

This is incorrect. Thinking that Gradle is just about calling methods is the wrong mental model. Thinking that Gradle is "programming a build" is wrong. Gradle uses a programming language as a foundation for "configuration as code", but it is really about configuration, not programming. Surely a programming language is interesting to use in the "configuration as code" paradigm, but focusing on that aspect and assuming that it works as a general purpose language is wrong and will inevitably lead to confusion.

Gradle is also not an experiment: it’s the most advanced tool you can find in this area, and it works because the language builds on top of an engine which is made to execute workflows. A number of build tools, including modern ones, are only fast because they put the maintenance and optimization burden on the build author: declare everything, re-generate build scripts, etc. We, in the Gradle team, think that we can be more correct, more performant without having to compromise on user experience.

We even had teams challenging us on performance with modern tools like Bazel, and we proved that with unique features like configuration caching, Gradle was able to outperform it in almost all scenarios: you get the benefit of terse, maintainable scripts, with performance. win-win.

At this stage, it’s time to deconstruct some misconceptions of the blog post, because as long as this message is propagated, we will not achieve what should be the focus of our industry: safer, faster, reproducible builds for everyone.

1. You’re not Configuring, You’re Programming

You should see Gradle build scripts as configuration scripts, which actually configure a model. The surface is a DSL, a language, but what you do, what you should do, is to declaratively model your software.

For example, this Groovy script:

plugins {
   id 'java-library'
}

dependencies {
    api 'org.springframework:spring-core:2.5.6'
    implementation 'org.apache.commons:commons-lang3:3.3.10'
}

Is a build script which configures a library written in Java, while this build script, written in Kotlin, configures a different kind of software component:

plugins {
    id("org.gradle.presentation.asciidoctor")
}

presentation {
    githubUserName.set("melix")
    githubRepoName.set("gradle-6-whats-new")
    useAsciidoctorDiagram()
}

which is actually a reveal.js slide deck, built with Gradle! This is the full build script for this, and it’s extremely important to realize that: the "code" that you get shouldn’t have any custom tasks, in particular: all of the complexity, which is nothing more than the "model", is hidden in a plugin.

I strongly encourage you to read my Gradle myth busting: scripting blog post which describes exactly that.

In addition, you can watch this 10 minute video I made about writing idiomatic build scripts:

What it means is that while Gradle build scripts are executed, this is code which configures the model, so you’re "programming the configuration", if you will.

2. Groovy is Not Java

There’s not much to say here, because it barely has anything to do with Gradle.

You will notice that when I showed build scripts above, I showed a Groovy build script and a Kotlin one. It’s important to notice that Gradle allows both, because under the hood, it’s an API. As I said, Gradle provides a foundation for building software components, a dependency resolution engine, an execution engine. Plugins are at the core of the system and are responsible for building models of software we build: a Java library is different from an Android application, so there’s no reason to have the same source layout, for example. The fact that Gradle uses Groovy as a language to do its configuration is an implementation detail. I think it’s a mistake to consider that you need a programming language to do what Gradle does: it helps, and lots of people actually appreciate Gradle’s flexibility in that regard, but it’s not what you should focus on.

Both Groovy and Kotlin have pros and cons. If you use Intellij IDEA, for example, the Kotlin DSL has very good arguments and makes this model completely visible: completion is available, you don’t have to "guess" what to type: depending on the plugins you apply, you get the configuration blocks you need, and nothing more!

To come back to this section, a common misconception is that Gradle is written in Groovy. It’s not. Gradle is written in Java, mostly. There’s a lot of Groovy code in Gradle for testing (we use Spock, in particular), but we also have Kotlin code. The build scripts, however, use Groovy or Kotlin.

The DSL design definitely was influenced by Groovy, though, that’s very true.

3. Gradle Uses a Domain-Specific Language

Yes it does. I should say it’s an "extensible" DSL. But when I’m reading this:

How helpful is this DSL syntax, really? I have to translate it into function calls in my head when I read it. So for me it’s additional cognitive overhead which is ultimately a hindrance. The DSL operations can all be done with function calls, and programmers already understand function calls.

I’m thinking that this is again seeing the problem from the wrong angle. You should not see this as function calls, or code being executed. You should see this as a model being configured. You can’t, actually, assume when this code is going to be called (because, we have configuration avoidance, for example). So a DSL is really what it means: it’s a language meant to configure the model, nothing more. By trying to interpret how Gradle does this, you’re actually distracted from what matters: what are you trying to build?

4. There are Many Ways to do the Same Thing

This one is one of my favourites. I always read "there are too many ways to do the same thing". Sure there are. Just like in Java, can you tell me how many ways you can write a loop? Let’s see…

indexed loop

for (int i=0; i<items.length(); i++) {
   ...
}

foreach loop

for (String item: items) {
...
}

while loop

Iterator<String> it = items.iterator();
while (it.hasNext()) {
  ...
}

(not mentioning the do…while variant)

streams

items.stream()
   .map(...)

streams again

items.forEach(...)

Does anyone complain that Java has too many ways to do the same thing? Certainly not, because there are different reasons: historical (good old loops came first), performance (sometimes streams are not that fast), semantic (iterator lets you delete things while iterating), … Even Python has many ways to do the same thing. Isn’t it the case when you’re free to write code? Some consider it’s a reason alone not to use Gradle. It would be a mistake, because you don’t stop programming Java because it has too many ways to write a loop.

The thing is "use the right tool for the right job". Gradle is no different. Except that there are not so many ways to do things 😉 Some are legacy patterns, and there we need to do better at documenting, some are just because people are trying to write code in their build scripts when they shouldn’t (write plugins!).

I should mention that we started something called the idiomatic Gradle project for these excellent reasons. The goal is to encourage best practices, document the most idiomatic way to do something, covering multiple use cases and their differences. There will be a blog post from the team about this project, in the meantime you can already read some results:

a sample showing how to build multi-project Java applications
documentation about structuring larger builds
a presentation from my colleague Jendrik Johannes about idiomatic Gradle at JokerConf

Last but not least, we have ongoing conversations about a "strict mode" in Gradle to actually enforce some language constructs, or even a different DSL, to limit what you can actually do in a build script.

5. Magic

Any sufficiently advanced technology is indistinguishable from magic.

— Arthur C. Clarke
Profiles of the Future: An Inquiry Into the Limits of the Possible

This whole section is basically about understanding the separation between configuration and execution phases, but really what bothers me most is that it’s much easier to grasp if you just don’t write inline tasks into build scripts.

For example, if I write, in a plugin:

public abstract class ProcessTemplates extends DefaultTask {

    @Input
    public abstract Property<TemplateEngineType> getTemplateEngine();

    @InputFiles
    public abstract ConfigurableFileCollection getSourceFiles();

    @OutputDirectory
    public abstract DirectoryProperty getOutputDir();

    @TaskAction
    void execute() { ... }

then the separation between inputs (properties annotated with @InputXXX), outputs @OutputDirectory and the actual task execution @TaskAction is obvious. More importantly, if you do this Gradle helps you by providing you guidance on what annotations to use, how to make it compatible with caching, etc for free (via embedded linting).

I repeat, nobody should every inline build logic in a build script. Nobody.

Accidentally, the scripts Bruce shows in this section are using Groovy, which, for backwards compatibility reasons, uses eager configuration APIs. If you use Kotlin scripts, all the task creation, registration, is done lazily, avoiding unnecessary configuration. You can do this in Groovy too, if you use the configuration avoidance APIs.

5bis, the lifecyle

This section again discusses configuration vs execution. Just a terminology comment here: a lifecyle in Gradle and other build tools is a different thing. See my other Gradle myth busting: lifecyle blog post for reference.

Other Issues

The Gradle documentation assumes you already know a lot.

Yes, and no. The Gradle documentation is large and as I said in the intro, we do have sections for beginners. What is true, and I’m quite mad about this, is that our docs layout is terrible, and that good documentation is very difficult to find out.

We also started to rewrite sections so that they are use case centered, see the dependency management docs for example.

Slow startup times

In most cases, "slow startup times" actually refer to "slow configuration times". This is in general because of too much code being executed at configuration time. That’s why we have configuration avoidance APIs and that you can possibly use build scans which tell you that you are doing too much at configuration time. However, with the experimental configuration cache, those configuration time issues should soon be a thing of the past.

Be careful with some (very) popular plugins: some plugin authors just don’t realize that the way they configure builds, reach to other projects or perform cross-configuration has a significant impact on build performance, and therefore user experience.

All that said, we’re doing even more: I mentioned the configuration cache earlier. This is still work in progress, but we have amazing results with this. Very large multi-project builds are configured very quickly. We’re working with customers and the Android team on this.

It’s not that easy to discover Gradle’s abilities, and there are so many abilities that you often don’t know what’s possible

Oh yes. Oh, yes. Gradle is probably the most advanced build tool of the market. It has dozens of features, including unique features you don’t find in any other build tool. They are hardly discoverable. We need more advocacy on those, and make them more visible. I am not sure how we can do this, honestly.

Now That I Get It

The conclusion of the blog post is quite positive. However it mentions IDE support. I should say, despite being a Groovy developer, I use it everyday, that if you use IntelliJ IDEA in particular, discovering Gradle using the Kotlin DSL is much easier. You get smart, contextual, IDE completion, which really helps in understanding how Gradle works. There are drawbacks in using Kotlin build scripts, in particular because Kotlin compilation is painfully slow (Scala developers would understand). Independently of the JetBrains team, which is working on performance of the compiler, Gradle 6.8 will also provide some nice Kotlin DSL performance improvements.

Conclusion

In this blog post I have commented on some remaining misconceptions about Gradle, which are still visible in Bruce’s blog post. However, there’s no blocker there. I would however really encourage you to go the idiomatic Gradle route and try to follow modern Gradle design principles. If you have time, here’s a talk I gave recently at the Madrid Groovy User Group about modernizing the Groovy build, I think it summarizes quite a lot why a number of the patterns described in Bruce’s blog post are outdated and shouldn’t be promoted:

Dependabot and automated dependency upgrades

Sun, 11 Oct 2020 00:00:00 +0200

Dependabot starts with a great idea: the state of our industry, in terms of dependency management, is sad. In particular, maintaining dependencies up-to-date is difficult and while some solutions exist, the vast majority of projects are out of date. This isn’t a big deal for bugfixes and feature updates, but it is clearly a problem for security issues.

Therefore, any kind of automation to make the situation better is a good idea. I am in deep empathy with people working in this area: this is a difficult problem and there are no perfect solutions: as an example, in order to improve security of builds and reduce the risks of attacks via the toolchain, Gradle introduced dependency verification as a first-class citizen. We deeply care about security and definitely support any attempt to make things better in this area.

The genius idea about Dependabot is that it does it at scale and submits pull requests that you just have to review and merge if everything goes ok.

The best example in the Java ecosystem happened yesterday: the JUnit team fixed a security vulnerability in JUnit 4.13. At first, you might wonder what kind of security problems you can have with a testing framework, which, by definition, just like a build tool, runs any arbitrary code on your machine. The reality is that most people are unaffected by this problem: it would only be the case if your tests write sensitive data (like credentials) in a shared environment, or that you somehow expose whatever is written in the test directory to the outside world.

Nevertheless, because it’s a CVE, Dependabot submitted automated pull-requests to all projects using JUnit 4 and Dependarmaggedon happened: tens of thousands of pull requests. This might sound cool, a security fix needs to be fixed, but I find this noisy, and in the end dangerous.

Here’s why.

Automatic detection

Dependabot is as good as an automated tool can be. I mean that its implementation, at least in the Java and JVM ecosystem in general, is too simplistic and that it’s inherently wrong. I don’t have the expertise to tell if it’s different in the other supported ecosystems (Go) but I suspect a lot of the problems I’m going to describe here exist also in those ecosystems.

In practice, Dependabot is nothing more than a search and replace engine: it looks into some particular files that it knows about (think pom.xml), looks for some patterns, then tries to identify dependency coordinates and eventually replaces them. I’m simplifying a bit, because it does some interpretation of the versions it finds, for example to match version ranges, but that’s roughtly how it works.

For example, it will identify this:

<dependency>
   <group>junit</group>
   <name>junit</version>
   <version>4.13</version>
   <scope>test</scope>
</dependency>

and it will replace the 4.13 with 4.13.1, then submit a PR for you to review. WIN!

Yes, and no. Because this is extremely easy to defeat.

First, as I said, it only looks for known files. For Maven, it’s pretty simple, there’s only one place to look for: pom.xml files, but for other build tools, like Gradle, Ivy, Ant, Bazel, Pants, … that’s not the case: dependencies can be declared in different files, not all named the same. Dependency versions can be separate (in properties files) or inherited from context (parent POMs, platforms, …).

A more complete strategy is to look for patterns too, like .gradle, but then, they also miss the .gradle.kts files (they do), or the plugins which can contribute dependencies. In short: it’s a whack-a-mole game.

One must think that it’s not a big deal, because catching some cases is better than not catching at all, and I agree, but that’s not the expectation when you read what the tool promises.

In the end, any tool which tries to parse build files and tries to interpret its model based on string evaluations is doomed to fail.

Even if it manages to catch some problems, things can go wrong. As an example, here’s what we got at Gradle for this JUnit upgrade:

For a single repository, it created dozens of individual pull requests to ask us to upgrade. That’s nice of you, Dependabot, but it’s totally missing the point: this repository contains generated code, used in performance testing. The way the code is layout, the repetition of dependency declarations, etc, everything is intentional.

If you put apart the fact that Dependabot could have created a single pull request for all dependencies, instead of creating hundreds, it highlights a general problem with those automated detection tools:

Context matters!

At Gradle, we often receive emails from security teams telling us things like: "The Gradle distribution contains a dependency on this X library which has vulnerability Y". However, in almost all cases we’ve investigated, this was a no issue for us. Because of the nature of Gradle, the fact that by definition it runs arbitrary code on a machine, most vulnerabilities are not a problem in this environment: context matters. What’s more, even if your tool uses a library that has a critical vulnerability (we use Jackson for example, which had a number of thems recently), it doesn’t mean that your code exercises the path which is vulnerable: context matters!

It’s not harmful to upgrade, but it creates a lot of noise.

Wrong impression of security

But the biggest problem with such tools, and again I’m using Dependabot as an example, is that they give a wrong impression of security.

As I explained, Dependabot will basically read your POM files, or Gradle files, or whatever build tool files you use, and assume that the version it reads is the version which is going to be used. We, as developers, and even more as build tool authors, know that this is wrong: the dependencies you declare are rarely the ones you resolve.

That’s the reason why tools like NPM (Javascript), Gradle (multi-language) or Cargo (Rust) make use of dependency locking. Dependency locking is a concept which helps with build reproducibility. Say, for example, that you write:

dependencies {
   implementation 'org.mylib:awesome:[1.0, 2['
}

This means that Gradle can pick whatever version of awesome as long as it’s in the range. This is a typical declaration for semantic versioning, when you expect a library to be upgradable up to the next major. In Gradle you can even improve this by explaining what version you prefer in the range, if nobody else cares:

dependencies {
   implementation('org.mylib:awesome') {
      version {
          require '[1.0, 2['
          prefer '1.3'
      }
}

However, ranges come at a price: if a new dependency version is released, your build would suddenly start using this new version, which breaks reproducibility in case you checkout old code, for example, for a bugfix. Therefore, as soon as you use ranges (and by extension any dynamic dependency version), you should use it with dependency locking. Dependency locking will make sure that whatever version of a library resolved at time T is used at time T+N even if a new version is out. It does so by writing the result of the dependency resolution process in a lock file.

Upgrading dependencies becomes an intentional process that you do from time to time by generating a new lock file.

What is the relationship with tools like Dependabot?

By using the versions you declare, they have absolutely no guarantee that the versions you resolve are not vulnerable.

In the example above, imagine that I use range [1.0, 2[ and that I’m resolving 1.3. At some point in time, version 2.1 is released and is vulnerable. You could imagine that because you used a range and that 2.1 is out of the range, you are safe. That’s what those tools would assume, but they would be wrong: the reality is that despite this declaration, by the play of conflict resolution (Ivy, Gradle, …) or ordering (Maven), a totally different version can be selected, even 2.1!

Again there are tools to mitigate this problem, but the reality is that by just reading the declaration, you’re not safe.

It’s even worse than that: what about transitive dependencies?

Imagine that you have:

Foo → Bar → Baz

And that you depend on Foo. What if a vulnerability is discovered on Baz? Will you be notified? What kind of automated pull request can such a tool make to make sure that you upgrade Baz?

Can we do better?

I have some good and bad news for you.

The good news is that we can do better. The bad news is that it’s not easy.

First, instead of relying on the declaration, those tools should really rely on the result of dependency resolution. If, for example, they used the lock files instead of the build files, they would know exactly what a build resolves. This, however, is only possible if the build uses dependency locking, and only for dependency graphs which are actually locked. It’s a reasonable assumption to say that what is locked is the most relevant information, though.

Of course, I said there were bad news. As soon as you use the resolution result, you almost have to give up on automated remediation (pull requests). One thing they could do is patching the lock file. However, this is in general not a good idea, because, as I explained, lock files are generated: they present to you the result of resolution. Partially upgrading a lock file, manually, is possible but then you cannot make any guarantee that the app is going to work, because introducing a different dependency version may introduce different transitive dependencies!

An alternate solution is to gather the information about resolved dependencies during the build: this is what build scans do for Gradle and Maven, for example. This information can be extracted during the build and Dependabot would know precisely, reliably, what is resolved by a project. We even offer a Tooling API to do this kind of work.

Then there’s remediation. This is the hardest problem. Because what most people like about Dependabot is actually the automated remediation: pull requests are nice, we all love that.

But say, that to fix the transitive dependency issue above, Dependabot suggested to add a first level dependency with a different version. For Maven, this would work, since it’s order and depth sensitive. But it would break your model: the application doesn’t depend on Baz: it depends on Foo, which, by the transitive game, depends on Baz. You don’t want to introduce a first level dependency on Baz because it doesn’t make sense.

For Gradle, you could use dependency constraints instead: constraints are meant for this use case. A constraint adds, as it name implies, a constraint to the equation of the resolution of the graph (a bit like in constraint programming). They would participate in the dependency graph resolution if, and only if, the dependency they talk about appears in the graph. In that sense, they don’t break the application model, by introducing arbitrary first-level dependencies.

Our Java ecosystem is polluted by hundreds of accidental first level dependencies and exclusions because of this lack of modeling: it is important to get things right.

Last but not least, how you declare dependencies matters. In Gradle, using rich version constraints, you can explicitly reject bad versions, and you can explain why.

Conclusion

In conclusion, I think that Dependabot’s intent is legitimate and that today it’s better than nothing. Let’s detect projects using vulnerable dependencies and propose automated remediation.

We, as build tool authors, also need to consider the wider context, which is dependency resolution in general, which isn’t as simple as it seems. In particular we consider that detection is an easy problem if you use the right tools, while remediation is a hard one.

I think the current implementation of Dependabot is mostly wrong (at least in the Java ecosystem) as it relies on the declaration. This raises a number of issues: - it is dependent to the patterns it recognizes - it assumes that what you see is what you get - it cannot recognize transitive dependencies actually resolved by your project, so it misses real vulnerabilities - it doesn’t matter about the context of use of your dependencies

The context thing is difficult to solve, but it’s actually painful because of the noise it creates, in case the "vulnerable dependency" is actually not in your case. However, I think there are improvements which can me made by actually using the actual resolution results instead. Then it raises some interesting technical challenges, like how to sandbox execution of builds (GitHub actions already do this) but more importantly how to create an automated pull request from the result of the analysis.

Note that I also understand that from a Dependabot creator point of view, having to implement build-tool specific logic, like calling the Tooling API, to gather information about resolved dependencies might sound scary. I still think this is the right thing to do to be correct and, if our goal is really to make the industry safer, that’s what we should do. However, we have alternate solutions. For example a few weeks ago I experimented with a fork of my friend and colleague Paul Merlin’s Gradle Command GitHub Action which automatically generates a JSON file of resolved dependencies during the build.

Schalk Cronjé also mentioned to me the OWASP plugin, which I forgot to mention when I originally wrote this blog post, but I think it a great answer and currently better answer because it does exactly what I describe: rely on what you actually resolve, not what you declare, and lets you carefully review the results via a generated report.

I’m not sure this is the best answer, but it shows that we can attack the problem from different angles.

Eventually, the key takeaway of this blog post should be: don’t assume that you are safe because you use Dependabot. You’re not.

Configuring git to use main instead of master

Wed, 17 Jun 2020 00:00:00 +0200

I don’t need to explain the backstory, and I am not interested in a discussion whether this change makes sense or not, nor am I interested in the technical problems it could cause if someone changes the name: technical issues shouldn’t be a reason to stop us from improving.

Actually, if you are reading this, there are good chances you’re already in the mindset of removing potentially offensive language from your project, thank you!

The fact is, we have better, more descriptive technical names than "master" and "slave" and we don’t have to use terms which refer to dark ages of our history, so let’s get rid of them in tech.

In this blog post, you will learn:

how to make git init create your project with a main branch instead of master
how to rename your master branch to main
how to make the change on GitHub

Be aware that renaming branches may break your existing integrations (typically with CI or Slack notifications) so only do it when you’re ready.

Configuring git init

By default, when you create a new project, git init will create a master branch. There’s no option in git init to change this, but it’s actually quite easy to set it up differently.

First, in your ~/.gitconfig file, add the following:

~/.gitconfig

[init]
    templateDir = ~/.config/git/template/

Then create, if it doesn’t exist, the following directory:

mkdir ~/.config/git/template

next create this file:

~/.config/git/template/HEAD

ref: refs/heads/main

(don’t forget to add a new line at the end of the first line)

Starting from now, calling git init will create a main branch instead of master!

Now let’s deal with existing projects.

Renaming your master branch

For your current projects, you may want to rename the master branch to main. This is also single command in git:

git checkout master
git branch -m main

Then you can push the branch to your remote… but be careful, it will create a new branch on the remote:

git push origin main

Configuring GitHub to use `main` instead of `master`

Once you’ve pushed your new branch, you need to tell GitHub that this is your new "default branch". To do this, go to Settings → Branches and select your new branch as the default one:

Last but not least, you need to delete the old branch from GitHub, which can be done by calling this:

git push origin :master

Done!

Using Java feature previews with Gradle

Thu, 11 Jun 2020 00:00:00 +0200

This blog post has been updated with the latest Java Toolchain support from Gradle, making it even easier!

You’ve probably heard about Java providing features like records, multi-line text blocks or sealed types and you’d like to try them.

Those features are called feature previews and it means a couple of things:

the Java team wants you to test them and give feedback. They want honest feedback about how it feels to use them, whether you like them or not. Both ways, feedback is important.
because they are feature previews, you shouldn’t them use in production, but you can play with them for toy projects.

There’s actually an important thing about point 2 if you are a library author: never, ever publish a library which uses feature previews on Maven Central. The reason is that the feature previews leak to consumers: as soon as you start using them, any project depending on your code will also have to enable them. This is not a problem for toy projects, it’s clearly a problem for published libraries. In particular, there are no guarantees that the generated bytecode will be compatible with future Java releases, and there are no guarantees that the feature preview will make it to Java eventually.

The goal of this blog post is not to explain what records or sealed classes are, there’s already a lot of litterature about it. Instead, we’re going to explain how to configure your Gradle build to use feature previews and therefore report issues/bugs to the JDK team.

Configuring your Gradle build

The code in this blog post is available on a GitHub repository.

First, make sure you are using latest Gradle versions. If you use the repository above, the Gradle wrapper will make sure that you do. Gradle 6.5, for example, works perfectly fine with Java 14 and even Java 15.

Imagine that you want to compile the following record:

public record Person(
        String firstName,
        String lastName,
        boolean likeJavaRecords) {

}

And execute this test:

class PersonTest {
    @Test void testRecord() {
        Person cedric = new Person("Cédric", "Champeau", true);
        Person otherCedric = new Person("Cédric", "Champeau", true);
        assertEquals(cedric, otherCedric);
        assertTrue(cedric.likeJavaRecords());
    }
}

Then you need to configure the build to do a couple of things:

First, we need to tell the Java compiler to use feature previews:

tasks.withType<JavaCompile>().configureEach {
    options.compilerArgs.add("--enable-preview")
}

Then we need to tell the test runtime to use feature previews:

tasks.withType<Test>().configureEach {
    useJUnitPlatform()
    jvmArgs("--enable-preview")
}

And that’s it, you can run the build with ./gradlew test.

Configuring Gradle to use a different JDK from its own runtime

Usually, people use the same JDK version for running Gradle, than they use to compile and execute tests. It doesn’t have to be that way, and actually the Gradle team added support for toolchains to simplify the configuration of builds in this case. All you need is to tell what JDK to use and Gradle will take care of finding a JDK on your machine which matches the spec, or download it!

java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(15))
    }
}

Now you don’t have any excuse not to try feature previews!

Mon engagement pour Montaigu-Vendée

Sat, 7 Mar 2020 00:00:00 +0100

J’ai 40 ans et je suis arrivé sur Montaigu voici 17 ans. Comme beaucoup d’entre nous, c’est pour des raisons professionnelles que nous y avons emménagé avec ma jeune épouse. Elle avait trouvé un travail sur Les Herbiers, je travaillais sur Nantes, Montaigu était l’endroit idéal pour nous. Nous voici en 2020, nous avons 2 grands garçons, et si notre situation professionnelle a évolué (nous sommes tous 2 télétravailleurs) et que nous aurions pu déménager, nous avons au contraire choisi de faire notre vie ici et sommes attachés à ce territoire. Il y a 6 ans, nous quittions notre petit logement sur Montaigu et achetions donc notre maison sur Saint Hilaire de Loulay, en village. Après presque deux décennies passées ici, après nos engagements associatifs, nos nouveaux amis ici, il était temps, enfin, d’entrer au service des citoyens de Montaigu-Vendée.

La liste Montaigu-Vendée Ensemble et Autrement

Si le territoire de Montaigu-Vendée est riche et, économiquement, le territoire est attractif, il y a beaucoup de choses à améliorer. J’ai choisi de m’engager sur la liste de Vincent MATHIEU, Montaigu-Vendée Ensemble et Autremement parce que je suis convaincu qu’une autre voie est possible, basée sur 3 piliers qui sont aussi chers à mon coeur:

la transition écologique, parce que je suis convaincu que c’est en travaillant au niveau local, en engageant des actions simples à notre niveau que nous feront de ruisseaux de grandes rivières
la démocratie participative, où comment remettre le citoyen au coeur des décisions, avoir des élus au service des citoyens et non l’inverse
le mieux vivre ensemble, en prenant en compte tous les habitants, personnes âgées, jeunes, personnes en situation de handicap, de tous milieux sociaux et en exploitant notre très actif réseau associatif

La Transition Ecologique

Je suis particulièrement attaché à l’écologie depuis ma jeunesse. Lorsque je parle d’écologie, je prône toujours une écologie pragmatique, pas dogmatique et encore moins ésotérique. Je suis pour l’écologie qui fait avancer les choses dans la concertation, une écologie qui prend en compte tous les acteurs, entreprises, paysans, citoyens et enfants. Le 5 mars dernier, nous étions le "jour du dérèglement", ce jour de basculement où en France, nous commençons à produire plus de carbone que l’écosystème peut en absorber. C’est dire l’urgence de la situation, mais l’écologie ne se limite pas au prisme du réchauffemenent climatique. En campagne, il est facile de constater les ravages sur la biodiversité: en 40 ans, ce sont 60% des espèces sauvages qui ont disparu, 40 ans seulement! Comme le citait Antoine de Saint Exupéry, nous n’héritons pas de la terres de nos ancêtres, nous empruntons celle de nos enfants.

C’est ici une différence fondamentale avec la majorité sortante, qui, soyons clairs, n’a rien fait pour l’écologie. Pire, lorsqu’elle affirme dans les journaux que Vincent MATHIEU "n’est pas issu du cru local, ses parents n’ont pas de pieds de vigne dans le secteur", nous constatons avec effarement un tel mépris de notre génération, des générations futures, mais aussi des 600 nouveaux habitants qui chaque année, arrivent sur notre terroire. 600 habitants, que, au passage, les élus actuels sont heureux de voir participer à l’activité économique, heureux d’avoir pour qu’il soit, come une fois de plus annoncé dans les journaux, nommé Président d’une Communauté d’Agglomération qui n’existe pas encore (faute d’habitants suffisants !), mais dont il nie l’expression de la démocratie lorsqu’ils ne sont pas issus du secteur. Dont acte.

A titre personnel, je suis aussi attaché à la lutte contre la pollution lumineuse. Je constate, en tant qu’astronome amateur, la dégradation du ciel nocture. Je constate que les gens sont surpris lorsqu’ils découvrent, lors de la manifestation des Etoiles du Lac à La Guyonnière (dont l’existence est menacée par la majorité en place), qu’ils peuvent voir la voie lactée. Je constate aussi que nous n’avons, même ici en campagne, presque plus de papillons de nuit et que les insectes ont presque tous disparus. Il s’agit d’une catastrophe contre laquelle nous pouvons, tous ensemble, lutter.

Que dire de cette majorité, qui, à l’heure où l’on parle de planter des arbres, a rasé tous les arbres de la place de la mairie ? Que dire de cette majorité qui a coupé les magnifiques cerisiers en fleurs du Boulevard Auguste Durand pour les remplacer par… du goudron ! Que dire lorsque tous les cours d’eau du territoire ont leurs indicateurs de qualité dans le rouge ?

Alors peut-on faire mieux ? Avec un tel bilan pour la majorité sortante, ça n’est pas compliqué. Je vous invite à lire notre programme sur la transition écologique. J’insisterai sur ces points:

nous souhaitons aider toutes celles et ceux, entreprises ou particuliers, qui sont prêts à s’engager dans la transition écologique.
nous favoriserons l’implantation d’entreprises vertueuses, et aiderons les entreprises existantes pour qu’elles réalisent des économies d’énergie (et donc d’argent !) et réduisent leur impact. Ceci passera par la mise en place d’indicateurs et de mesure du progrès tout au long du mandat
nous aiderons les habitants de villages dans la mise au normes de l’assainissmenent. Des solutions écologiques et collectives existent, et aujourd’hui personne n’aide ces habitants dans des choix qui sont souvent extrêmenent onéreux.
nous favoriserons l’accueil de nouveaux paysans bio ou 0-pesticide et développerons l’alimentation collective par l’approvisionnement local, tel que celà se fait sur la Guyonnière
nous rendrons l’accès à la déchetterie plus simple et gratuit. Nous mettrons aussi à disposition des outils pour ceux qui ne peuvent pas facilement se déplacer (broyage, compostage, …)
nous développerons et sécuriserons les développements doux (pistes cyclables, chemins piétonniers, …) et mettrons en place des solutions de transport collectif entre les communes déléguées

La démocratie participative

Ah, la belle démocratie Montacutaine ! Qui, il y a 6 ans, avait lu qu’il y aurait fusion des communes dans les programmes ? Qui a été consulté pour cette fusion ? Nous avons bel et bien eu une consultation, celle pour choisir le nom de la commune. D’ailleurs, nous, habitants, avions choisi "Montaigu Les Maines", mais nos élus avaient déja choisi Montaigu-Vendée. Tout, absolument tout sur Montaigu est à cette image: les élus prennent les décisions dans leur coin, et informent les citoyens a posteriori. Ceci n’est pas ma vision de la démocratie. Même s’il fallait faire cette fusion, il était nécessaire d’en informer et de faire une concertation avec les citoyens. Nous ne sommes pas dupes, à Montaigu, l’élu est au service du Maire, ses ambitions personnelles annoncées par voie de presse, et non le contraire. Je ne suis même pas certain qu’ils se rendent compte de ce qu’ils font. Qui, par exemple, a déja vu sur l’affichage public l’annonce des conseils municipaux ?

Nous souhaitons remettre le citoyen au coeur des décisions, et celà se fera en mettant en place des structures adaptées, participatives: rganisation de conseils municipaux délégués, composés pour moitié d’élus et pour l’autre moitié de citoyens tirés au sort parmi les volontaires. Il ne s’agit pas, comme le propose l’autre liste, de choisir nous mêmes les participants, ça n’est pas notre vision de la démocratie: nous souhaitons une vision non partisane des choses !

Notre territoire est grand et aussi organisé en villages et quartiers. Il nous semble donc indispensable de refléter cet aspect dans les prises de décision, avec l’organisation de comités locaux, par thématique, afin que les riverains puissent prendre part aux décisions.

Il est temps que le citoyen redevienne le centre de la discussion publique.

Le mieux-vivre ensemble

Il s’agit encore d’un point qui me tient particulièrement à coeur. Notre territoire se doit d’être accueillant, quelle que soit la population. Nos élus, par exemple, on fait le choix de payer des amendes plutôt que de respecter l’obligation de 20% de logements sociaux. Ils mettent aussi en place des "nouveaux quartiers" tels que les Hauts de Montaigu, où le ticket d’entrée pour une maison de 110m2 et un terrain de 250m2 se situe aux alentours de 250k€ ! Nous pensons que nous pouvons, que nous devons faire autrement: création d’éco-quartiers intergénérationnels, par exemple.

Ces quartiers ont à la fois un intérêt écologique, mais aussi de relation sociale. Avec une partie de la population qui veillit et une autre partie de la population qui croit rapidement (pour des raisons économiques), nous pensons que nous pouvons organiser nos quartiers de manière à encourager la mixité sociale, la relation entre personnes âgées et jeunes en vivant Ensemble et Autrement.

Nous pensons qu’il y a une alternative aux EHPAD et qu’en particulier le maitien des personnes âgées dans un environnement au milieu d’actifs et de jeunes est à la fois une garantie d’un veillissement dans de meilleures conditions, mais aussi plus humaine.

Nous avons aussi la chance, sur notre territoire, d’avoir un tissu associatif extrêmement actif. Toutes ces bonnes volontés ont besoin d’être encouragées. Il est déja très difficile de trouver des bénévoles, alors que dire lorsque la municipalité ne s’engage pas ou, pire, tente de saboter des projets parce qu’ils sont jugés trop "populaires" ? Nous nous appuierons au contraire sur ces associations et nous mettrons en place la transparence dans l’attribution des subventions.

Nous transformerons aussi nos mairies annexes en maisons de services aux citoyen, de façon à favoriser l’interaction, mais aussi aider toutes les personnes en difficulté. Parfois, il s’agira de difficultés de logement, économiques, mais aussi parfois des difficultés dans l’appréhension des outils numériques (nous pensons par exemple à la télé-déclaration de revenus).

Nous pensons aussi à tous les citoyens, en particulier ceux en situation de handicap. La situation de la ville de Montaigu est catastrophique à cet égard: impossible pour une personne en fauteuil roulant de faire ses courses en centre-ville. Nous souhaitons, au contraire, renforcer les centre-villes en les rendant accessibles pour tous, en travaillant avec les petits commerçants qui font vivre nos villes, plutôt qu’en développant, comme à l’heure actuelle, les grandes surfaces commerciales en périphérie.

Enfin, il s’agit aussi d’aider notre jeunesse. Combien de fois avez-vous croisé des jeunes en train de faire du skate sur la voie publique ? Notre majorité actuelle a préféré offrir une réduction sur le tarif des terrains aux promoteurs immobiliers, plutôt que d’investir 80k€ (sur un budget de 16M€) pour donner à nos jeunes des lieux où s’amuser. Que dire de l’état des pistes cyclables ? Pensez-vous que vos enfants peuvent, en toute sécurité, partir du collège et se rendre à leur entraînement de sport à Saint Georges de Montaigu ?

Bref, dans une semaine, le 15 Mars, vous devrez choisir vos élus. Je me présente humblement sur la liste de Vincent MATHIEU, une liste composée de citoyens impliqués, d’horizons et compétences différents et j’espère être bientôt à votre service pour une vie meilleure, Ensemble et Autrement.

Comme l’a si bient dit Michel PAVAGEAU, le doyen de notre liste, à Commune Nouvelle, Équipe Nouvelle !

Chronicles of an introvert remote worker

Fri, 3 Jan 2020 00:00:00 +0100

Note	This post is the translation in English of the most popular blog post I wrote in October, 2019, which was originally posted in french. Lots of folks commented they found it useful and that they recognized themselves in the description so I decided to translate it to English.

The other day I was listening to Les Cast Codeurs podcast which mentioned the topic of remote working.

Some comments made me raise an eyebrow, this post is going to talk about my own experience. This post is mostly about me and being an introvert.

Remote working and being an introvert

I’ve been almost full-time remote worker for almost 10 years. I’m saying almost because in practice, I’m moving from time to time for profesional reasons (conferences, meetings). However, for my daily job, everything is done from home. Before being a remote worker, I was driving a lot: my job was in Nantes, around 45km from home. That was about ~90km a day, but more importantly between 2 and 3 hours of commute a day. The day I switched jobs, working for VMware on the Groovy language, the first shock was that I was recovering those 2 to 3 hours a day. That’s a lot of time to spend with your kids (dropping them at school, driving them to sports, …) or doing housekeeping (fixing, gardening, sports, …). I won’t come back to those advantages because they are mostly known, the only real drawback to me is being capable of handling your own work hours, that is to say, contrary to the belief, not working too much.

The thing I want to talk about is this comment from Antonio Goncalvés who said:

Dump your coffee maker, go out, see people, it’s important to see other human beings!

This made me react so I wanted to come back to this topic.

First, this advice only works if you live in a city, which is not my case. I (intentionally) live in the countryside, 5kms from the town center. A "small coffee break" turns into an expedition in those conditions.

But let’s come to the core of the topic: "It’s important to see and talk to human beings", which relates to the necessity not to be alone. This is a topic I care about and resonates with a question I often answered when I’m talking about remote work:

But aren’t you tired of being alone and not seeing anybody?"

Honest answer: no.

I understand people who need this, but don’t think everybody works like you and need to see other people. Some folks like me live perfectly well without seeing anybody. I can spend days alone, without me going mad or missing any social relationship.

I’m part of this population called the introverts.

An introvert, contrary to the popular belief, isn’t antisocial, selfish or without any kind of empathy.
An introvert is someone whose energy is drained by the crowd, by social relationships. It’s the opposite of an extravert who needs to see people to recover energy.
An introvert is someone who can’t stand smalltalk, because they don’t bring anything to him/her except from draining their energy.
An introver is someone who can talk hours on a topic (s)he’s passionate about because it’s worth their energy.

When I was a kid I already avoided being in a group of people. I didn’t have many friends (but the friends I had back then I still have them today).

I always freaked out at the idea of being in a group where I knew nobody, be it at school, at sport or even family parties. Even if I knew someone, being in a group, having to entertain social relationships was of extreme difficulty (and it still is today, to some extent). Everything I cared as a child about back then was being "excellent" at school and my parents be proud of me. This caused me to get some nicknames ("the nerd"), being hassled or even get some not so nice words written on the back of my classroom photographs, for a large part of my childhood.

As an example, I always refused to go to summer camps (the only experience I got, a week in a stud farm, was so painful to me I still remember it). I also refused going to snow camps. However, I did accept going to England, Ireland and Spain with school, not without fighting myself, because I knew I could learn something out of it, but also because I knew I would be in a small group with a friend.

Similarly, having to stand in front of the rest of the classroom to recite a poem, or worse, sing, was pure torture to me. On the contrary, I loved being at the University of Science in Nantes, where I could spend some time with 2 or 3 friends max at the coffee shop, discussing with them. I didn’t like much the software engineering school I joined to after that, where I had to be extremely creative to avoid the student integration parties, after-work parties and all those driking games most students were fond of: it was incredibly difficult to be recognized as a person without being excluded because I didn’t participate in those events.

Let’s talk about those "integration" parties. What’s fun for some, being drunk, "gently" humiliating others, was just disgust and fear for me.

Now that I think about it, few people know it, but I spent my whole scolarity in Nantes, not because everything was available there (it was, really), but because I was petrified at the idea of beling "alone" in a new University, an engineering school, somewhere else at the other end of France, … I remember a teacher who blamed me for not going in a well known French school despite I had the "capacity" do it. In retrospect I don’t think it was a big mistake, I’m quite happy where I am today!

I would never hire someone like you!

Everything I explained may sound surprising from someone who, today, gives talks at conferences, sometimes in front of hundreds of people, which sounds a bit irrealistic. So what’s the difference?

Well the difference is that when I talk to you in a such a context, I understand the topic (or at least I think I understand it ;)). I can talk to you openly because I know the questions I will get are directly related to the topic I’m talking about. There’s almost no room for the unknown.

That’s also for this reason that it’s still very difficult to me, even today, to reach out to others. I am still completely incapable of doing small talk. This goes even beyond that: I’ve traveled abroad a number of times, multiple times in the US for example, but I never went outside alone to visit (at best I can walk an hour or two in the neighorhood). That’s because doing this drives me outside of my comfort zone: having to talk to people I don’t know, ordering a meal, asking for directions: all this is draining my energy. It’s always easy to say you should go out of your comfort zone when doing this is not a trial for you.

However, if I know one or two people and that they kindly ask me to join them, in most cases it will be fine and I would appreciate visiting! Everybody’s telling me I’m incredibly lucky to travel. Well, maybe, but I, for one, hate this, at least I hate it if I have to do this without my family… Even going to Paris, realizing how many people live there, feeling like an ant in the anthill, remind me how much I hate this!

As a consequence, I avoid as much as I can "parties" like speaker dinners or corporate events. They are often described as the must do of networking. While it’s true that you can learn a lot in those events, they are also incredibly stressful to me. When I join them, most of the time I’m trying to identify someone I know, stick with them but I would often leave early and I have to go to my hotel room alone to recover. I wouldn’t necessarily sleep, as some people think I do: often I would work late in my hotel room, I just need to be alone. The most terrifying thing to me in those parties is when conversations become extra-professional which is something I have no "skill" for. Some would say I’m a boring guy, that there’s no interest in talking to me and that’s probably right if you’re looking for someone with an extraordinary life, mine is not that much :)

A natural conclusion that some people draw from this is that of course I can’t be a good team player. In fact I think that’s quite the opposite: I value trust like no one else and I think you can trust me. One shouldn’t mix my incapacity to have simple inter-personal relationships with my capacity to have useful professional interactions, which is another totally different dimension.

As I explained, I’m regularly giving talks and I love more than anything else sharing knowledge and information, helping others. I also enjoy working with people more competent than me so that I can continue to learn and make progress: in this context working in a team is extremely effective.

Also, it’s not because you’re working remotely that you don’t see anyone and that, effectively, you work alone. It’s quite the opposite: we work as a team. As other teams we do daily standups (via hangouts or Slack), we organize work. Sometimes we even do face to face meetings or leverage conferences to meet together and do some work. So it’s not because you’re an introvert that it’s a problem: I think my colleagues would tell you that it works pretty well actually.

In short, don’t be afraid of this aspect: being alone is not a problem for everyone. Maybe it is for you, but definitely not for everyone. On the opposite, being alone can also boost productivity: no interruption, no useless meetings, ability to focus on a problem for hours etc… Being introvert doesn’t mean either that someone is not capable of working in a team: it doesn’t prevent at all from helping others, colleagues or customers. On the contrary, the professional relationship is focused.

So how does it work in practice? At Gradle, almost everybody is working remotely. For the socializing aspect, we have an optional weekly meeting, called "coffee time", that anyone willing to can join and chat about extra-work topics. Some need this, I don’t: the good thing is that we recognize not everybody’s the same. Actually I would be in difficulty to join this meeting and know what to talk about. Sometimes I even try to avoid being the first one joining a remote call to avoid having to do some small talk because I’m so uncomfortable with this.

However, again, talk to me about a topic I know about or that I’m passionate about. Ask me about a problem to solve, ask me about helping others and I’ll be happy to do so because there’s no room for the unknown.

Similarly if we meet in person at a conference and that you wish to talk to me, go ahead: it’s very likely I will enjoy our conversation: I had several times the opportunity to talk to very famous folks of the Java community (James Gosling, Brian Goetz, Mark Reinholds, …) and so many others but I just couldn’t, it was, physically, impossible to me.

In fact I have so much difficulties coming to someone and opening a discussion that if you do the first step it’s much easier.

So: don’t be surprised if I walk the area 3 times in a row without asking for directions. Don’t be surprised if I don’t phone or text you. I’m not ignoring you.

Don’t be ashamed of being an introvert but make others understand what it means

Last, in a Tweet I was saying that I was "healing myself". That’s both true and false at the same time. It’s true because I’m aware of the importance of the ignorance of a lot of people about this condition of mine, so I try to work, for example, speaking with others in different contexts. I am for example the secretary of my karate club, which forces me to talk to people I don’t know. I am also the coach of my younger boy’s (10 yo) basketball team, which I enjoy very much doing: it’s an activity which allows both spending time with my kid, requires some public communication skills and is very gratifying (seeing kids happy to play, win a match, share this with their parents etc…).

Last but not least it’s also a message for my son who I know feels very much like me: he also has difficulties speaking to others and I know he works hard to get accepted. He’s also so proud when he gets recognition from his community.

It took me years to put a name on what I am: I’m an introvert and often I have to fight myself not to be ashamed of this.

Gradle myth busting: scripting

Sun, 3 Nov 2019 00:00:00 +0100

I don’t particularly enjoy Twitter as a medium for debating (no surprise my bio mentions "this is not a support channel"). This happened again this week, I got caught in a Maven vs Gradle debate, one more, after I replied to Lukas Eder that his tweet was a call for FUD. And it did happen: no surprise, when you submit something like that, the only answers you’ll get are either people going in your direction "oh yeah, Gradle sucks and here is my personal experience" or similar, this is just human nature.

The good old debate

In the end, most of the answers cycle around the same, good old, debate: Gradle uses scripting (Groovy or Kotlin) vs Maven uses declarative. You’ll aways find people telling you that XML is better because it locks you down, its fully declarative (is it, really?) and everybody is forced to do the same. I don’t counter those arguments, this is a strength of Maven, but it also comes with a number of drawbacks.

I promised some of the folks in the conversation some answers (please look down for direct answers to tweets), so here they are. I’m answering on a blog post because again Twitter is not good for this, it’s causing a lot of misunderstandings, because you get into multiple, parallel conversations with different people who accidentally get mentioned, and get scuds fired at you without even having time to answer… Even a blog post is not enough, there’s so much to say on this topic.

First, on the so called "declarative vs imperative" model, I will always disagree on this dichotomy. I disagree that Gradle isn’t declarative. It’s as declarative as you want it to be. Take this build file I wrote recently, which is an Asciidoctor Reveal.js presentation template (it allows writing slide decks with Asciidoctor and reveal.js). Here’s what my build file looks like:

plugins {
    id("org.gradle.presentation.asciidoctor")
}

presentation {
    githubUserName.set("melix")
}

dependencies {
    asciidoctor("org.asciidoctor:asciidoctorj-diagram:1.5.11")
}

tasks {
    asciidoctor {
        requires("asciidoctor-diagram")
    }

}

I wouldn’t particularly say this is imperative. It looks very declarative to me. Concise too. The fact it uses an imperative language is orthogonal, but it does, however, create the ability to write imperative code in the build. Note, however, that a dependency was declared for asciidoctor. This is a major, and probably the most important, difference with Maven: compile or runtime doesn’t make sense here. We declare a dependency for asciidoctor rendering. There’s no Java library being built here, it’s a presentation. Gradle lets you model precisely what you build.

So, in the end, I think what matters is not declarative vs scripting. I think what people really want is to reduce the risks of writing bad things. Locking down using XML is one way to achieve this, but it’s not the only one. For example, Gradle build scripts may be linted. In other words, you can apply on a Gradle build the same tooling you are used to work with when dealing with your own code: checkstyle, findbugs, … You don’t have to, but you can.

The consequence is that yes, there are many different ways you can layout your build with Gradle. This is not different from how you can layout your code in a project: we don’t tell you in which package you should put your beans, services, … However, there’s a big misconception that I’d like to fight:

By default, for a Java project, Gradle follows the same conventions as Maven.

That’s as simple as that. Sources will be in src/main/java. Tests will be in src/test/java. Gradle gives you the freedom to diverge from this convention, but this is not encouraged, and to be honest, I’ve almost never seen any build diverging from those conventions. On rare occasions, those were actually builds migrated from other build systems (in particular Ant) where at the time there wasn’t any convention. Gradle offers the flexibility to reuse an existing layout without much hassle.

Gradle is too flexible?

All in all, the argument that "Gradle is too flexible" is a fallacy. It’s all about good engineering practices, putting the right tools in place, and this is nothing different from any engineering work we do everyday. If you can do it for your code, you can do it for your build. The interesting thing is why you think you shouldn’t have the same quality expectation levels for your build as you have for your code. Often the answer is just "I don’t care much about the build, I’m writing code, this is what I’m paid for".

And this is where the discussion becomes interesting, because I think this is a bias that lots of developers have. They don’t even realize how much time they are losing.

Tell me, how likely is it that you change your build scripts, compared to the number of times you’re effectively going to run mvn clean install or gradle test? The reality is that you’re running the build much more often that you change it. Therefore, correctness, incremental builds, incremental compilation, compile avoidance, task output caching, are far more important to developer productivity than the declarativeness aspect.

Sure pure declarativeness is a good thing and this is why I encourage Gradle users to write nice, synthetic build files, but this is not the most important aspect for developer productivity.

My point is therefore that if you only focus on the surface, that is to say the language used to express the build (XML vs Groovy/Kotlin), then you’re missing the most important part to me, which is the underlying Gradle model, far more advanced than what you have in other tools. The Gradle API surfaces this model and has a number of advantages:

A task can be seen as a function. It declares inputs and outputs. For the same inputs, the output are always the same: this provides up-to-date checking and cache-ability.
A task inputs can be another task outputs. This provides implicit dependencies: Gradle knows that if you want to run "test", you have to compile first, but it also knows that whatever else is an input to the tests need to be executed.

As a consequence, I already wrote about why it’s wrong to think that Gradle doesn’t have lifecycle tasks. In fact, Gradle has them, but is also significantly more precise. The "phase" approach of Maven is way to coarse: it’s doomed to execute too much, prevents smart parallelism, and leads to dirty workarounds (-x …. on the CLI to avoid things you know are not necessary).

Similarly, say you want to test your application on different JDKs and have a single build execute tests for all target JVMs, which is different from the JVM which runs the build tool. With a scripting approach like Gradle, this is totally doable. I won’t say easy because we can definitely do better to make this use case better, but the underlying model makes it quite simple. You don’t want to rebuild your application for each target VM. All you want is to test on different platforms, and therefore the only step should be a different target VM for test execution. Tools like Maven force you into arbitrary things like defining Maven profiles, and force you into rebuilding everything. This is a giant waste of time for something you don’t need!

In a different topic, this is no surprise that Gradle can build for different ecosystems: Java, Scala, C++, Kotlin, Kotlin Native, Python, … The underlying infrastructure makes it possible. Even for a single ecosystem, Gradle can declare what the difference between a Java Library, a Java Platform or an application is.

I could talk hours about why it’s important to model properly software, and actually with the release of Gradle 6 we’ll have a series of blog posts explaining why we think it’s a game changer in terms of dependency management. If you’re tired of having to fix the same "multiple slf4j bindings" in each and every project, tired of Guava being upgraded from jre to android, frustrated by incompatibilities of Scala 2.11 and 2.12 dependencies, tired of not knowing which of those Maven optional dependencies is important for you to add, you’ll understand what I mean.

Direct answer to some tweets

I’ll try to answer more direct questions in this section. Sorry if I missed yours, I got quite a few comments/answers…

I would sacrifice caches, dependency locks and better plugins to version to have a declarative build process instead of an imperative one. Give me a declarative Gradle and I will love it.

Again, I think this is the wrong tradeoff. Given that you run the build way more often that you change it, declarativeness (that you can have with Gradle) shouldn’t be the goal. Your goal should be to reduce your build times, make your build reproducible, improve your developer productivity. Declarativeness is not a goal, it’s at best a mean, but not sufficient by itself. A declarative Gradle, whatever that means, would help you reduce the cognitive overhead, but wouldn’t help you better model what your application needs.

make a one liner the ability to publish on different repos the snapshot and release artifacts. The way it was done on Gradle 4.x was broken on 5.x and the only way we found to do it is a horrible hack

Here’s a webinar about publishing. Publishing is not complicated with Gradle. It used to be poorly documented, and the old publishing plugins didn’t help. But publishing to a snapshot repository should be trivial already.

Some people will prefer to do their own way, some people will prefer to have a less expressive tool that will produce similar build processes on their projects. Gradle give you the former, Maven the latter. As I say, a matter of taste.

A less expressive tool reduces the risks of writing bad builds. It doesn’t help, however, in developing correct, reproducible, fast builds. An, again, I disagree that Gradle leads to "custom builds" everywhere. Most people stick to the defaults and are very happy with them. The more complex builds you find in the wild are those which have indeed very specific needs, or need to be tweaked for performance, producing more artifacts, combinations of artifacts or testing. Things that you can’t easily do with Maven profiles, for example, because profiles are adhoc solutions which do not combine well.

Gradle tries to create a fake sense of declarativeness, but it is just an illusion.

It’s not an illusion. Gradle has a clear separation between its configuration model and execution model. All tasks have declared inputs. The plugins create either new tasks or conventions. This is not an illusion, this is the reality. Now, because you can write if or loops doesn’t mean it’s not declarative, it’s imperative-declarativeness. And yes, you can end up with giant build scripts with "code" inside. If you have, do yourself a favor, refactor your build like you would with your code, because no one should tolerate this. Use buildSrc, this is your friend.

I think library dependencies is not correctly supported by IDEs and Java modules are better.

That’s not correct. We’ve been using the native Gradle IntelliJ support for years at Gradle, with api and implementation separation, and it works exactly like it should. Implementation dependencies are hidden from consumers, like they should. If you don’t see this, either you didn’t declare the dependencies or you have a bug in the IDE, in which case it needs to be reported.

Even worst, Gradle don’t have an official plugin to deal with module-info. there was an old post that says it is not necessay with Gradle because lib dependencies were better (they don’t)

I don’t think anyone said you don’t need module-info. There are different things in play:

separation of API and implementation: Gradle supports this, and it maps to requires vs requires transitive
declaration of public API packages: Gradle used to have this with the deprecated "software model". It still has to be backported to the current configuration model. For this, module-info works fine but it forces you into using the "modular world", which a lot of libraries, frameworks and IDEs are not ready for.
declaration of services: Gradle doesn’t support this.

Can you use modules with Gradle? Yes, there’s a quite good plugin to do it. We are planning to support modules and modularity in general better in Gradle, but not short term, because we have bigger pain to solve for our users first. It doesn’t mean we don’t consider this important.

I don’t get why Gradle allow you to explain what your app is better than Maven. In fact I think it is more difficult to explain it on a script that descriptivelly.

I think the question is what Gradle models better than Maven. A good example here is api vs implementation dependencies. Because Maven uses the same descriptor (pom.xml) for the producer and the consumer, a dependency declared in the <compile> scope ends up on the compile classpath of the consumers. This is leaking internal implementation details to consumers, which is very bad because it makes it very hard to evolve libraries, because changing an internal dependency would break consumers which accidentally started depending on your own transitives. This is just an example of course, there are many other differences (like, why we consider that exclude is a bad workaround in general, more on this topic in Gradle 6, if you want to read our docs).

The builds I’ve seen have been very spaghettish and clearly copy-and-pasted together un-understood recipes from SO.

Yes, there are bad builds out there. With Gradle it’s frequent for quite old builds from early adopters. More recent builds tend to be much cleaner, because we made a significant effort in guides, getting started samples, documentation. You’ll always find bad things, and it should be encouraged to fix. On this topic, tools like build scans really help. And copy/pasting from SO is indeed a bad thing. If you copy and paste without understanding what it does, well, bad things can happen… That said I’ve seen very scary Maven builds too, and believe me or not, some of our customers wouldn’t be proud to show you their Maven builds. It’s the "personal experience fallacy". I’ve experienced very clean Gradle builds, you’ve experienced very bad Gradle builds. I’ve also written bad Gradle builds, which I dramatically improved, making them more correct, faster, … Gradle is like any other technology: learn it and you can understand what it brings.

Gradle performance/caching are very attractive but the scripting possibility is a deal breaker. A « declarative-only » Gradle would be perfect for people like me.

Again I think "declarative" is the wrong term here. Locked down to reduce the risks of doing bad things is what you want. It doesn’t matter if it’s Kotlin, Groovy, XML or whatever else. It doesn’t matter if you can use if or for loops. What matters is what you can express, and what should be limited. It’s all good engineering that we must share within the industry, find the best patterns, discourage the bad ones. There are quite a few things in the Android world (which uses Gradle) in this direction. We, at Gradle, should do more, but it’s always a matter of priorities: fixing the most important user pain first. By the way, we provide a Maven build cache with Gradle Enterprise. That is to say, the ability to cache Maven builds using Gradle Enterprise. However, this is limited to "clean builds" (which Maven users are used to do in any case), because of the limitations of the Maven execution model (no knowledge of what each plugin or mojo does, where it writes files, …).

IMHO the biggest feature of Gradle that Maven doesn’t have is the ability to change the version of the project

Well, this is just an accidental example of the interest of having access to the API in a build script. It offers a number of options for the release process, but that’s not the only one.

My only complain about @gradle is how it is unnecessarily complex to deploy a multi-module project to central. Too much copy & paste, or you need to make an init script, which I still haven’t managed to do.

Technically the problem is not "how to deploy a multi-module project to Central", but rather, how do I avoid duplicating configuration between scripts. This is what buildSrc is for. As soon as you have repetition, then, it means a plugin makes sense. buildSrc can be seen as "local plugins", and this is where you should write your common code. Then each project applies a plugin to publish. This is a composition model, as opposed to the inheritance model of Maven.

I like all those (caching, incrementality, …) , in theory, but for my needs they are more complexity than feature

I don’t think those are complexity. A task declares its inputs. If you do, you benefit from up-to-date checking, and with a bit more configuration, caching. You don’t have to. If you don’t declare the inputs/outputs, you’re back to the "Maven" approach where the build tools knows nothing about what a task does, at the difference that Gradle knows that it knows nothing, so can be a bit smarter. As soon as you start declaring your inputs, you benefit from more. It’s more work, for sure, but it’s not that complex and the benefit is huge.

Chroniques d’un introverti en télétravail

Sun, 29 Sep 2019 00:00:00 +0200

L’autre jour j’écoutais le podcast Les Cast Codeurs qui abordait le sujet du télétravail. Quelques remarques m’ont fait tiquer, je m’en vais donc ici vous raconter mon expérience, mais surtout vous parler de moi et de l’introversion.

Le télétravail et l’introversion

Voici maintenant presque 10 ans que je suis presque à 100% en télétravail. Je dis presque, parce qu’en pratique, j’ai quelques déplacements professionnels (conférences, meetings) de temps en temps, mais pour mon boulot de tous les jours, c’est tout à la maison. Avant de faire du télétravail, je roulais beaucoup: mon emploi était sur Nantes, à 45km de chez moi. Ca faisait donc 90km par jour, mais surtout entre 2 et 3h de transport par jour (eh oui, j’avais la chance de devoir traverser le pont de Cheviré, pour ceux qui connaissent…). Bref, le jour où j’ai changé d’emploi, pour passer chez VMware pour travailler sur le langage Groovy, le premier choc pour moi ça a été de récupérer ces 2 ou 3 heures par jour. C’est autant de temps qu’on peut passer à s’occuper de ses enfants (les déposer à l’école, les emmener au sport, …) ou à faire des activités à la maison (jardinage, sport, …). Je ne reviendrais pas sur ces aspects, parce qu’ils sont largement connus, le seul "inconvénient" pour moi étant surtout d’être capable de maîtriser ses horaires, c’est à dire de ne pas, contrairement à ce qu’on pense, faire trop d’heures !

Non, ce sur quoi je veux revenir, c’est cette remarque d’Antonio Goncalvés, qui, en conseil, dit en substance:

"Balancez votre cafetière, sortez dans les bars, voyez des gens, c’est important de voir des êtres humains!"

Et ceci m’a fait réagir, j’aimerais donc revenir sur le sujet. Tout d’abord, revenons sur le fait que ce conseil ne fonctionne que si vous habitez en ville, ce qui n’est pas mon cas. Je suis en campagne, à 5km du bourg, donc faire une "pause café", ça prend du temps dans ces conditions.

Mais revenons surtout sur le fond du sujet: "C’est important de voir des êtres humains", ou ce besoin de ne pas être seul. C’est un sujet qui me tient à coeur et qui résonne avec une des questions que j’ai souvent lorsque je parle du télétravail:

"Mais tu n’en as pas marre d’être tout seul, de ne voir personne ?"

Alors, répondons franchement: non.

Autant je comprends les personnes qui en ont besoin, autant, ne croyez pas que tout le monde fonctionne comme vous et a besoin de voir du monde. Certaines personnes, comme moi, vivent très bien sans voir personne. Je peux passer des journées entières seul, sans que celà ne me pose le moindre problème, ni ne me cause le quelconque manque de relations humaines.

Je vais partie de cette catégorie de population qu’on appelle les introvertis.

Un introverti, contrairement à une croyance populaire, ça n’est pas un associal, un égoïste ou quelqu’un sans empathie.

Un introverti, c’est quelqu’un dont l’énergie sera drainée par la foule, par les relations inter-personnelles, c’est l’opposé d’un extraverti qui a besoin de monde pour regagner de l’énergie.

Un introverti, c’est quelqu’un qui ne supportera pas les conversations de comptoir, parce qu’elles ne lui apportent rien d’autre qu’une perte d’énergie.

Un introverti, c’est quelqu’un qui pourra vous tenir une conversation passionnée sur un sujet qui l’intéresse, parce qu’il en retire quelque chose: l’interaction en vaut la chandelle.

Déja tout petit, j’évitais les groupes, je n’avais pas beaucoup d’amis (mais ceux que j’avais à l’époque, je les ai toujours). J’ai toujours eu une peur bleue de me retrouver dans un groupe où je ne connaissais personne, que ce soit à l’école, au sport, … Mais même si je connaissais quelqu’un, se retrouver dans un groupe, à devoir cultiver des relations proches, extra "professionnelles", était d’une extrême complexité pour moi (et ça l’est toujours). Tout ce qui m’importait à l’époque, c’était d’apprendre, être "fort" à l’école, ce qui m’a valu d’être traité "d’intello", parfois chahuté, ou d’avoir des remarques pas particulièrement gentilles d’autres élèves au dos de ma photo de classe, ceci pendant une grande partie de mon enfance…

Par exemple, j’ai toujours refusé les camps de vacances (ma seule expérience, une semaine à faire du cheval, fût tellement une horreur pour moi que je m’en souviens encore). J’ai aussi refusé toutes les classes de neige. En revanche, j’avais accepté d’aller en voyage scolaire en Angleterre, en Irlande, en Espagne, non sans lutter contre moi-même, parce que je savais que je pouvais en tirer quelque chose, mais aussi parce que j’avais la garantie de me retrouver en petit nombre, avec un copain, dans une chambre…

Dans la même veine, devoir se retrouver au tableau pour réciter une poésie, ou, pire, chanter, tenait de la torture. J’ai adoré la fac' de sciences, parce que je pouvais passer du temps au "café" avec 2/3 potes maximum, à discuter. J’ai détesté mon école d’ingénieur, où j’ai du rivaliser d’astuces pour éviter les soirées d’intégration, tonus et autres dont étaient friants 95% des étudiants: se faire reconnaître sans se faire exclure (mais j’ai encore trouvé quelques bons amis que je tiens de cette époque).

Aaah, les soirées d’intégration: ce qui est marrant pour certains, se bourrer la gu…, humilier "gentilement" les autres, c’était juste pour moi quelque chose d’horrible. Tiens, d’ailleurs, peu le savent, mais j’ai fait toute ma scolarité à Nantes, non pas parce que tout était disponible là bas (c’était le case), mais parce que j’était terrifié à l’idée de me retrouver seul dans une FAC, une école d’ingénieur, à l’autre bout de la France… Certains profs m’avaient reproché de ne pas tenter une "Grande Ecole", mais je pense qu’au final, je m’en suis plutôt bien sorti !

Jamais je n’embaucherais quelqu’un comme vous!

Ceci pourrait vous paraitre surprenant pour quelqu’un qui, aujourd’hui, donne des conférences devant, parfois, des centaines de personnes. Mais alors, quelle est la différence ?

La différence, c’est que lorsque je vous parle dans un cadre professionnelle, je maîtrise mon sujet (ou je pense le maîtriser, ah ah !). Je peux vous parler ouvertement, parce que je sais que les questions qu’on va me poser seront sur ce sujet. Il n’y a que peu de place à l’inconnu.

C’est aussi pour celà qu’il m’est très difficile, encore aujourd’hui, d’aborder des gens que je ne connais pas. Je ne sais pas faire de "small talk", comme on dit. Mais ça va plus loin: j’ai fais pas mal de déplacements à l’étranger, aux US par exemple, mais JAMAIS, je ne suis sorti seul pour visiter (au mieux je marcherai une heure ou deux dans les alentours). On rentre dans cette zone d’inconnu qui me met tellement mal à l’aise : devoir s’adresser à des gens, qui plus est à l’étranger, pour commander un plat, pour demander son chemin: celà me demande trop d’efforts, me draine trop. En revanche, si je connais une ou deux personnes et qu’elle me proposent de les accompagner, ça se passera bien: j’apprécie fortement une sortie en "petit comité". Tout le monde me dit que j’ai tellement de chance de partir voyager, voir du pays, mais moi, je déteste tellement voyager, en tout cas sans ma famille… Rien qu’aller à Paris, me rendre compte du nombre de personnes qui vivent là bas, me sentir étouffer, qu’est-ce que j’ai horreur de ça !

La conséquence, c’est que je fuis comme la peste les "soirées" comme les speaker dinner, les événements d’entreprise, … Souvent, on les présente comme des incontournables du "réseautage", et c’est vrai qu’on y apprend souvent des choses intéressantes, mais elles sont extrêmement stressantes pour moi. La plupart du temps, j’essaie de trouver une tête connue et je "m’accroche", mais je ne fais jamais long feu. Ce qui me terrorise s’est d’en arriver à des sujets extra-professionnels, là où je n’ai AUCUNE "compétence". Certains diront que je suis chiant, que je n’ai aucun intérêt, et c’est probablement vrai si vous recherchez quelqu’un à la vie extraordinaire, la mienne n’est pas ultra passionnante :)

Si vous en arrivez là, vous devez vous dire que je ne dois pas être un très bon "team player", comme on dit. Paradoxalement je ne pense pas que ce soit le cas. En effet, il ne faut pas confondre mon incapacité à avoir des relations inter-personnelles "simples" avec la capacité d’échange, d’interaction professionnelle, qui est sur une toute autre dimension.

Comme je vous l’ai expliqué, je donne régulièrement des talks, et j’adore plus que tout partager ces connaissances, aider les autres. J’aime aussi travailler avec des gens plus compétents que moi pour avoir l’autre face de la pièce: apprendre, comprendre de nouvelles choses, progresser. Et dans ce contexte, travailler en équipe aide énormément.

Alors, ce n’est pas parce qu’on est en télétravail et qu’on ne voit personne qu’on travaille seul. Non, on est en équipe. On fait des "daily standup" (via Hangout/Slack), on s’organise, on fait des petites réunions "face to face" de temps en temps, bref, je pense, mes collègues vous le diront si ça n’est pas le cas, que ça se passe plutôt bien.

Bref, n’ayez pas peur de cet aspect: la solitude n’est pas, pour tout le monde, un problème. Elle peut, au contraire, aider à la productivité: pas de réunionite inutile, capacité à se focaliser sur un problème des heures durant, … Elle n’est pas, non plus, signe de quelqu’un qui ne peut pas travailler en équipe: ça n’empêche en RIEN d’aider ses collègues, ses clients, bien au contraire: puisqu’on sait que la relation est professionnelle, elle est focalisée et bien plus efficace.

Pour vous donner une idée, chez Gradle, tout le monde est en télétravail. Mais pour cet aspect "socialisation", nous avons une réunion hebdomadaire, facultative, appelée "coffee time" où chacun peut rejoindre la réunion et parler de sujets extra-professionnels. Certains en ont besoin, pas moi. Au contraire, j’aurais du mal à arriver à cette réunion et savoir de quoi parler. Pire, j’évite au maximum d’arriver en premier à une réunion de peur de ne pas savoir engager une conversation…

Mais encore une fois: parlez moi d’un sujet que je maîtrise, demandez moi d’expliquer un problème, de chercher une solution, d’aider quelqu’un: il n’y aura aucun problème, parce qu’il n’y a aucune place à l’inconnu. Il est même fort probable que j’apprécie énormément une conversation entre nous: si je ne viens pas vous voir, ça n’est pas que je ne souhaite pas vous parler, c’est que même si je vous connais, engager une conversation reste quelque chose de très compliqué pour moi. Venez me voir, engagez la conversation et tout se passera bien ! :D

Alors, ne vous étonnez pas si je vais trois fois le tour du quartier plutôt que de demander mon chemin. Ne vous étonnez pas si je ne téléphone jamais, ne texte jamais.

Mais si vous faites partie de mes amis, vous avez énormément de chance: c’est que je tiens beaucoup à vous et vous tiens en haute estime !

N’ayez pas honte d’être introvertis, faites comprendre ce que c’est

Je terminerais pas une remarque: dans mon Tweet original, je disais que je "me soignais". C’est vrai et c’est faux à la fois. Quand je dis que c’est vrai, c’est parce que conscient de l’importance et de la "non reconnaissance" de cette difficulté par les autres, j’essaie de travailler, notamment la prise de parole. Je suis, par exemple, secrétaire du club de Karaté, ce qui me force à communiquer avec des gens que je ne connais pas. Je suis aussi le coach de l’équipe de basket de mon fils de 10 ans, ce que j’adore faire: c’est une activité avec son enfant, qui est à la fois publique, mais aussi tellement gratifiante, voir ces enfants heureux de jouer, gagner, … et ça me permet de travailler sur les relations avec les parents !

Et surtout, c’est aussi un message pour mon fils, qui a 10 ans donc mais me ressemble tellement: il a lui aussi beaucoup de mal à parler aux gens, mais je qu’il travaille dur pour se faire accepter. Lui aussi est si fier lorsqu’il a la reconnaissance de ses pairs… Il m’a fallu des années avant de mettre un nom à ce que je suis, un introverti, et aujourd’hui encore, je lutte pour ne plus en avoir honte.

Dependency Management at Gradle

Thu, 11 Jul 2019 00:00:00 +0200

Opinions are my own, not those of my employer

I joined Gradle 4 years ago, and I spent the last 2 years almost exclusively working on dependency management. The more I work on this topic, the more I am frustrated by the state of our industry. In this blog post I will mostly illustrate my points with Java, which I’m mostly familiar with, but a lot of them stand true whatever the ecosystem (Javascript, Python, Go, …).

Soon, we should release Gradle 6 (fall 2019, no promise, as always, this is software). This version will be the first one to turn the spotlights on dependency management. This will be for me the achievement of 2 hard years of work (not alone, of course) but this is just the beginning. Gradle probably has the most advanced dependency engine of the market (forgive me if I’m wrong, let me know). But what makes it so different is not that it has powerful features like dependency substitution, customizable resolution, dependency locking, alignment, etc… No, what makes it different it’s that we put a lot of emphasis on semantics: model software correctly so that you, as a developer, have less to handle.

Often, folks "from the outside" are considering us arrogant, because we don’t immediately support things that other engines do (eg. "you don’t want to implement optional dependencies, look, Maven has it for years you s***"). While I understand the frustration, it doesn’t mean that we don’t hear. In fact, we hear you loud, but the time it takes to land is just the consequence of willing to implement the correct solution to a problem.

For example, what does an optional dependency mean? A few months ago, I read this from the Yarn documentation which got me scared, and, to be honest, a bit despaired too:

This looks like total non-sense to me. Because we needed the infrastructure, it took some time but we implemented what we think is a better solution. In a nutshell, optional dependencies are not optional. They are things that you need if you use a particular feature of a library. So it means that if you depend on something, and that this something has optional features, you can say what features you need and Gradle would give you the correct dependencies for this. We called that feature variants. Granted, the name isn’t cool, but it’s a matter of time until we refactor the documentation to make it easier to discover.

Similarly, we implemented back in Gradle 3.4 the separation of API and implementation dependencies for the Java world. In fact, that’s one of the reasons the compile configuration has been "soft deprecated" for years now, and that we’re going to nag you starting from Gradle 6. I still hear people claiming they don’t need this and that it’s hard to understand but I stand by the fact they deperately need them.

Why is it important? Because if you are a library author, you know that there are things that are "your internal sauce": things which are implementation details, not part of the public API, and that should never be exposed to consumers. This is the very reason Java 9 shipped with the module system: strong encapsulation. However, how does it work for dependencies? There are also dependencies which are part of your API and others which are not. Say you use Guave internally. None of the Guava classes are visible on your public API, and it would be an error to do so. Then it means that Guava is an implementation dependency of your library. When you do this, you should be allowed to replace Guava with something else at any time, without breaking any of your consumers. The horrible reality is that if you use the Maven <compile> scope, those dependencies are going to leak to the consumers, and they could accidentally start using Guava just because it’s available on the classpath (yeah, IDEs love this, the class is available for completion you know!).

This is a problem with Maven because the POM file is used for 2 distinct purposes:

representing the point of view of the producer, where <compile> means "I need this on my compile classpath"
representing the point of view of the consumer, where <compile> semantics are there broken: it means "add this to your compile classpath", but it means you leak dependencies which should have been on the <runtime> scope!

With the Gradle Java Library plugin, those issues go away: we model those correctly, and when we generate a POM file, it has dependency declarations which make sense for the consumer. I’ve been battling to explain this in conferences for years, yet, lots of people don’t get how harmful to the whole ecosystem that design decision from Maven was: it just prevents smooth dependency upgrades, and contributes to the "classpath explosion".

In other words, you, as a developer, should always know what you directly use (your first level dependencies).

As another example, we’ve been yelling for years how bad exclusions are, but we did a really bad job at explaining why. The problem is also that we didn’t have all the tools we needed to workaround them, so we said "yeah it’s bad" but didn’t really say that you can still use them in some cases.

In a nutshell, an exclusion is bad because the dependency resolution engine doesn’t know why you excluded a dependency:

is it because the producer had bad metadata, that the dependency should have been "optional", or just absent altogether?
is it because you don’t use a specific code path of a dependency and you just want to slim down the size of your distribution?
is it because something breaks if you use this dependency?
is it because it’s yet another logger on the classpath and this is not the one you use?

Well, again, for all those use cases, the solutions are different. The logging use case is an illustration of this: how many of you have faced this infamous problem of having multiple SLF4J bindings on the classpath? How do you fix that with Maven? Exclude of course!

But wouldn’t it better if you could express that "between all those logger bindings, this is the one I choose". Wouldn’t it better if, as a producer of a logger binding, you could say "I implement this logger API, and it doesn’t make sense to have multiple logger bindings on the classpath at the same time". Well, this is what Gradle offers you, as a producer and as a consumer.

Modeling software is important, because it makes the industry better as a whole. By explaining why instead of how, the dependency management engine can take better decisions. It can fail if things are incompatible, it can ask you to choose between different implementations.

Do you need another example? Look at all those "Maven classifiers". How many times did you got multiple conflicting implementations of the same thing just because they had different classifiers (looking at you, Guava!).

The reality is that a classifier is a workaround for a bad model, incapable of expressing that you have different things published at the same coordinates. Sometimes I read comments like "Gradle is just Maven with a Groovy DSL". This is wrong at so many levels: Gradle empathizes on strong modelling of software. We need to think in terms of components ("this is a library", "this is a Boot application", "this is a Gradle plugin"), not in terms of conventions. Models are orthogonal to conventions: conventions are just a tool to make modelling easier to implement, but what matters is the model. The DSL also doesn’t matter: Groovy, Kotlin, whatever. What makes Gradle powerful is that it understands what you build.

This is one of the reasons we came with Gradle Module Metadata, which is going to be enabled by default in Gradle 6.

We, at Gradle, think that we can save a lot of developer time by simply putting more sense in those things we publish. It’s a waste of time and energy that we all have to fix, constantly, those conflicts between libraries.

And can we talk about ethics? It’s beyond me we can tolerate, as developers/engineers, repeated approximations, errors, just because Maven does it this way? I don’t really care that we’re not compatible with Maven, as long as we solve the problem and that we move towards well designed, better solutions. Sometimes we have to make tradeoffs, but we can’t compromise on reproducibility, correctness and performance. Of course, we do our best to provide Maven compatibility, and again, sometimes make it even better in some situations (the Java Library plugin).

My goal is that we, as a whole, move towards a better software delivery world.

The good thing is that what we’re not alone. This, is the result of years of experience from the team with very large builds, from customers of small to very large organizations, and discussions with talented open-source developers (special thanks from me to Andy Wilkinson from the Spring team).

I posted this earlier this month on Twitter, this is a real dependency graph, from an organization I have the opportunity to work with:

This is not uncommon, this is the reality we live in.

And when you have such a dependency graph, you desperately need more modelling. Does it mean that Gradle is perfect? Hell, no. We’re going to make mistakes, we already have and we will continue, but what matters is that we’re moving towards our goal.

There are lots of good things coming in Gradle 6, stay tuned. In the meantime, a lot of what I discussed here has been presented in a couple of webinars:

Another webinar is planned after summer, around multi-repository development.

Hope you’ll enjoy!

Goodbye, Groovy!

Thu, 21 Mar 2019 00:00:00 +0100

I’m stepping down from the Apache Groovy project

Today is a very sad day. I’ve decided to resign from the Apache Groovy PMC, as well as a committer.

The trigger was that I dared pushing a Gradle Kotlin DSL and this had to be reverted. Lots of people in the Groovy community see Kotlin as a threat. I don’t. I use both languages everyday, for different purposes. I wrote most of the static compiler for Groovy, and I have no problem saying that I believe Kotlin is better as a static language: the experience is more consistent, and very pleasant. But nothing in Kotlin reaches the simplicity that Groovy offers. Running a single line script, the awesome Spock framework for testing, …

The fact I work for Gradle Inc was also mentioned as a reason why I did push this file. To me, this is questioning my integrity, which is something I cannot accept.

In practice, since Gradle announced a couple of years back that they would start using Kotlin as a language for its DSL, I’ve been in a very complicated position. Privately, I always get pinged whenever I say something good about Kotlin. Like, you should not do this, this is not good for us, or, Gradle is not nice with Groovy, what are you doing. People always make jokes about Kotlin when they talk to me. I was at Gr8Conf Europe a few days after the announcement and man, that wasn’t a pleasant experience.

I am Cédric. I am not Gradle Inc.

I am Cédric. I am not Kotlin.

I am Cédric. I am not Groovy.

Technologies live and die, I’m not interested in being married with a technology, I’m interested in working with something that excites me. The reality is that I’ve been more excited with Kotlin than Groovy lately, and the attacks, conscious or not, public or private, that I get everytime I mention this language are tiring.

Now, when I’m going to say something good about Groovy, I hope folks will believe me.

Special thanks

I want to thank Guillaume Laforge for this adventure. Man, you gave me the opportunity to join the project, its awesome community. You gave me a job, I was lucky enough to be paid to work on open-source software, and that’s all because of you. For that I owe you everything, thank you!

Second, thanks to Paul King. You are the one that the Groovy community should never loose. People have mentioned that I would be a great loss, but that’s not true: the reality is that I contributed almost nothing recently, and you did all the hard work. You are also an amazing person always trying to get the best of us. I will miss it.

Then Jochen Theodorou: thanks to you for all the hard work you’ve done with the compiler. I would not have been able to write the static compiler without your help and insights on the internals of Groovy.

I’d also would like to personally thank folks that make the community vibrant: Søren Berg Glasius, Graeme Rocher, Jeff Scott Brown, Iván López, Álvaro Sánchez and all the others I miss, don’t take it personally, this is not easy for me.

Last but not least, thank you to the Groovy community. You’ve been awesome, and I wouldn’t be where I am without you. I may have been a bit sharp saying that the Groovy community as a whole sees me as a Trojan horse, that’s not true, but the truth is that I don’t want to fight anymore. Do not assume that because I quit the community is toxic: it’s not. There’s just a lot of understandable fear for the future, in my opinion. My advice is, embrace the change, don’t look back, and take inspiration.

I want to say good luck to the team, in particular to its newer members, like Daniel Sun, who are the future of this language: they have the energy, the skills to make it better.

Last but not least, I’m still a friend of Apache Groovy, you should be too!

A simple native HTTP server with GraalVM

Tue, 19 Mar 2019 00:00:00 +0100

Writing a simple HTTP server with GraalVM

In my daily work, I often need to start a simple HTTP server to serve local files. For example, this week I’m going to give a talk at Breizcamp, and because my presentation uses a Reveal.js slide deck and that it loads resources dynamically, I need a "real" web server to serve the files. So far, I’ve been quite happy using the Python simple http server. Using it is as easy as running:

python -m SimpleHTTPServer 8000

But knowing that the JDK has an embedded HTTP server, and that there’s a lot of hype around Graal those days, I wanted to see if we could achieve the same thing, with a fast startup, with GraalVM. The answer is yes, but the road wasn’t so easy, at least for Groovy.

Show me the code

The code for this experiment can be found on GitHub. We’re going to use:

Gradle to build
the Gradle Graal plugin from Palantir

And because I like the Groovy, and especially its static compiler, my first attempt was to use statically compiled Groovy to do this. Well, it turned out to become a nightmare, so after an hour trying to make it work, I switched to Kotlin, and try to make it work there first. Knowing that Kotlin is a statically compiled language from the ground up and that it doesn’t have the whole dynamic history of Groovy, I did expect it to be simpler.

So, in the end, here’s what the Kotlin server looks like:

HttpServer.kt

fun main(args: Array<String>) {
    val port = if (args.size > 0) args[0].toInt() else 8080
    val baseDir = if (args.size > 1) File(args[1]).canonicalFile else File(".").canonicalFile

    create(InetSocketAddress(port), 0).run {
        createContext("/") { exchange ->
            exchange.run {
                val file = File(baseDir, requestURI.path).canonicalFile
                if (!file.path.startsWith(baseDir.path)) {
                    sendResponse(403, "403 (Forbidden)\n")
                } else if (file.isDirectory) {
                    val base = if (file == baseDir) "" else requestURI.path
                    sendResponse(200, "<html><body>" +
                            file.list()
                                .map { "<ul><a href=\"$base/${it}\">${it}</a></ul>" }
                                .joinToString("\n") + "</body></html>")

                } else if (!file.isFile) {
                    sendResponse(404, "404 (Not Found)\n")
                } else {
                    sendResponse(200) {
                        FileInputStream(file).use {
                            it.copyTo(this)
                        }
                    }
                }
            }
        }
        executor = null
        println("Listening at https://localhost:$port/")
        start()
    }
}

It’s quite simple indeed, and making this work as a GraalVM native image is extremely easy too. This is the whole build file, this is all you need:

build.gradle.kts

plugins {
   kotlin("jvm") version "1.3.21"
   id("com.palantir.graal") version "0.3.0-6-g0b828af"
}

repositories {
   jcenter()
}

dependencies {
   implementation(kotlin("stdlib"))
}

graal {
   graalVersion("1.0.0-rc14")
   mainClass("HttpServerKt")
   outputName("httpserv-kt")
   option("--enable-http")
}

As you can see, we just apply the Kotlin plugin to build our code, then the GraalVM plugin and configure the basics of the GraalVM plugin (version, main class, …).

Building the image can be done by calling:

./gradlew http-kotlin:nativeImage

As you can see, building the whole thing takes around 15s on my laptop. That is to say, compiling the server and generating the native image. Then you can try to serve files running:

http-kotlin/build/graal/httpserv-kt 9090 /path/to/files

You’ll see that the server starts immediately: there’s absolutely no wait time, it’s there and ready to answer. The whole process took me less than 30 minutes, the native image is only 11MB. Success!

Making it work with Groovy

Now that I had a proof-of-concept with Kotlin, I went back to Groovy. And, I can say, despite the fact I love this language, that it was a nightmare to make it work. At some point, I even thought of abandoning, however, using perseverance, I managed to work around all problems I faced.

Before I explain the problems, let’s took a look at the final Groovy server:

HttpServerGroovy.groovy

@CompileStatic
abstract class HttpServerGroovy {

    // VERY dirty trick to avoid the creation of a groovy.lang.Reference
    static File baseDir

    static void main(String[] args) {
        def port = args.length > 0 ? args[0].toInteger() : 8080
        baseDir = args.length > 1 ? new File(args[1]).canonicalFile : new File(".").canonicalFile

        def server = HttpServer.create(new InetSocketAddress(port), 0)
        server.createContext("/", new HttpHandler() {
            @Override
            void handle(HttpExchange exchange) throws IOException {
                def uri = exchange.requestURI
                def file = new File(baseDir, uri.path).canonicalFile
                if (!file.path.startsWith(baseDir.path)) {
                    sendResponse(exchange, 403, "403 (Forbidden)\n")
                } else if (file.directory) {
                    String base = file == baseDir ? '': uri.path
                    String listing = linkify(base, file.list()).join("\n")
                    sendResponse(exchange, 200, String.format("<html><body>%s</body></html>", listing))

                } else if (!file.file) {
                    sendResponse(exchange, 404, "404 (Not Found)\n")
                } else {
                    sendResponse(exchange, 200, new FileInputStream(file))
                }
            }
        })
        server.executor = null
        System.out.println(String.format("Listening at https://localhost:%s/", port))
        server.start()
    }

    private static List<String> linkify(String base, String[] files) {
        def out = new ArrayList<String>(files.length)
        for (int i = 0; i < files.length; i++) {
            String file = files[i]
            out << String.format("<ul><a href=\"%s/%s\">%s</a></ul>", base, file, file)
        }
        out
    }
    ...

The first thing you will notice is that it’s far from being idiomatic Groovy. Of course I used @CompileStatic, because the static nature of GraalVM would have made this an even greater challenge to make it work with dynamic Groovy. However, I didn’t expect that it would be so hard to make it work. The resulting file is both a consequence of limitations of GraalVM, and historical background of Groovy.

Where are my closures?

The first code I wrote was using idiomatic Groovy, with closures. However, as soon as I started to build my native image, I noticed this obscure error:

com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call: java.lang.invoke.MutableCallSite.<init>(MethodHandle)

It’s funny to see this MethodHandle error when you know that the code is fully statically compiled, and that it doesn’t contain a single method handle. However, the Groovy runtime does, and this is where the fun began. First of all, GraalVM tells you what method is problematic. This was org.codehaus.groovy.vmplugin.v7.IndyInterface.invalidateSwitchPoints. Things are getting a little clearer: for some reason, the Groovy runtime is initialized, and we load the dynamic IndyInterface, that I won’t ever need.

The "some reason" needs a bit of explanation. Despite the fact that we use statically compiled Groovy, we’re still implementing Groovy specific interfaces. For example, the GroovyObject interface. Similarly, we honor class initialization the same way as a dynamic class, meaning that when a statically compiled Groovy class is instantiated, even if it doesn’t contain any dynamic reference, we will initialize its metaclass, and as a consequence try to initialize the Groovy runtime.

However something was wrong: looking at my code I could not figure out what would cause initialization, because my entry point was static. In fact, the answer was easy: it came through the closures.

Well, that’s what I thought, because even after eliminating closures, I still got the damn error. In fact, it turns out the situation is far more complex. For example, I had this innocent looking code:

def baseDir = args[0]
server.createContext("/", new HttpHandler() {
    @Override
    void handle(HttpExchange exchange) throws IOException {
        ...
        someCodeUses(baseDir)
    })

The fact that we use baseDir within an anonymous inner class, and that Groovy uses the same code generation under the hood for both closures and anonymous inner classes, that the baseDir variable is allowed to be mutated in the inner class. Of course here I’m not doing it, but because the compiler doesn’t eliminate that possibility, what it does is generating a groovy.lang.Reference for my local variable, that is used in the inner class. And, initializing the Reference class would cause an additional path to this IndyInterface method call…

In the end, the problem is not that much that there’s a MethodHandle, it’s that there are potentially different code paths that lead to this, and that GraalVM can’t figure out in the end a single method to be called: we’re just defeating the system!

For example, even creating an anonymous inner class would still trigger the creation of a metaclass for it: this means that even if we replace the closure with an inner class, in the end, we would still trigger the initialization of the Groovy runtime.

I tried to be smart and remove the IndyInterface from the code that GraalVM is using to generate the native image, knowing that in the end, this code would never be called if I didn’t register the Java 7 plugin (that I wouldn’t use in any case). However, it turns out that GraalVM doesn’t like this, as it has special handling for Groovy, and that if you remove that class, it fails with:

Error: substitution target for com.oracle.svm.polyglot.groovy.Target_org_codehaus_groovy_vmplugin_v7_IndyInterface_invalidateSwitchPoints is not loaded. Use field `onlyWith` in the `TargetClass` annotation to make substitution only active when needed.

So instead I spent hours eliminating those paths, which involved:

turning that shared variable into a field in order to workaround the reference initialization
removing all closures
removing usages of GString (interpolated strings, which is why you see String.format instead)
replacing the short-hand syntax for creating lists (def foo=[]) with an explicit call
removing calls to + with strings (first attempt to remove GString…)
eliminating some classes from the Groovy runtime
replacing some classes of the Groovy runtime with stubs, preventing static initialization

In the end, I have something which works, but you can see that the build file is far more complex.

In particular, it makes use of a little known Gradle feature called artifact transforms. Basically, I’m asking Gradle to transform the Groovy jar before GraalVM uses it. This transformation involves filtering out classes, so that GraalVM doesn’t try to be too smart about them.

Once this is done, we can finally generate a native image for Groovy too:

./gradlew http-groovy:nativeImage

It takes about the same amount of time as with Kotlin to generate a similar 11MB native image. Running it is as easy:

http-groovy/build/graal/httpserv-groovy 9090 /path/to/files

And again it’s super snappy!

Conclusion

At this stage, you might consider that it’s a success: we got both Kotlin and Groovy code compiled into a native image that is very snappy and starts even faster than the Python server. However, getting the Groovy version to work was hours of pain. Each time I managed to fix a problem, another one arose. Basically, every method call, every extension method you call is likely to trigger initialization of some Groovy subsystem, or trigger additional paths to this IndyInterface code. In the end it would be nice if GraalVM could completely eliminate the need for having this class, but until then I just cannot recommend anyone to use Groovy to build native images: it’s just too frustrating. And remember that even if you manage to make it work, it takes both a significant amount of time to do so, but also forces you to write non idiomatic code. Last but not least, any addition to your code is likely to force you to update your GraalVM configuration to make it work. In the end, it’s just way easier to write plain old Java code, or go Kotlin.

Note that I’m not saying that it’s not possible with Groovy, but folks usually face different problems than I did, in particular when it’s just about configuring classes accessed by reflection: this is a simple problem. I’m not saying either that you should avoid Groovy: I just think it’s not suited for this use case. I still use Groovy everyday, in particular in tests or for simple scripts (in replacement to bash scripts). However, more worrisome is that if an application transitively depends on Groovy, it’s unlikely to be "GraalVM compatible".

Eventually, if you look at the Kotlin version and the companion Gradle build, it’s extremely simple, thanks to the great work done by the Palantir team!

Note	After this blog post was published, I received a pull request from Szymon Stepniak improving the Groovy code a lot. The resulting file is, however, twice as big (23MB!). It does not change my vision on this either, because it took 2 men to reach this point, in a significant amount of time.

Gradle myth busting: lifecycle

Tue, 11 Sep 2018 00:00:00 +0200

The build lifecycle

There’s a very common misconception I read from Maven users about Gradle: that there’s no default lifecycle. Not only this is wrong, but actually the Gradle lifecycle is significanly richer.

First, let’s explain in a few words what the lifecycle is in Maven. Detailed explanations can be found here, but in a nutshell, the idea is that _any build will always consist in a sequence of phases, and that each phase is built on top of the previous one. This has the advantage of being simple to explain: to deploy an application, you first need compile it, run the tests, package it, perform validation (checkstyle, …) and deploy. Let me ignore install which is an artifact of how Maven works.

Maven plugins attach themselves to those lifecycle phases, and define goals on different phases. For example, a code generator would attach itself to the generate-sources phase, and define a goal that runs at this phase. Fun begins when you have dependencies between the goals, and ordering matters… Anyway, the general idea is that if you want to get the outcome of, say, packaging, you have to execute all previous phases and consequently all goals that are defined on those phases. The other consequence is that the Maven lifecycle is biased towards the Java model, and more specifically, building Java libraries. It’s even clearer when you think about the term deploy, which doesn’t mean "deploy this application on production", but "push this jar on the external Maven repository". Similarly "install" doesn’t mean "install this application on my laptop", but "copy this library into my local .m2 repository". This, I would argue, is rather counter-intuitive…

Gradle, on the other hand, is a generic build tool. It’s aimed at the Java ecosystem, but also the native one, the Android ecosystem, Python, Go, … It doesn’t matter. All of those ecosystems have an underlying model, and the way you build applications or libraries in each of those ecosystems is different. Gradle offers the APIs to model the builds of each ecosystem. This flexibility is often what troubles Maven users, and makes them think there’s no lifecycle, but it’s not true.

Goals vs tasks

To understand why, we need to explain that Gradle model is not phase based, but task based. It’s a bit like what Ant did, but the similarity stops there. While Ant didn’t define any convention, any lifecycle, and everything was different from one build to the other, this is not true with Gradle. By default, if you apply this "Java library" plugin, you’ll get all the conventions you find with Maven:

plugins {
   id 'java-library'
}

This is the minimal build file you need to build, test and package a Java library with Gradle, with the same conventions as Maven src/main/java, … By applying this plugin, Gradle internally applies a sequence of plugins, which, in turn, would define new tasks, and more specifically for the topic of this blog post, lifecycle tasks.

In Gradle, a task is responsible for executing a unit of work. For example, "compile this source set". A task has inputs (the source files) and outputs (the class files). But a task also has dependencies. In particular, dependencies on other tasks. Gradle makes sure that a task graph is a DAG (direct, acyclic graph). This means that if you run compile on the command-line, what Gradle does is:

compute the task dependencies of compile
execute the tasks in order

Task dependencies can be explicit (say, compile.dependsOn(compileJava)), or implicit (because you add a source set as an input, and that this source set is generated by another task, then we know we need to run the code generation for this source set before). This model is nice because it’s significantly more fine grained than the phase one. When you execute a task, Gradle will always perform the minimal amount of work required to get the output of this task. Let’s illustrate with an example: say you want to run the unit tests of your library. You would run the test task with Gradle. Gradle would then determine that:

it needs to compile the sources of the library (compileJava)
but the sources includes a generated source set, so it needs to execute it (generateSources)
it would also find that the "resources" are an input of the test classpath, so execute the processResources task
etc…

But, in the end, it would not generate the jar file. Because Gradle knows that to run the tests, there’s no need to get the jar: we can build a classpath that consists of the generated classes and the resources directories. It’s actually very easy to figure out what the task dependencies are by running with --dry-run, or using a build scan.

So, will you tell me what the lifecycle of Gradle is then?

This is the trick. With Gradle, everything boils down to tasks, which are a bit like functions, with inputs and outputs. But there are special kinds of tasks, that we call "lifecycle tasks", which are binding other tasks together. They, effectively, produce no output individually. Their only role is to have dependencies on other tasks, so that we have nice shortcuts to produce our outputs. For example, the check task is a lifecycle task which has dependencies on the test task, but also the checkstyle task, etc… Plugins are free to add dependencies to the check task, and enrich the check lifecycle this way. But even better, by defining dependencies between tasks like that, and clearly defining the inputs and outputs of each task, we make it possible to get correct incremental builds, as well as caching (and no, this has nothing to do with ~/.m2).

The good news is that because lifecycle tasks are just regular tasks, it means they can also depend on each other, and you can build your own lifecycle tasks. It becomes very easy to model your build production pipeline. So here is a simple correspondance matrix for Maven users, for the Java library plugin:

Table 1. Lifecycle correspondance matrix
Maven	Gradle	Description
clean	clean	Removes the outputs of tasks
compile	classes	Generates the classes from source files
test	test	Executes unit tests
package	assemble	Creates a jar
verify	check	Runs all tests, integration tests, quality checks, …
install	publishToMavenLocal	Gradle doesn’t need a local repository, but should you need Maven interoperability, you can add the `maven-publish` plugin to add this task
deploy	publishToMavenRepository	This tasks is not available by default, as it depends on which type of repository you deploy to. In general you just apply the `maven-publish` plugin to add this task

But remember: in Gradle, tasks depend on each other. So it means that if you run a lifecycle task, only the tasks required for that specific target are going to be executed. Nothing more.

See us at Devoxx Belgium!

If you want to discover more of the differences between Gradle and Maven, come see my colleague Louis Jacomet and I during Devoxx Belgium, we’re giving a deep dive into Gradle where we’re going to cover what is explained here, and much more!

Assistants vocaux et robotisation: mon point de vue

Tue, 3 Jul 2018 00:00:00 +0200

Ce billet essaie de résumer une conversation que j’ai eu avec mon frère, concernant les craintes autour des assistants vocaux, et, plus généralement, la robotisation. Je partage une grande partie de son point de vue, mais j’ai aussi de larges différences que je voulais résumer ici. C’est parti !

"Ok Google, crée un malaise"

Le week-end dernier, à l’occasion d’un barbecue en famille à la maison, ma moitié a, au détour d’une conversation, lancé cette phrase par pur réflexe:

"Ok Google, <question anodine>"

Là, mon frère, Guillaume, a fait un malaise. Etendu sur le sol, pris de spasmes incontrôlables, en position foetale, il baffouillait des phrases pour la plupart inintelligibles, mais, par moments, j’arrivais à comprendre des bribes comme "pas eux", "GDPR", ou autres "Saint Qwant venez-vous en aide". Après quelques minutes, l’intervention d’un seau d’eau et rongé par la honte de se montrer ainsi devant junior, Guillaume repris ses esprits, et commença une discussion trop courte parce qu’interrompue par la nécessité autrement plus primaire de faire griller quelques saucisses.

Au final, la question se résume à celle-ci: pourquoi ? De mon côté, je suis assez ouvert sur l’utilisation des assistants, et je dois dire que c’est quelque chose que j’attendais véritablement depuis des années. De lonnngggues années. Guillaume n’avait probablement rien noté jusqu’ici parce que mon assistant prenait la forme d’une enceinte bluetooth classique, alors qu’il s’agit en fait une enceinte Sony intégrant la technologie de Google. J’avais choisi ce modèle principalement parce qu’une des utilisations que je fais de cet assistant, probablement la plus fréquente, est de lui demander de me diffuser de la musique (via Deezer), ou la radio. La domotique, c’est un peu un rêve de gosse pour moi, j’adore tout piloter, par une appli, ou par la voix. Donc parmi les autres utilisations que j’en fais, il y a:

contrôler mes lumières, mon thermostat
contrôler ma box domotique (et donc les appareils non directement "connectés")
demander des informations (météo, à quelle heure est le match de la France, …)
enclencher un minuteur pour la cuisine, ajouter des rappels

Le problème, qui choque Guillaume, est que pour avoir accès à toutes ces fonctions, je partage énormément de données avec El Diablo (Google). Oui, mais j’en suis conscient. Et, par ailleurs, mon compte est configuré avec une extrême précaution. Notamment, j’ai complètement désactivé la personnalisation des annonces. Je garde néanmoins certaines fonctionnalités comme le traçage de ma position géographique (qui, par une occasion, m’a permis de démontrer à des amis qu’on était bien allé les voir au mois d’Octobre :)). J’ai plus de mal, en revanche, sur l’historique des recherches. Mais le noeud du problème, c’est qui détient les données, ce qu’ils en font et en particulier avec qui ils les partagent. Pour ma part, la seule raison pour laquelle j’utilise les services de Google est qu’ils me permettent techniquement de faire ce que je veux. Si, demain, j’ai la possibilité d’avoir une box locale qui m’offre les mêmes capacités, y compris lorsqu’invoquée via IFTTT, je signe à quatre mains. En attendant que celà existe, et j’espère sincèrement que celà arrivera au plus vite, je fais des compromis, et il faut être conscient de ce que l’on a a la maison.

Là où je suis moins d’accord, c’est la diabolisation à l’extrême des assistants. Je reviendrais plus tard sur quelques raisons, mais en particulier un argument que j’entends souvent, c’est "oui mais là tu as un micro qui écoute tout le temps chez toi et qui envoie tes données à Google". Ah, nous y voilà. Quel est le problème ?

ça écoute tout le temps ?
ça envoie tes données ?
ça les envoie à Google ?
qui traite des infos ?

Intention malicieuse ou non ?

J’avoue que j’ai du mal à comprendre les réticences ici. Probablement parce que je suis un développeur naif, mais les bugs, les amis, je suis désolé de vous l’apprendre, il y en aura toujours. Donc, oui, il arrivera que votre Google Home, votre Amazon Alexa ou autre, se mette à enregistrer une conversation qu’il n’aurait pas du, et l’envoie sur les serveurs de Google. En règle générale, on le comprend assez vite, parce que la douce voix robotique vous répondra quelque chose du genre "désolé, je ne sais pas comment vous aider". Que s’est-il vraiment passé ? La box a cru entendre un mot clé. Souvent, on se demande parfois pourquoi Google n’offre pas la possibilité de changer ce mot clé ("Ok Google", "hey Google", …). Je pense qu’une des raisons est ce que l’interprétation de ce mot clé est forcément locale. La puissance nécessaire pour reconnaitre parfaitement l’expression, et son contexte de déclenchement avec le bruit, est restreinte. Donc, parfois, pas de bol, ça croit entendre "Ok Google", ou, pire, ça s’enclenche lorsque la pub passe à la télé. En règle générale, ça fait plutôt rire mes enfants, rien de dramatique. D’autant qu’on sait que ça s’enclenche grâce au signal sonore "attention, Google écoute". Donc, un bug, c’est un bug. Ca n’est pas parce que ça peut envoyer des données sans votre accord que l’intention derrière est malicieuse. Dois-je vous rappeler les histoires d’activation à distance des webcams des Mac Book par la NSA ? On n’est pas dans la même catégorie, là. Enfin, des micros qui écoutent tout le temps, vous en avez en poche depuis des années: vos téléphones mobiles. Ils font exactement la même chose, sont activables à distance, et probablement infectés de tonnes de malwares, parfois installées en usine, à des fins d’espionnage gouvernemental. Si je dois me méfier de quelque chose, personnelement, c’est plus de ça que de Google recevant ma discussion sur tata Simone.

Par ailleurs, une fois la conversation envoyée sur le "cloud" (ça fait peur, le cloud, on ne sait pas ce qui s’y passe), elle est analysée. Pensez-vous que ce sont des humains qui analysent votre question et renvoie la réponse en 2s ? Non, la puissance de calcul nécessaire pour interpréter correctement votre question et titanesque. Elle nécessite des techniques avancées (TALN, statistiques, oui, vous savez, la même chose que ce gros mot "big data"), et l’apprentissage pour l’amélioration de la qualité des réponses requiert le stockage de quantités monstrueuses d’information. Je ne suis pas convaincu que Google doive pour autant stocker toutes les conversations, mais je conçois parfaitement qu’en analysant les requêtes de millions d’utilisateurs, avec des voix différentes, des accents différents et des contextes culturels différents, on est capable de faire beaucoup mieux qu’en ne stockant rien et analysant localement. Le temps où il fallait patiemment passer des heures à répéter des tonnes de phrases à son logiciel de reconnaissance vocal n’est pas si loin…

Donc, oui, Google va recevoir ma voix. Oui, il va l’analyser et construire un profil, qu’il pourra partager. Je suis au courant. Et j’ai, dans la mesure du possible, restreint ces possibilités. Je serai d’autant plus heureux le jour où je n’aurai pas besoin de lui confier mes données. Mais l’intégration, la simpliciter d’utilisation, et le seul fait que ça marche, aujourd’hui, est la raison de son adoption. Enfin, on dit que ça marche, mais après des mois d’utilisation, il ne faut pas très longtemps pour comprendre que malgré les quantités énormes de données dont Google dispose, dans de très nombreux cas, l’assistant est complètement à la ramasse. Genre, on lui demande si un match passe sur TF1, il est incapable de répondre. On lui demande "quel temps va-t-il faire", il comprend "température". On apprend même à reformuler nos questions pour que l’assistant les comprenne. Bizarre, et frustrant. Mais, ça montre les limite de la technologie, et casse quelques mythes. En particulier, il faut se méfier des annonces autour de Google Duplex, l’assistant qui prend des rendez-vous à votre place: oui, ça marche, c’est bluffant, mais il est conçu pour cette tâche spécifique, et il est finalement assez simple, d’après les premiers retours, de lui faire perdre le fil.

La relation homme-machine

Néanmoins, la conversation a dévié vers l’intelligence artificielle en règle générale, et la relation qu’on a avec les machines. Sur ce sujet, je suis extrêmement ouvert. Je suis très introverti, et, personnellement, avoir une conversation avec une machine a un côté rassurant. Me demandez pas pourquoi, c’est un fait. Je narrais aussi cette anecdote à mon frère: un de mes enfants a des difficultés à parler aux adultes. Il marmonne, ne les regarde pas dans les yeux, et met beaucoup de temps à leur accorder sa confiance (mais une fois qu’elle est acquise, il n’y a plus de pb). Personnellement, ça ne m’a jamais inquiété, j’étais pareil enfant, et ça n’a (je pense) pas fait de moi un être associal (j’espère). Disposant de Google Home, mon fils s’est naturellement mis à l’utiliser lui aussi. Et ses premières expériences étaient frustrantes: n’articulant pas, la machine n’était pas capable de le comprendre. En quelques semaines seulement, il a progressé plus qu’en plusieurs années: désormais, il articule et parle fort, y compris aux adultes (à de rares exceptions près pour ceux qui l’impressionnent comme le médecin). Ce sujet en a apporté un autre: celui du fait de faire dire aux enfants "merci" ou "s’il te plait" aux machines.

Guillaume est, si j’ai bien compris, contre l’humanisation des machines. Pour ma part, j’y suis favorable, et je suis donc pour la "politesse envers les machines". Fondamentalement, je pense qu’une partie des craintes de Guillaume se fondent sur le fait de confier l’empathie aux machines et que des algorithmes s’en servent pour nous manipuler. L’autre aspect est l’aliénation de l’homme à la machine, c’est à dire apprendre à des enfants à obéir non plus à des hommes mais à des machines. Soit, c’est une crainte légitime. Je suis plus cynique que celà: je doute qu’une machine puisse faire pire qu’un être humain. En fait, j’ai plus d’espoir envers les machines que je n’en ai envers les êtres humains. L’histoire a montré a de trop nombreuses occasions que l’empathie n’est pas l’apparat de tout le monde, bien au contraire. L’utilisation des biais cognitifs à des fins de manipulation (vente, politique, abus de confiance), ça n’a rien de nouveau. Est-ce qu’il faut craindre que les algorithmes fassent de même ? Oui. Serait-ce pire qu’avec des humains ? J’en doute, bien au contraire. L’imagination perverse des hommes n’est plus à démontrer. Leur intolérance non plus.

Suis-je humain ?

J’ai grandi avec Star Trek.

Cette série est probablement celle qui a influencé une grande partie de ma pensée. Déja dans Star Trek TOS, l’ordinateur et la domotique étaient au coeur de la relation homme-machine. La machine était naturellement intégrée, un personnage "vivant" en quelque sorte. Puis, arriva Star Trek The Next Generation, avec des personnages encore plus humanistes, mais un personnage en particulier, Data, change la donne: un Androide dont la seule quête était de devenir humain. Un membre à part entière de l’équipage. Un épisode en particulier traite spécifiquement du sujet du droit des robots: Data devait-il être considéré comme une machine ou un humain ? Un scientifique a-t-il le droit de le démembrer pour l’étudier sur le seul prétexte qu’il s’agit d’une machine ?

J’adore cette série pour celà: son humanisme, sa capacité à rechercher ce qu’il y a de meilleur chez l’homme, et son ouverture en général. Subtilement mais sûrement, les sujets sensibles sur le racisme, le transhumanisme, l’homosexualité y sont traités.

Je fais donc partie de ceux qui préfèrent avoir une "conversation" avec une machine polie, lui répondre "merci", tout en sachant qu’il ne s’agit que d’une machine, plutôt de de causer à un con… raciste, homophobe, ou plus fréquemment un télécommercial essayant toutes les techniques possibles pour me vendre ses panneaux solaires sachant pertinnement que je veux terminer ma conversation avec lui. Je pense que la politesse, le "savoir vivre", procure une certaine satisfaction, une relaxation, qui est un concept totallement orthogonal avec le sujet (l’homme ou la machine). Il n’est pas surprenant pour moi que des autistes s’ouvrent plus facilement avec la présence d’un animal (chien, chat, peu importe), ou que certaines expériences montrent des enfants faisant des progrès en parlant à des robots. L’interaction, la socialisation est importante. Et si le robot est meilleur que l’homme sur ce sujet, on doit s’en réjouir ! L’homme a toujours cherché à créer des choses à son image (ça ne vous rapelle rien ?) et il se trouve que notre cerveau est conçu pour reconnaitre la douleur de l’autre (empathie) et y réagir. Je préfère donc 100 fois, que dis-je, 1000 fois un homme poli envers un robot, qu’un homme traitant une autre personne comme une machine. C’est peut-être naif, mais j’ai espoir qu’en apprenant à des enfants à être polis envers des machines, ils apprendront d’autant plus à l’être envers les autres, et qu’on effacera une partie de ce mal, qui est la disparition à l’échelle de la société de l’empathie. Et mon rêve de gosse, c’est de voir un Data émerger. Montrer qu’avec suffisamment de puissance, une machine puisse prendre son autonomie. On en est encore loin, mais, on réduit déja ce qui est "visible" entre homme et machine. Si je ne suis pas capable de faire la différence entre un homme et une machine dans une conversation, qu’est-ce que celà dit de moi ? Qu’est-ce que celà dit de la machine ? Et si, nous mêmes, n’étions finalement qu’une machine un peu sophistiquée, issue de l’évolution. Une machine douée de pensée et de sentiments, mais une machine quand même.

Parlons Gradle : Episode 1

Mon, 30 Oct 2017 00:00:00 +0100

Suite à de nombreuses discussions via Twitter, j’ai souhaité organiser des discussions en ligne pour que nous puissions parler de Gradle. Le premier épisode aura lieu mercredi prochain, notez la date !

LE MERCREDI 6 DECEMBRE , à 17h GMT+1 (Paris)

La rencontre se fera via Google Hangouts. Si vous êtes intéressés, envoyez-moi un mail (cedric -at- gradle -point- com) ou un tweet avec votre email pour que je vous envoie une invitation. Pour des raisons pratiques, elle se fera en français uniquement !

De quoi va-t-on parler ?

L’objectif pour moi est de récolter votre ressenti sur Gradle, son utilisation, vos problèmes au quotidien, la documentation, les exemples, mais aussi ce que vous aimez, votre utilisation, … Bref, votre expérience utilisateur. Certains sont encore convaincus que Maven est supérieur, ce qui, de mon point de vue, reste un mystère, et j’ai donc besoin d’entendre votre feedback pour savoir comment corriger (si nous n’avons pas encore entrepris de démarche dans ce sens).

Bref, il s’agira d’une discussion ouverte, d’environ 1h. Bien entendu nous pourrons parler du futur de Gradle si nous avons le temps.

Au plaisir de vous voir !

Pourquoi je soutiens Benoît Hamon

Wed, 1 Mar 2017 00:00:00 +0100

Cette élection présidentielle est unique. A bien des égards. La première raison, qui m’est insupportable, est que pour la première fois, une candidate FN, Marine Le Pen, est en position de gagner cette élection: un mirroir de ce qui se passe à l’échelle mondiale, avec un repli sur soi et une montée inquiétante du populisme. La seconde, parce qu’évidemment jamais le rejet des politiques en place n’aura été aussi important. Pour autant, c’est aussi une opportunité, pour une fois, de changer radicalement de société. Les choix qui s’offrent à nous sont particulièrement clairs: une droite extrêmement dure avec Fillon et Le Pen, ou une gauche sans compromis avec Hamon ou Mélenchon. Sans oublier, bien sûr, l’outsider désormais favori Macron, dont je ne sais toujours pas s’il a des convictions ou s’il est simplement le reflet de la nature, qui a horreur du vide (je ne sais pas où voter, allons là où tout le monde semble se rallier).

Pour ma part, je vote par conviction. J’aurais bientôt 38 ans, et je dois dire que jamais, je n’ai été si proche des idées d’un candidat. Je le suis depuis plusieurs mois, bien avant la primaire, et mes amis proches savent que je leur en parlais et que je croyais en sa victoire. Dans ce billet, je veux vous donner quelques pistes pour lesquelles je soutiens Benoît Hamon, et pourquoi je vous invite à le rejoindre.

Note	A mes lecteurs habituels, ce billet n’a rien à voir avec mes interventions techniques. Libre à vous de l’ignorer. Par ailleurs, je souligne évidemment que ce qui est écrit ici n’engage que moi, et personne d’autre que moi.

Le revenu universel

Commençons par le revenu universel, parce qu’il est le plus clivant. Même pendant la primaire des gauches, j’ai entendu des absurdités incroyables à son sujet, ou des questions très orientées des journalites simplement destinées à décridibiliser cette proposition. Pourtant, c’est une idée neuve, défendue par de nombreux économistes, de droite comme de gauche, ainsi que d’éminents chefs d’entreprise, tels Elon Musk, le fondateur Tesla, Space X, devenu richissime avec la revente de PayPal, … Pas spécialement un exemple de ratage. Rappelons son principe de base: donner, à chaque citoyen, un revenu identique (dont le montant varie selon les propositions), universel, indépendemment de ses resources, à compter de sa majorité. A droite, on crie alors à l’assistanat. A gauche, on hurle qu’il ne faut pas donner aux riches qui ont déja bien assez.

Mais le monde change. Le travail change. Quiconque vous dit le contraire est soit aveuglé, soit vous ment. Je travaille dans l’informatique, et par mon activité professionnelle qui m’amène à rencontrer de nombreux développeurs du monde entier, je pense être assez bien placé pour voir que ce je développe, ce qui arrive, est une nouvelle révolution industrielle (mais pas seulement, les services sont aussi touchés).

En France, vous aurez déja remarqué que les caissières se font moins nombreuses, qu’il n’y a plus personne aux péages, que le métro n’a plus de conducteur, qu’on nous vend du pain frais dans un distributeur… Mais nous sommes très en retard.

En Chine, où sont fabriqués nos précieux smartphones, les monstrueuses usines employant des dizaines de milliers d’ouvriers ont déja entamé leur migration vers une robotisation intégrale.
A Singapour et aux Etats-Unis, les taxis sont remplacés par les voitures autonomes (notez l’ironie, au passage: pour un Fillon ou un Macron dont le modèle est l’Uberisation de la société, l’auto-entreprenariat, là où il est clair depuis des mois que le but de cette même entreprise est le 0 salarié…). Des poids lours conduisent tout seuls sur des centaines de kilomètres. Amazon livre par drones.
Au Japon, ce sont mêmes des employés d’une assurance qui sont remplacés par une intelligence artificielle.
Watson, l’IA développée par IBM, a déja diagnostiqué une leucémie que les médecins avaient ratée.
Un robot pourra bientôt transporter des charges lourdes à notre place

Quoi qu’on en pense, à moyen terme, la révolution robotique est en marche, et c’est une bonne chose. Bien sûr, on ne parle pas d’un remplacement à court terme. Tout celà est progressif. Des métiers vont disparaître, d’autres se transformer, d’autres encore vont apparaître, mais il est indéniable que le volume de travail global disponible à l’humain fond comme neige : nous sommes toujours plus nombreux sur la planète, le PIB augmente, pour autant le volume d’heures travaillées évolue peu. L’étude la plus optimiste de l’OCDE indique que près de 10% des emplois sont menacés à court terme. Une autre étude parle elle de 50% des emplois remplaçables par des robots à l’horizon 2030. Lorsqu’on choisit une politique, il ne faut pas penser à court terme, mais à nos enfants. Même s’il est idiot d’affirmer que le travail va disparaitre (qui peut sérieusement le croire ?), il se transforme, et il faut embrasser cette transformation.

J’en suis convaincu, les politiques qui ignorent ces changements vivent dans un monde qui n’existe plus: ils achètent leur pain au chocolat à 10 centimes, font leurs courses chez Prisunic et rêvent de payer leurs assistants parlementaires européens en Francs. Ignorer les changements du travail, c’est ignorer l’avenir, s’aliéner à ceux qui embrassent ce changement, perdre notre souveraineté. Au lieu de combattre ce futur, il faut au contraire s’y préparer et repenser notre société autour de ces changements. L’automatisation, la robotisation, n’engendre pas de perte de valeur: celle-ci est toujours présente, les prix ne baissent en général pas: la productivité augmente et on réduit la pénibilité. Deux choix de société s’engagent alors: profiter de cette amélioration pour vivre mieux, ou une fuite en avant productiviste. A ce sujet, je conseille la lecture de ce billet à mes lecteurs anglophones: ou comment le temps de travail est fonction des évolutions de la société, pas une constante.

Ceci appelle à une révolution fiscale: là où aujourd’hui on taxe le travail, il faudra demain taxer les robots, comme le propose aussi Bill Gates. Si ce terme peut faire sourire, il cache en réalité de nombreuses ramifications: évasion fiscale en particulier. Quelle pire injustice que de voir nos petites entreprises crouler sous des 30% de taxes, lorsque les grandes entreprises étrangères qu’on utilise tous (smartphones, distribution), qui robotisent, automatisent et n’embauchent quasiment pas, placent "leur" argent dans des paradis fiscaux et ne paient pas d’impôts, grâce à des montages financiers complexes avec la complicité de nos cabinets comptables ? Ne marchons nous pas sur la tête ?

Alors, oui, on peut continuer à sourire, dire qu’il faut baisser les charges pour les entreprises et les salariés (ce qui serait une bonne chose), mais si on n’embrasse pas cette virtualisation de l’économie, on ne fait que réduire l’assiète des cotisations, et se ruiner un peu plus.

Alors que vient faire le revenu universel là dedans ? Il permet:

de mettre fin à la grande pauvreté, en particulier en remplaçant le RSA, belle idée mais tellement complexe que nombreux sont ceux qui y ont droit et ne le demandent pas. Par ailleurs, si le revenu est effectivement universel, l’argument de la fraude aux aides ne tient plus: tout le monde y a droit, point.
de favoriser le temps partiel. Dans certains Etats, comme les Pays-Bas, c’est déja la norme, y compris chez les hommes, réduisant ainsi les inégalités homme-femme. Le PIB de ce pays n’a pas chuté, les employés n’en sont que plus heureux.
de maintenir l’emploi. Des histoires d’agriculteurs qui travaillent comme des fous et ne s’en sortent plus, on en lit tous les jours. Avec 350€ par mois, franchement, vous resteriez? Je connais personnellement l’un d’entre eux qui a fini par abandonner. Avec le revenu universel, on permet à ces travailleurs pauvres de rester en poste.
de choisir de travailler plutôt que rester à la maison. Je ne sais pas pour vous, mais le nombre de fois où j’ai entendu quelqu’un me dire "avec mes frais de garde et de transport, ça me revient plus cher d’aller travailler que de rester chez moi". Pas franchement encourageant, non?
de valoriser des activités non commerciales mais importantes pour la société : travail associatif, aides à la personne, …
de partager le travail: à coût égal pour une entreprise, il sera possible de prendre 2 salariés au lieu d’un seul, et ainsi donc réduire les risques de TMS (pour les travaux manuels) ou burn-outs : il existe une limite physique à la productivité.

En revanche, le RU ne rend pas les riches plus riches. C’est un revenu. En conséquence, il est imposable. Il s’ajoute donc à votre salaire et suivant vos revenus, vous devrez payer ou non des impôts dessus. Il est donc incorrect de dire que ça coute 400 milliards (ou plus). En pratique, il coûte sensiblement moins, en particulier si son implémentation se fait via un crédit d’impôt. La justice fiscale est maintenue. Le soumettre à conditions de ressources, c’est augmenter son coût intrinsèque (gestion, contrôles) ou inciter à la fraude pour quelque chose qui ne change au final rien. Quant à l’argument de l’incitation à l’oisiveté, il me fait doucement sourire. A tous ceux qui me l’ont avancé, je leur ai posé une question simple: si on vous donnait 600€ par mois, est-ce que vous quitteriez votre boulot ? Est-ce que ça vous suffirait ? Je pense que je n’ai pas besoin de vous donner les réponses…

Alors, certains taxent cette proposition d’irréaliste, utopique, irréalisable et j’en passe. Ils doutent de sa crédibilité. Mais que proposent-ils ? Toujours les mêmes solutions ! Réduire les charges des entreprises (merci Fillon pour son exonération sur les bas salaires, effet d’aubaine pour n’augmenter personne, alors que la répartition de la valeur n’a jamais été aussi dévaforable aux pauvres et classes moyennes). Augmenter la TVA (belle justice fiscale !). Qui se souvient, à l’inverse, de la baisse de la TVA pour la restauration, en l’échange, soit disant, d’emplois et de baisses de prix ? Qui se souvient encore du badge 1 million d’emplois de Pierre Gattaz en l’échange du pacte de compétitivité ? Travailler plus, en défiscalisant les heures supplémentaires, encore une mesure populaire, mais qui ne crée pas d’emplois (la vraie injustice), et n’est qu’un pansement sur une jambe de bois: est-ce que cette politique a déja une seule fois marché ?

Autre question: qui peut encore croire que repousser l’âge de la retraite est une solution, alors même que les seniors sont "trop vieux pour travailler" dès 50 ans ? Je ne suis fondamentalement pas contre repousser l’âge de la retraite (qui finalemement va de pair avec l’augmentation de l’espérance de vie), mais faire ça alors même que nos seniors sont au chômage n’a aucun sens. Il faut d’abord revaloriser l’expérience, changer notre vision. Comment peut-on imaginer créer des emplois en faisant travailler plus ceux qui ont la chance d’en avoir un, quitte à les briser ?

60 milliards ont été engloutis dans le CICE, pour quel résultat ? Autre solution classique, se reposer sur une croissance, dont les prévisions sont systématiquement revues à la baisse, et donc les conséquences sur notre planète commencent sérieusement à se faire sentir ? Ou faciliter le licenciement, en croyant dogmatiquement que ça facilite l’embauche ? Ce dont ont besoin les entreprises, ce n’est pas de pouvoir licencier. Personne ne veut licencier par plaisir. Ce qu’il faut, c’est de la stabilité. Connaitre les règles du jeu à l’avance, et qu’elles ne changent pas tous les jours, ainsi qu’une juste concurrence : que les petites paient les mêmes impôts que les grosses, et que les différences de législation entre partenaires commerciaux ne soient pas un frein aux entreprises locales (donc, un protectionnisme raisonné). Alors, dites-moi, qui est irréaliste et utopique ?

Alors, on va nous parler de "valeur travail". Ou la réalisation par le travail. Est-ce là une valeur que je veux transmettre à mes enfants ? Non. Si le travail est important, il ne doit pas être le critère de réalisation, en particulier dans une société ou le travail se raréfie. C’en est de plus insultant pour tous ceux qui ne trouvent pas d’emploi, ou pour ceux qui s’investissent autrement (associations, sportifs, artistes) et contribuent à la grandeur de notre pays. Ce que j’enseigne à mes enfants, c’est qu’il faut se donner à fond et ne pas se fermer de portes. Je leur apprends la tolérance. Je leur apprends à apprécier la chance qu’ils ont par rapport à d’autres. Je leur apprends que l’effort est récompensé, mais que parfois, la vie est injuste. Travailler, oui, mais choisir. Le choix, la liberté sont la clé de la réalisation.

Et d’avenir, parlons-en.

La révolution écologique

Lorsqu’on parle de révolution industrielle, on ne peut ignorer la transition écologique à mener, et son potentiel énorme en termes d’emplois. La France a trop longtemps dirigé tous ses crédits vers le nucléaire, en en ignorant volontairement le coût réel (démantèlement, notamment) et son impact sur l’environnement (bien sûr, un accident n’arrive jamais). Mais le plus gros mensonge, c’est encore notre fameuse indépendence énergétique. Faut-il rappeler d’où vient le combustible si précieux dont nous avons besoin ? Pour autant, il ne faut pas être naif: sortir du nucléaire prend du temps, mais c’est aussi une énorme opportunité. En échelonnant dans le temps, comme le propose Benoît Hamon, on permet de maintenir notre capacité de production, tout en ouvrant de nouvelles voies de développement, créant des emplois. La catastrophe serait de faire comme en Allemagne, où toutes les centrales ont été remplacées par des centrales à charbon, dont le bilan carbone est redoutable… Cette carte interactive est plus que parlante…

Mais l’écologie ne se limite pas au nucléaire. L’opposition ferme de Benoît Hamon aux perturbateurs endocriniens est un autre exemple de ce que j’aime dans sa vision: il est temps d’en finir avec la dictature de la croissance, qui détruit notre environnement et provoque de graves maladies. Penser que je puisse être responsable du futur cancer de mes enfants m’est juste impossible: si je peux faire quelque chose aujourd’hui, quitte à sacrifier un peu de confort (je suis le premier à changer de smartphone tous les 3/4 ans, mais en ai-je vraiment besoin ?), faisons-le. Il existe des solutions: circuits courts, production raisonnée, retour à une agriculture prenant en compte les spécificités régionales, réduction de notre consommation de viande, … Nous devons aussi nous responsabiliser en tant que consommateurs: acheter toujours moins cher, c’est inciter à la baisse des salaires ou à la délocalisation.

La République

Un des derniers points que je souhaite discuter ici est ma vision de la République. Tout d’abord, je souhaite le retour au septennat. Je pense que le passage au quinquénat a été une catastrophe, contribuant à la peopolisation de la politique: on ne cherche plus à faire un projet, on parle tout de suite de la prochaine élection. Même si je n’ai pas d’idée précise de ce que doit être la prochaine République, il me semble clair que celle-ci est à bout de souffle. Conçue pour le Général de Gaulle, l’homme providentiel. Je pense qu’il faut revoir le rôle de Président, et qu’il ne soit plus qu’un garant de nos institutions. Pour celà, il nous faut une assemblée consistuante. Benoît Hamon n’en parle pas spécialemement, et c’est peut-être un des points où je suis plus en accord avec un Mélenchon que lui: il faut bien des points de désaccord. En particulier, je trouve l’idée du 49-3 citoyen avec seulement 1% du corps électoral potentiellement dangereuse : la Manif pour Tous aurait pu bloquer le mariage homosexuel (oui, je suis de ceux qui pensent que donner des droits égaux à tous les citoyens n’est pas supprimer vos propres droits), ou, du temps de Mitterand, je doute que la peine de mort aurait pu être abolie. En clair, je suis plutôt pour une proportionnelle intégrale, répondant à une crise de représentativité qu’entretient élégamment le FN. Mais je suis aussi surtout pour la stochocratie, où le fait de tirer au sort des représentants qui, eux, éliront ceux qui nous gouvernent.

Laïcité et peur de l’autre

Une de mes plus grandes désillusions de l’ère Hollande. C’est là que le Parti Socialiste m’a perdu. J’ai été affligé par la Loi Renseignement. Non seulement elle est liberticide, mais elle est aussi dangereuse pour nos entreprises, réduisant considérablement les sécurités et garanties qu’elles peuvent mettre en place pour leurs clients. Mais le plus grave, le divorce, ce fut la déchéance de nationalité. Sous prétexte de protéger nos concitoyens, nous créons deux catégories de personnes, ce qui est le contraire même des fondements de notre République. Plus encore, une trahison des valeurs de gauche. Aux élections qui ont suivi, pour la première fois de ma vie, je n’ai pas voté PS. Fort heureusement, cette mesure n’est jamais passé, j’aurais eu honte, franchement.

En ce qui concerne le débat sur la laïcité, là encore, je suis à 100% derrière la position de Benoît Hamon: la loi 1901 et rien que la loi 1901. En clair, il s’agit de respecter la liberté de culte. Et respecter toutes les religions. Point. Il faut être sacrément culotté (ou aigri), pour dire "Benoît Hamon est en résonance avec une frange islamo-gauchiste". Malek Boutih, je vous le dis sincèrement, cest propos sont une honte pour la gauche. Les simples sous-entendus de cette phrase me dégoutent. Pire, vous avez donné des arguments à l’extrème droite, que nous combattons de tout notre coeur. C’est indigne et tellement typique de cette gauche archaïque (un paradoxe quand vous vous qualifiez de progressiste) et loin de ses valeurs. Après de telles sorties, ne soyez plus surpris que les gens votent FN et vous bottent les fesses aux élections.

Non, je crois aux valeurs humaines. Le FN tente de nous faire croire que 2000 migrants sont responsables du chômage en France, par pitié, non. Pire encore, les migrants violent nos femmes et mangent nos enfants. Et ils n’avaient qu’à rester chez eux, ou apprendre à nager. Nom d’un chien, où est passé le coeur de la France ? Doit-on laisser mourrir tous ces gens, les suspecter des pires crimes, ou être à la hauteur de notre histoire, nous, le pays des Droits de l’Homme ? Se dire que l’allemagne accueille des millions de migrants dans le même temps où l’on fait des procès à Cédric Hérrou. Dois-je vous rappeler ce que la stigmatisation nous a apporté voici 80 ans ? Ce monde tourne sur la tête…

Les cas Macron / Mélenchon

Pour terminer, parlons rapidement d’Emmanuel Macron et Jean-Luc Mélenchon. A la lecture de ce billet, certains peuvent se dire que je pourrais choisir Mélenchon. C’est vrai, lui et Hamon partagent beaucoup d’idées, tout comme les Verts et à ce titre je suis dépité qu’un accord entre les 3 n’ait pas eu lieu. C’était, à mon humble avis, une opportunité unique pour la gauche compte-tenu de la conjecture actuelle. Je pense, pour ma part, que Benoît Hamon a véritablement compris la transition économique qui nous attend et a une personnalité moins clivante. Les solutions de repli sur soi ne fonctionneront pas. Pas plus que les solutions de Macron, qui reste une énigme. Ses discours sont d’une platitude déconcertante, ses solutions en matière d’économie sont peu ou prou les mêmes que celles de Fillon (libéralisation à outrance du marché du travail), et pourtant… il monte, il monte… Le ralliement de François Bayrou, celui là même qui disait, il y a quelques semaines, que Macron était piloté par les banquiers (ce qui est toujours possible puisqu’il refuse d’indiquer qui le finance)… Je peux me tromper, mais je ne crois pas non plus à la fin du clivage droite-gauche. Nos idées sont différentes. Nos visions de la société sont différentes. Progresser, c’est choisir une vision, et y aller. Faire cohabiter des idées si différentes au sein d’un même gouvernement ne peut conduire qu’à prendre de "petites" décisions ne frustrant personne. Soyez rassurés, cependant, je ne mets pas Emmanuel Macron au même niveau que François Fillon ou Marine Le Pen…

En conclusion, je ne suis pas crédule non plus, tout ceci n’est pas réalisable en un jour et c’est pour celà je j’aime l’approche de Benoît Hamon: jamais il ne s’est présenté comme le candidat providentiel, qui a la réponse à tout. Il faut laisser le temps à une politique de se mettre en place (d’où le septennat). Hamon est un homme qui a une vision, il a travaillé ses dossiers, mais c’est aussi un team player. Toutes ses mesures sont réfléchies, travaillées, et vont dans le sens d’un projet à long terme: un avenir désirable.

La société que l’on souhaite pour nos enfants. C’est à eux que je pense en le choisissant. Ce dont ils ont besoin, c’est d’espoir. Je ne veux pas qu’ils grandissent dans ce monde qu’on nous promet:

Et vous ?

Gradle and Kotlin, a personal perspective

Sun, 22 May 2016 00:00:00 +0200

Gradle embraces Kotlin, what about Groovy?

First of all, it’s been a long time since I last blogged, and I’d like to remind that everything written here are opinions of my own and not the views of my employer, which happens to be Gradle Inc as write those lines.

A few days ago, Gradle and Jetbrains announced a partnership to make Kotlin a first class language for Gradle builds, both for build scripts and plugins. Most likely, you know Gradle has been using Groovy since its inception. Lots of people think that Gradle is written in Groovy, which is actually wrong. Most of Gradle is written in Java. The builds scripts are written in Groovy, lots of plugins are written in Groovy, our test cases are written in Groovy (using the best testing framework out there, Spock), but Gradle itself is written in Java.

From my perspective, this is situation has been very disturbing and continues to be so. I have very good friends in the Groovy community, and this move has been seen by some of them as a betrayal. As an Apache Groovy committer, and someone who spent almost 4 years full time implementing new features of the language, most importantly its static compiler, seeing Kotlin promoted as the language of choice for Gradle’s future, it’s a little strange. One could legitimely say, WTF? I’ve been aware of this work for several months now, and my colleagues Rodrigo B. de Oliveira and Chris Beams have done an amazing job in a very short period of time. From a long time Groovy user and Groovy developer point of view, it’s hard not to make this move an emotional thing. However, business is not about emotions. In particular, what are we trying to acheive with Gradle? We’re trying to help developers build their applications. We’re trying to make this elegant, reproducible, scalable and fast. We’re language agnostic. We can build Java, Groovy, Scala, Kotlin, C++, Python, … Gradle has never been the tool to build Groovy applications: it’s been a tool to build software. It’s a tool about automation. And I’ve been complaining enough about communities that build their own tool for their very specific language to understand that this is super important: Gradle is (or aims at) the best tool for building any kind of software. In short, we must think in terms of what is best for our users, and sometimes, this means changing technogies. A product should not be bound to a technology, but a company should even less be bound to it. And given the response that we had after the announcement, supporting Kotlin seem to drive a lot of excitement around Gradle, and that’s a very good thing. So, let’s take that out, and think what it means for Groovy.

Groovy support is not abandoned

First of all, I already said it several times, but better continue to spread the message: support for Groovy in Gradle is not deprecated nor removed. You can still write your scripts in Groovy, you can write your plugins in Groovy, and you will still be able to do it. But Gradle will likely encourage users to migrate to Kotlin. To be clear, Kotlin support is incubating, and there’s a lot to do to make it as usable as the Groovy version. Second, there are tens of thousands of builds written using Groovy, hundreds of plugins written in Groovy, so it’s not tomorrow that Kotlin is going to replace Groovy. However, we care about the future, so we need to think about what it means in the long term. Should we be excited about supporting Kotlin? Yes we should, because Kotlin is an amazing language. Should we continue to be excited about Groovy? Of course we should, because it’s also an amazing language. But it’s old and as such brings a lot of legacy with it. As someone who implemented the static compiler for Groovy, I know it very well. There are things that are hard to change, because a large part of the Groovy community is very fond of its dynamic behavior.

So let’s focus on the two major aspects that led to embracing Kotlin in Gradle. The fist one, and principal, is IDE support. Let’s face it: even before I joined Gradle, when I was giving talks about it, people were complaining about IDE support. Compared to a tool like Maven, supporting Gradle build scripts is complicated. Supporting XML is easy (to some extent). Supporting a dynamic DSL is not. Some say it’s Groovy’s fault, and I want to correct this statement right now: it’s not Groovy’s fault. While Groovy let’s you design dynamic DSLs, the design of the DSL can be changed to make it easier for tools to "discover" things. But when Gradle was designed, there wasn’t any statically compiled Groovy. The idiomatic way to write DSLs in Groovy, at that time, was to heavily rely on runtime metaprogramming. While loving metaprogramming, I’ve always prefered compile time metaprogramming over runtime metaprogramming. For multiple reasons:

because in most cases, what you want to do at runtime can be done in a unique, setup phase. For example, create your metaclasses, enrich existing types, configure property missing, method missing, … If it’s setup, it’s better done at compile time, because you can report errors, and because it gives higher performance. This led the way I designed the static compiler, and more features of Groovy after that (traits, type checking extensions, …) : describe what you want to do at compile time.
because it makes the life of tools easier. While IntelliJ or Eclipse support DSL descriptors that help them provide completion, those are hard to implement, and often inaccurate. They can only approximate what is going to happen at runtime. And in the end, you’re doing the same job twice: you’re writing a runtime for your DSL, which is dynamic, then you need to write a DSL descriptor for the IDE to understand it. Wouldn’t it be better if all was done in a unique place? Something that both the compiler and the IDE can understand?

So while we know we can describe dynamic Groovy DSLs so that they are understood by IDEs, it’s effectively a lot of work. And if you want to support multiple IDEs, it’s even more work. But in the case of Gradle, it’s even worse: each plugin can provide it’s own "micro DSL". While there’s an "idiomatic" way to configure Gradle builds, it’s no single rule. One can implement it’s own Groovy DSL within the Gradle build. And no luck the IDE would ever understand it. Another pain point is that Gradle adds complexity to complexity in terms of DSL capabilities. For example, when you have a build script that has:

dependencies {
   compile libraries.groovy
}

greeter {
   message = 'hello'
}

sign {
   signature = top
}

often people do not realize that: - dependencies is found on a Project instance - libraries is a user declared variable, that can be found in a plugin, another build script, a project properties file, … (how does the IDE find about it?) - greeter is a convention object, defined by a plugin, to configure the default values of its task - sign is a task, which has a signature property, and top references an extension property from the project

So while this build script is simple to read, it’s hard to understand how it effectively works, because objects can be found at different places, can be provided by different providers (plugins, properties, extensions), but everything is accessed using a single notation. This is bad, because it makes it almost impossible for an IDE to understand what is going on.

The question is, is it Groovy’s fault? My answer is not totally. The fault is mostly on the DSL design, and Groovy made it too easy to do so. But again, that was designed at a time when dynamic Groovy was the rule. I gave a talk, recently, about building modern DSLs with Groovy, where I discourage such practices, and encourage the use of static DSLs instead.

That leads me to the second main reason of embracing Kotlin in Gradle: performance. When we talk about performance, lots of folks tend to think that Groovy is slow. This is not the case. Groovy is pretty fast. However, depending on the design of the DSL, you can easily fall into traps that can lead to catastrophic performance. Before I go further with it, I’m reading way to often that Gradle is slow because it’s written in Groovy and that Groovy is dynamic so it’s slow. F* no, those who tell you that just didn’t profile a build. As I said, Gradle is mostly written in Java. And I’ve spent the last 3 months optimizing the performance of Gradle, and I can tell you that of the dramatic performance improvements that one can see in Gradle 2.13 and 2.14, almost none was obtained by rewriting Groovy to Java, or rewriting Groovy code. None! Most of the hotspots were pure Java code. Period. However, as soon as you use plugins, which are today mostly written in dynamic Groovy, or that your build scripts imply a lot of nested closures, things start to become complicated for "Groovy". Let me explain that clearly. I think at some point, someone made a terrible design decision in Groovy. I don’t know who it was, but the idea was to rely on exceptions to control the flow of resolution of properties. This means that when a property is missing, typically in a closure, an exception is thrown. When a method is not found, an exception is thrown. When a property is not found, an exception is thrown. That seemed to be a good idea, because in the end, you want to provide the user with an error, but in practice, this is catastrophic, because Groovy can capture those exceptions. Typically, in a delegation chain (nested closures), a containing closure or class can actually have this property defined, or implement property missing/method missing. Now, re-think a second about the "simple" example of Gradle build above: how do you now where to look up for message, top, signature, …? Now you know: f* exceptions are thrown, stack traces are filled, and eventually captured because some composite dynamic object finally wants to answer the message… In practice, for some builds I have profiled, it was tens of thousands of exceptions being thrown and stack traces filled for nothing. And that has a terrible impact on performance. So even if we have implemented strategies in Gradle to try to avoid throwing those exceptions (which are responsible for part of the performance improvements in 2.14), this is very hard to do it, and we’re still throwing way too many of them. A static language doesn’t have this problem, because every single reference in source is resolved at compile time. So, if you’re writing a plugin in Groovy, for the sake of performance, please add @CompileStatic.

So there goes Kotlin. Kotlin has excellent static builders support, that make it practical both for IDE support, which will dramatically improve user experience in terms of understanding what do write, what is an error, having documentation, refactorings, … and is a very pleasant language to work with. Honestly, I don’t have anything bad to say about the language (apart from the fun keyword that I don’t like). To some degree, it’s not very surprising: Kotlin has heavily inspired by Groovy and another popular JVM language: Scala. And again, being the one behind the static compiler of Groovy, I can’t blame them for doing what I like about static languages. Their builder support is awesome, and very elegant. And it’s supported out of the box by IntelliJ of course, but also Eclipse.

A static DSL for Groovy?

Ok, so one might think at this point that I’m mad. I wrote a "competing" language, and I’m happy to see Kotlin being promoted in Gradle. I wrote the static compiler, that is capable of doing everything Kotlin can do (minus reified generics, plus superior scripting support, type checking extensions, …), so wtf? Ok, so let’s be very clear: I have absolutely no doubt that Groovy can do everything that we’ve done with the Kotlin support in Gradle. It can be statically compiled, provide an elegant DSL that is statically compiled, and it can be understood by the IDE. I had no doubt before the Kotlin work started, I have even less doubts now. And I can say I have no doubts because I tried it: I implemented experimental support for statically compiled Gradle scripts, written in Groovy. Here’s an example:

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'groovy'
apply plugin: GreetingPlugin

repositories {
    mavenCentral()
}

dependencies {
    compile 'commons-lang:commons-lang:2.5'
    compile "commons-httpclient:commons-httpclient:3.0"
    compile "commons-codec:commons-codec:1.2"
    compile "org.slf4j:jcl-over-slf4j:1.7.10"
    compile "org.codehaus.groovy:groovy:2.4.4"
    testCompile 'junit:junit:4.12'
    runtime 'com.googlecode:reflectasm:1.01'
}

tasks.configure('test', Test) {
    jvmArgs '-XX:MaxPermSize=512m', '-XX:+HeapDumpOnOutOfMemoryError'
}

dependencies {
    compile 'org.codehaus:groovy:groovy-all:2.4.4'
}

extension(GreetingPluginExtension) {
    message = 'Hi'
    greeter = findProperty('greeter')?:'static Gradle!'
}

tasks.create('dependencyReport', DependencyReportTask) {
    outputs.upToDateWhen { false }
    outputFile = new File( project.buildDir, "dependencies.txt")
}

class GreetingPlugin implements Plugin<Project> {
    void apply(Project project) {
        project.extensions.create("greeting", GreetingPluginExtension)
        project.task('hello') << {
            println "${project.extension(GreetingPluginExtension).message} from ${project.extension(GreetingPluginExtension).greeter}"
        }
    }
}

class GreetingPluginExtension {
    String message
    String greeter
}

This is an example Gradle build that is compiled statically. It has none of the problems I described about the Groovy implementation in Gradle above. It uses all the techniques that static Groovy provides: extension methods, powerful scripting with implicit imports, type checking extensions, … All this works. And interestingly, the work that is done to enable support for Kotlin also benefits to statically compiled Groovy, and Java! Let’s not forget about the latter, which is years behind in terms of "modern" languages support. So if this works, why do we need Kotlin? To be honest, I asked it to myself many times. It was very difficult to me, because I knew Groovy could do it. Again, I had no doubt about the language capabilities, no doubt about the performance impact of doing this. However, I missed two critical points:

IDE support. Even if support of Groovy in IntelliJ is by far the most advanced of all other IDEs, it still lacks behind when static compilation is on. But more importantly, it doesn’t know that my script is statically compiled, nor does it now about my custom extension methods. I tried to implement a GDSL descriptor to make it aware of them, and it somehow worked: I do have code completion, but errors are not marked as errors, and the IDE still doesn’t understand that it should only suggest to me what is relevant in the context. With Kotlin scripts which are natively static, there’s no such issue. The IDE understands everything natively, in IntelliJ and Eclipse. So, I have no doubt that Jetbrains can implement support for this, just like I had no doubt I could implement a static Groovy DSL, but who is going to write this? Me? Gradle? I don’t have the time to do it. And it’s not Gradle’s job to write IDE plugins. And what about Eclipse? One big issue that the Groovy community has, today, is that nobody is supporting Eclipse since Pivotal dropped sponsorship of Groovy. After more than one year, nobody took over the development of Groovy Eclipse. Nobody. While Groovy itself saw lots of new contributors, while we saw a lot of bugfixes, new contributors and that the download numbers where never as high as they are today, IDE support is critical. And nobody took over the development of it. I saw some people referring to what Jetbrains is doing as "blackmailing". Seriously? Jetbrains? Think of what they’ve done for Groovy. Groovy would never has been as popular as it is without them. They provided us with the best Groovy IDE possible. They are constantly supporting new features of the language, adding support for AST transformations, traits, … They even added the ability, in IDEA 14, to use Groovy (and not Kotlin, guys!) as the language to evaluate expressions in the debugger. And they would try to kill Groovy? Kill part of their business? Come on guys! So yes, they invested a lot in Kotlin and want to promote it, but how could it be otherwise? And it’s not like if the language sucked: it’s awesome!
Does it make sense? Now that we made the decision to support Kotlin, that we proved it would provide the level of user friendliness we want and that it is statically compiled by default, does it make sense to put resources to support static Groovy in addition? I don’t have an answer to this. I thought yes, but now I’m not sure. Kotlin does the job. And honestly, they have great engineers working on the language. Even if it lacks behind in terms of scripting and compilation times compared to Groovy, I have no doubt they will fix it. How arrogant would we be if we thought other languages could not do what we’ve done with Groovy?

The future of Groovy

The last point I want to address is what it means for the future of Groovy, and what it means for my future in Groovy. First of all, I always thought that the future of Groovy was in the hands of its community. It’s not Gradle that has Groovy’s future in its hands. It’s you. The move to the Apache Software Foundation was also done for this very same reason: community first. If you want to continue to use Groovy, to improve it, to support it, all you have to do is f* do it! And I will continue! I love this language, I know too well how far it can go in terms of DSL support, AST transformations, now in 2.5 we have macros, that’s just a crazily powerful language that’s super fun to use. Should we fear competition? No, we shouldn’t. Competition is good. It should be inspiring. And if Gradle moving to Kotlin means the death of Groovy, maybe the problem is elsewhere. And even if lots of people get introduced to Groovy through Gradle, it’s not the only entry point. Grails is another. Jenkins (through Flow) is another. And many, many more. There was a tweet a few days ago which showed the 100 most popular dependencies in GitHub projects. Groovy was one of them. No Kotlin. No Scala. Groovy. It’s everywhere, and it’s going to be there for a long time.

Part of the fears of the community is, after the Pivotal demise, if Groovy is a dying language. It’s not. It has never been so widely used. The move to Apache Software Foundation drove a lot of attention and brought us many more contributors. But the community has to realize what the problems with Groovy are, and it has to face them: the introduction of the static compiler was too late. IDE support is important. Java 9 support is going to be super important. If you love your language, contribute. Help it. Help yourselves. The future of Groovy must be in your hands. I can’t recall how many times I told this, since I joined VMware, a few years ago, to develop Groovy. In every talk I give, I’m always telling how important it is that you contribute. Jetbrains is not going to write Groovy Eclipse for you.

And I would like to finish with one word: if people move from Groovy to Kotlin, is it really a problem? Isn’t any technology inspired by another? Aren’t we, developers, always rebuilding the same things, but improving them, learning lessons from the past? Is Kotlin a better Groovy? I don’t have the answer yet. Maybe it is. Maybe not. Today Groovy remains greatly superior in terms of scripting, DSL support, but it comes with a price that Gradle doesn’t want to pay. And let’s not forget the original community of Groovy: a dynamic language for the JVM. There are still lots of people who like this aspect of the language (and I do too, typically when I write Groovy scripts in place of bash scripts, I don’t care about types). It’s compile time metaprogramming features also make it incredibly powerful. Modern Groovy definitely doesn’t deserve its "bad press". Would you compare Java 8 with Java 1? No. So don’t compare Groovy 2.4 with Groovy 1 either. Reputation should change, and you can help there too.

This leads me to what I should do. And there, I’m a bit lost, to be honest. I work for a company that embraced Groovy, that is now embracing Kotlin. I love my job, I love working with Gradle, I love Groovy, and I quite enjoy Kotlin. I’m a passionate developer. I just want to continue having fun. But if you think that as such, I’m not a good representative of the Groovy community anymore, maybe I should step off from the Groovy project. I would hate that, but I’ve kind of been hurt by the bad comments we (Gradle) received from some members of the Groovy community. I don’t want to fall into a language war, I don’t care about this. I care about users. What I love to do is helping people, period.

I would like to finish this post with a thought about what I’m going to do, as a Gradle developer, for you, Groovy users. In particular, I am convinced that the success of Gradle is largely due to its Groovy DSL, despite its problems. The fact that it’s simple, easy to read, is super important. I joined the Groovy project because I was using Groovy as a DSL platform in a natural language processing context. Groovy is super powerful for this. And I learnt a lot in terms of DSL design. In particular, I will try to make sure that it doesn’t become a Kotlin API. What I mean by that is that I think we should elevate from a Groovy DSL to a Gradle language. And this language is meant at describing builds. And our users are not Kotlin developers. Most of them are not Groovy developers either. They are, as I described earlier, from different horizons. And I would hate if a user would have to understand concepts like generics or type inference to write a build script. That would be horribly wrong. A build author should understand how to model an application, not what is a type, what is an extension method, or generic return type inference. It’s different for plugin authors, but for a build author, it’s super important. So I will try to make sure that Kotlin scripting support improves, even if it means that it would go even closer to what Groovy supports. I would do this not because I want Groovy to die, I don’t (and it wouldn’t help my royalties for Groovy in Action 2 ;)), but it would help users or Gradle. That’s what I care most about, just like I care about what Groovy users want when I work on the Groovy project.

As for talking about Gradle, Groovy and its future, I’ll be a GR8Conf next week, I’d be happy to answer you in person there too!

Keep on Groovying!

A week fighting a PermGen leak

Sat, 29 Aug 2015 00:00:00 +0200

A challenge at Gradle

A new job

This is my first blog post after 4 months being a full time employee of Gradle Inc.! I am pretty excited by this job, even though so far it didn’t give me much time to contribute to Apache Groovy (but we did manage to release 2.4.4 though). One of the reasons I joined the company was because I love technical challenges. And Gradle has a lot of them, with an incredible team of smart people working together to make software automation better. This week, I worked on my first true challenge, and I must confess that I miserably failed :-) This is a long post, and anyone who ever fought against the infamous "PermGen space error" in their application is going to understand why…

CodeNarc as the source of a leak?

The first suspicious piece of code that draw our attention was CodeNarc. CodeNarc is a source code quality analysis tool for Groovy, which is used by a lot of Groovy developers, including in Gradle itself (since Gradle intensively makes use of Groovy). CodeNarc can be seen as the equivalent of FindBugs for Groovy code. And problems seemed to start with an upgrade of the Gradle CodeNarc plugin to use CodeNarc 0.23. We actually saw reports like this one in the forums or this other one in GitHub but thought that the PermGen error was just a consequence of CodeNarc including more rules: rules are written in Groovy, so compiled down to classes and classes eat PermGen. So increasing the PermGen space was enough, and it did actually solve the error. Problem solved. Or not. The riddle only started for me with a seemingly insignificant question on our internal mailing lists: "Can some investigate why our build sometimes fails with a PermGen space error?", and I volunteered.

Interestingly, I had just finished pushing an upgrade of Gradle to Groovy 2.4.4 on our master branch, and I had noticed that I had to increase the PermGen space too. At first, I naively thought that it was also required because Groovy 2.4 consumed more memory, but I was wrong. I should have known, because before joining Gradle, I had actually worked on Groovy for Android, and a consequence of this work was that Groovy 2.4 had a reduced memory footprint: we generate less bytecode, which directly relates to a reduced PermGen space usage. So why on earth would Groovy 2.4 require more memory? And what is the relation with the CodeNarc plugin? Actually this plugin works in a Gradle version that uses Groovy 2.3.10, so why would there be a relation between the two?

In such a case, your best friend is a profiler. But as I will explain here, it can also lead you to wrong tracks. Be careful. The second best friend is the JVM options -XX:+TraceClassLoading and +XX:+TraceClassUnloading. I also used -XX:SoftRefLRUPolicyMSPerMB=0, an option that I had no idea it existed before my friend David Gageot told me. Basically, it will force the garbage collector to agressively collect all soft references, which is very useful to understand, in combination with the 2 other options, from which classloader we are leaking memory.

The first wrong track was actually thinking that CodeNarc was the source of the problem. I wrote an email on the gradle-dev list, explaining my findings, and I had indeed found a lot of classes from CodeNarc were not unloaded. Before I go further, let’s explain how the JVM is supposed to behave with regards to classes in Java 7. We all know that objects are garbage collected, but for a lot of people, classes are not. That’s why we have the PermGen space (which has disappeared in Java 8 but that’s another story): this segregated space of the JVM memory is used to store classes. And in Java, a class is loaded by a classloader. There is a strong reference between a class and its classloader. But what the JVM is able to do is actually simple: if there’s no instance of the class which is strongly reachable and that the classloader is neither strongly reachable, then both the class and the classloader can be unloaded. This means that PermGen can be recovered, and it is pretty useful, especially for a language like Groovy which can generate a lot of classes at runtime.

In Gradle, and particularily in the Gradle CodeNarc plugin, CodeNarc is executed through an Ant task, which spawns its own isolated classloader, containing both the CodeNarc and Groovy classes. So when the plugin execution is finished, if we do not keep track of the classloader, classes should be garbage collected. So a good candidate for the memory leak was actually the IsolatedAntBuilder that Gradle uses to execute the Ant task. And guess what? There is such a leak, because the DefaultIsolatedAntBuilder performs classloader caching! That was also discovered by my colleague Sterling, who immediately spot that: while we do cache the classloaders, keeping a strong reference on them, we don’t have any code to release the classloader in case of memory pressure. Conclusion, we’ve found the memory leak, hurray! And it has nothing to do with CodeNarc or Apache Groovy, pfiew!

So I immediately tried to disable the cache, which turned to be pretty trivial. Run the build again and… another PermGen space error. No CodeNarc classes unloaded, no Groovy classes unloaded. Wow. So the problem wasn’t solved, first "oh my!" moment of the week: there was another leak.

One test I did, then, is to totally comment out the code that, in the ant builder code, performed the definition of the CodeNarc task. Eventually, there was only that code left in the CodeNarc plugin:

    @TaskAction
    void run() {
        logging.captureStandardOutput(LogLevel.INFO)
        def classpath = new DefaultClassPath(getCodenarcClasspath())
        antBuilder.withClasspath(classpath.asFiles).execute {
            // ... thou shalt not leak!
        }
    }

I executed the code again, and there was definitely a leak: after several loops, a PermGen error occurred. CodeNarc was ruled out as the source of the leak. After some hours of trials, study of memory snapshots, I eventually came out with a piece of code that reproduces the problem independently of Gradle:

int i = 0;
try {
    while (true) {
        i++;
	URLClassLoader loader = new URLClassLoader(
            new URL[]{new File(GROOVY_JAR).toURI().toURL()},
            ClassLoader.getSystemClassLoader().getParent());
        Class system = loader.loadClass("groovy.lang.GroovySystem");
        system.getDeclaredMethod("getMetaClassRegistry").invoke(null);
        loader.close();
    }
} catch (OutOfMemoryError e) {
    System.err.println("Failed after " + i + " loadings");
}

As you can see, the code is very simple: it creates a new isolated classloader, which only contains Groovy on classpath. Then it invokes the creation of the Groovy runtime, by asking the metaclass registry, then it closes the classloader. On my JVM, after about 40 runs, the code fails with a PermGen space error: despite the fact that no class, no object is kept out of the classloader, the Groovy runtime is not unloaded, leading to a memory leak. The key point here is that I had noticed some oddities during my hours of debugging: a class, named ClassInfo, was at the center of those oddities.

In particular, although YourKit (the profiler I was using) was telling me that all classes, all classloaders were weakly or softly reachable (btw, it’s really a pity that YourKit doesn’t show them separately, that is, weakly referenced objects from softly referenced ones), the classes were not garbage collected. Also strangely, some classes appeared as GC roots, meaning they could be collected, but they weren’t! And when I navigated through some of the duplicate classes I was seeing, ClassInfo was present, as a value of a map entry of class value. Here we are, I had found the real source of the leak. Something had changed in Groovy. And the fact that CodeNarc was leaking since 0.23 was just a side effect of upgrading its dependency to Groovy 2.4! So despite Gradle was using Groovy 2.3.10, CodeNarc, by default, was using a more recent version of Groovy. That doesn’t explain why it leaks yet, but we now know who is responsible: Groovy.

ClassValue, friend or foe?

Since Groovy 2.4, Groovy uses a new mechanism for storing its runtime metadata in case you run on JDK 7 or later: ClassValue. ClassValue allows storing information on the class level. Typically, a language like Groovy would use it to store the metaclass of a class. In practice, Groovy doesn’t directly store the metaclass here, but a higher level concept called ClassInfo, which in turns gives access to the metaclass of a class.

Before Groovy 2.4, all that information was stored directly in the ClassInfo class itself, through a private static field. ClassInfo is therefore the global entry point for accessing runtime information about a class. While Groovy 2.4 still uses ClassInfo as an entry point, there are 2 possible storage mechanisms, based on the underlying JDK. If ClassValue is available; which is the case for any JDK 7+, then it is used, otherwise we fallback on the old mechanism. ClassValue is supposed to be more efficient and more direct. Of course, in any case (old and new storage mechanism), it is memory sensitive: in case a class is not available anymore, its ClassInfo is removed. What ClassValue storage provides is basically the same as ThreadLocal, but at the Class level instead of the Thread level. Users are allowed to store information here, but one should be aware that as thread locals, if you start using it, you may face memory leaks if you don’t use it properly.

That’s for the theory, let’s see how in practice this change led to a giant memory leak in the Gradle build.

The theory is that ClassValue should behave like ThreadLocal. That is, the entries stored in the internal map of the Class class, should be garbage collected when the referent is not strongly referenced anymore. This behavior is however not what the JVM does. It was confirmed to me by Charles Nutter (JRuby) a few hours later: although we all expected the JVM to collect the unreachable, it does not.

@inikem @cedricchampeau I talked with @haupz at JVMLS and he agrees that ClassValue *should* work like ThreadLocal wrt dereferencing.
— Charles Nutter (@headius) 27 Août 2015

Uh. Second "oh my!" moment of the week. I had now a candidate (Groovy) and a reason (ClassInfo leaking). However, it doesn’t explain by itself why the class loader is not garbage collected: if Groovy stores information on classes, it’s ok, as long as the classes to which it writes some metadata are from the classloader, or any child classloader, of the Groovy runtime itself. Everything would be self-contained, meaning we would have a graph of objects that do not leak outside of the isolated classloader. However… Groovy uses Strings, integers, List, … all coming from the system classloader. And that is the main difference with the old metadata storage mechanism: the old one only referenced classes from the system classloader. With ClassValue, we are modifying classes from the system classloader too! That is, the String class, for example, contains in its class value map, information from the Groovy runtime! The famous ClassInfo instance is present there! There we are! We leaked a ClassInfo instance into the system classloader! So what happens is that when we are done with our "isolated" Groovy runtime, we think it should unload because nothing references any object or class from that classloader. However, the Groovy runtime did update classes from the system classloader, and it started leaking into it! The ghost in the shell! Groovy is spoiling everywhere…

So far so good, I had the explanation, I could write a workaround: let’s iterate over all those classes that Groovy updated, remove the ClassInfo, and we’re done. I wrote that code, and it turned out to be a bit ugly, but… it worked! Here is, for information, the cleanup code:

    static void removeClassFromGlobalClassSet(Class<?> classInfoClass) throws Exception {
        Field globalClassValueField = classInfoClass.getDeclaredField("globalClassValue");
        globalClassValueField.setAccessible(true);
        Object globalClassValue = globalClassValueField.get(null);
        Method removeFromGlobalClassValue = globalClassValueField.getType().getDeclaredMethod("remove", Class.class);
        removeFromGlobalClassValue.setAccessible(true);

        Field globalClassSetField = classInfoClass.getDeclaredField("globalClassSet");
        globalClassSetField.setAccessible(true);
        Object globalClassSet = globalClassSetField.get(null);
        globalClassSetField = globalClassSet.getClass().getDeclaredField("items");
        globalClassSetField.setAccessible(true);
        Object globalClassSetItems = globalClassSetField.get(globalClassSet);

        Field clazzField = classInfoClass.getDeclaredField("klazz");
        clazzField.setAccessible(true);


        Iterator it = (Iterator) globalClassSetItems.getClass().getDeclaredMethod("iterator").invoke(globalClassSetItems);

        while (it.hasNext()) {
            Object classInfo = it.next();
            Object clazz = clazzField.get(`ClassInfo`);
            removeFromGlobalClassValue.invoke(globalClassValue, clazz);
        }

    }

After executing that code, no ClassInfo instance was leaking anymore into the system classloader, and the runtime could be shutdown properly. The garbage collector did its job, and yay! I’m so happy, I’ll be able to sleep soon! That was tuesday night. And that night, I thought I had found the solution.

Memory sensitive classloader caching

So wednesday, I spent the day trying to implement the same strategy inside Gradle. More precisely, inside the IsolatedAntBuilder thing I told you. I implemented the code, launched my test again and hurray! it worked! My test passed! No more PermGen space error! So all I had to do, now, was to reactivate classloader caching, otherwise, we would loose a feature that is important performance wise.

So I reactivated the cache, and boom! That time, the Gradle build did not fail with a PermGen error, but with very strange errors like this one:

groovy.lang.MissingMethodException: No signature of method: java.lang.Integer.plus() is applicable for argument types: (java.lang.Integer) values: [0]
>  Possible solutions: plus(java.lang.String), plus(java.lang.Character), abs(), use([Ljava.lang.Object;), split(groovy.lang.Closure), minus(java.lang.Character)

Mmmmmm… 3rd "oh my!" moment of the week. I understood what I had just done. By clearing the ClassInfo stuff from the classloader, I had effectively shutdown the Groovy runtime that was initiated in that cached classloader. So when some code was trying to reuse the runtime from that cached classloader, since I had disabled it, it was failing! And there’s no option to reinitialize the Groovy runtime. It’s just not doable, because everything happens in static initializers (private final fields, …). So unless the JVM had an option to allow to re-execute the static initializers of a class (and who knows what oddities it would lead to), I had no luck.

That’s about when I told my mates at Gradle "I think we have to choose between caching and leaking memory". But the night came, and I actually had an idea. I could implement a memory sensitive cache: by writing a smart cache structure with appropriate SoftReferences and reference queues, I would be able to execute the shutdown code only when I know that the GC is trying to reclaim memory. The idea is simple: we have a map, which key is a SoftRefence<String>, and the value is our cached classloader. The String represents the classpath that we are caching for the classloader.

Now imagine that the GC is out of memory. The semantics of SoftReference are clear: before throwing an OutOfMemoryError, the JVM will do its best and clear all soft references. Doing so, using a custom reference queue, we can be notified that the reference is cleared. Then, we can execute the Groovy runtime shutdown code, which will in turn make the ClassLoader collectible.

Honestly I was pretty happy with my implementation. I executed the code and it worked! Caching was working until the GC tried to reclaim memory, then I saw my shutdow code executed, memory reclaimed and green tests. Woooo!!! I had eventually knocked that memory leak down! Ha ha!

Then I remembered that my colleague Sterling had a test which involved a loop in an integration test. To make sure I had really fixed the leak, I asked him to tell me how he did that. The code was very simple, just involving a loop thanks to @Unroll in a Spock specification. I did it and… PermGen error showed up again. WAT?!

That was the fourth "oh my!" moment. And not the last one. I really then spent hours in modifying my caching code, refactored my code to add more complicated memory leak strategies, seeing that there were still thread locals, clearing them explicitly, adding a memory leak strategy for Ant itself, for the Java Beans introspector, … None of my attempts worked. In the end, it always failed. But there was always one mystery: I saw that the Groovy classes were unloading. But the Ant classes were not… And the rest, I should have discovered that much sooner. But when you have so many potential source leaks, that are much more evident, it’s so hard to figure out.

In particular, one thing would have made things much easier to discover. In YourKit, you can see that there are duplicate classes. Classes that have the same name, but come from different class loaders. However, there’s nothing that will show you those duplicates. You have to find them yourself. And in the end, when in the dump you see an instance of that class, all you can see is that it is an instance of ClassInfo. Nothing, visually, tells you that the instance of ClassInfo that you are seeing actually comes from a different classloader from the one you are seeing just next to it. A bit of color, for example, would help…. And it would have helped me seeing that some ClassInfo elements that I was seeing in the classes from Ant didn’t come from the "disposable" Groovy runtime… No. They were coming from… the Gradle runtime itself!

Where it all ends

Ok, That was the last "oh my!" moment of the week. The one that killed all my hopes. And to understand the problem, I now have to explain to you how IsolatedAntBuilder works. It’s a very small, yet very smart and practical piece of code. Maybe too smart.

Gradle, as a core feature, lets you execute Ant tasks thanks to code inherited from the Groovy codebase itself: AntBuilder. It’s a very elegant piece of code, that lets you write things like:

task check << {
    ant.taskdef(name: 'pmd',
                classname: 'net.sourceforge.pmd.ant.PMDTask',
                classpath: configurations.pmd.asPath)
    ant.pmd(shortFilenames: 'true',
            failonruleviolation: 'true',
            rulesetfiles: file('pmd-rules.xml').toURI().toString()) {
        formatter(type: 'text', toConsole: 'true')
        fileset(dir: 'src')
    }
}

While this works, there’s actually a lot involved behind that. Including classloader magic. In particular, in the example above, we create a task definition in Ant, which uses a classpath defined in Gradle. "ant" here is a global object which is shared accross the build, but it is possible to avoid the classes from the Ant tasks to be mixed with the Gradle classpath itself by using antBuilder instead. That’s what the CodeNarc plugin does:

antBuilder.withClasspath(classpath.asFiles).execute {
}

means "Gradle, please, create an isolated classloader for me, that will contain the classpath only necessary for CodeNarc, and execute that Ant task with it". It seems very trivial, but there is a problem. The code that you see here is found in a Gradle script. It means that the "antBuilder" object that you are seeing here comes from Gradle. It is our IsolatedAntBuilder instance. When we call "withClasspath", a new instance of IsolatedAntBuilder will be created, with an isolated classloader corresponding to the supplied classpath. Then calling execute with a closure that lets you configure the ant task using the Groovy AntBuilder syntax.

So the "Closure" class that we are seeing here comes from Gradle. Then, we have a classloader which contains the Ant runtime, and a "bridge" class, written in Groovy, called "AntBuilderDelegate", which has one responsibility: when the code of the Ant builder is going to be executed, it is likely that the version of Groovy which will be found on classpath will be different from the one that Gradle uses. That is exactly what happens with CodeNarc: Gradle 2.6 uses Groovy 2.3.10, but the CodeNarc plugin executes with Groovy 2.4.1, so the Ant task works with a different "Closure" class than the one that Gradle has. We will really have two distinct "Closure" classes here, and "AntBuilderDelegate" is responsible for filling the gap: when the Ant configuration code, which will use AntBuilder from the Ant classpath, is going to be executed, it will be calling AntBuilderDelegate instead of directly the Closure code. And that code will intercept the missing methods in order to "reroute" them to the builder. You don’t have to understand that in detail, it’s not really the point here, but it is important to understand that this "AntBuilderDelegate" class is instantiated… in Gradle, using the Gradle classloader.

Now you may see it coming. I told you I had upgraded Gradle to use Groovy 2.4.4. So what does it mean? Gradle now uses Groovy with ClassValue. And what is the problem with ClassValue? All classes "touched" by Groovy will have them "polluted" with necessary metadata information. So when we create an instance of "AntBuilderDelegate", we’re doing that using the Groovy 2.4 runtime from Gradle, which comes with its own ClassInfo. And that delegate references and AntBuilder which is instantiated using the Ant classloader, with a different Groovy runtime, having its own ClassInfo. So what I had found earlier was that the ClassInfo from the Groovy "Ant" runtime was leaking into Gradle. But I hadn’t realized that the opposite was also true! By bridging the runtimes, we were leaking Groovy "Gradle" into the isolated classloader, through ClassValue!

So, what happened, is that the Groovy classes from the isolated classpath were garbage collected, because no ClassInfo from Gradle leaked into them. However, the Ant classes were touched. And they were NOT collectible then.

And this is were I stopped. Because if I found a way to "unload" ClassInfo from the isolated classpath and the touched classes from the system classloader, I haven’t found a way to do the same for the ClassInfo instances that leak into the Ant runtime… Of course I tried a "brute force" thing, by removing all ClassInfo from those classes, but as you understood, it’s a desperate attempt: it’s equivalent to shutting down the runtime. And then, it totally breaks subsequent calls in the Gradle build, we’ve just broken the Groovy runtime from Gradle…

To add some confusion to the problem, I think it’s now a good time to explain that I actually simplified the isolated ant builder classloader hierarchy. There are actually (at least) 3 classloaders involved:

The classloader from Gradle, which loads the Gradle runtime, the IsolatedAntBuilder instance and also the AntBuilderDelegate instance
A classloader for the Ant runtime, which is isolated from Gradle, apart from logging classes necessary for Gradle to be able to capture the output. This classloader is per classpath, and is the one which is cached.
A classloader that is filtering some classes from the Gradle runtime classloader to make them available. This is what the bridging Ant builder uses. This classloader is shared among all isolated ant builder instances.

So when I say that something leaks, it can leak to any of those classloaders, and any of the parent loaders…

That’s were I felt desperate. After a week fighting those memory leaks, and so many "ah ah, got you!" moments, I was in front of a wall. Basically, while in the first case (isolated ClassInfo leaking into Gradle), I know I can totally clean all the ClassInfo references because I know I can shutdown the Groovy runtime, in the second case (Gradle ClassInfo leaking), I basically have no idea that a class comes from the isolated classloader or not. So it’s hard to say if you should remove the ClassInfo or not. I am currently experimenting a brute force "try to determine if a class belongs in the class loader hierarchy", but it is weak (ah ah!) because I need to know about several potential class loader types.

What’s next?

So, what can we do next? One has to remember that fixing Groovy is not the ultimate solution because Gradle uses Groovy internally, but the various tasks can very well use a different Groovy version which is beyond our control.

rollback Groovy in Gradle to use Groovy 2.3.10. It would avoid the Groovy classes from Gradle to leak into the Isolated classloaders, but is also unfortunate given the improvements that this version provides. Also, those who write Groovy applications for Android use Groovy 2.4+…
CodeNarc would still use Groovy 2.4+, we could downgrade it too. However, if people rely on features of Groovy 2.4+, they just have no choice, so we would still have the problem.
Use smarter techniques like instrumentation to track the leakages of ClassInfo, record them, and revert when we’re done. It’s doable, but it’s a huge amount of work, and relying on instrumentation for Gradle would be very bad for performance.
Update Gradle to use FilteringClassLoader everywhere, including in its main process, to prevent ClassValue to be found. This would work because without that class, Groovy wouldn’t use ClassValue to store the metadata and fall back to the old mechanism.
Wait for a fix in Groovy. Jochen is already working on that, but we know that the old mechanism isn’t perfect either, and has memory leaks too. That was one of the reasons to migrate to ClassValue.
Wait for a fix of the JVM. That’s beyond our control.
Increase the PermGen space for builds that use the code quality plugins, which internally use AntBuilder. It’s what we do today. It works, but it’s just hiding the problem. And we have to explain to our users to do it too.
Some smart people come with a smart solution, that’s why I wrote this too. During that week, I got help from lots of people, including Jochen "Groovy" Theodorou, Henri "EasyMock" Tremblay, David Gageot or Nikita Salnikov from Plumbr, thank you guys!

By the way, if you wonder, the same problem exist in JDK 8 too, it’s just not visible immediately because of the metaspace that appeared to replace the PermGen space.

Now, it was fun writing this "post-mortem", I hope it wasn’t too obscure, it helped me a lot because I had so many "got it" and "oh noes!" moments that I felt it was very interesting to share this story with you. And if you like technical challenges, do not forget that Gradle is hiring!

Update

After writing this post, I made great progress, and I did manage to get rid of the leak, but I also discovered that the leak happens with the various GroovyCompile tasks… Anyway you can follow my progress on this branch.

Improved sandboxing of Groovy scripts

Fri, 27 Mar 2015 00:00:00 +0100

One of the most current uses cases of Groovy is scripting. Groovy makes it very easy to execute code dynamically, at runtime. Depending on the application, scripts can be found in multiple sources: file system, databases, remote services, … but more importantly, the designer of the application which executes scripts is not necessarily the one writing those scripts. Moreover, the scripts might run in a constrained environment (limited memory, file descriptors, time, …) or your you simply don’t want to allow users to access the full capabilities of the language from a script.

What you will learn in this post

why Groovy is a good fit to write internal DSLs
what it implies in terms of security in your applications
how you can customize compilation to improve the DSL
the meaning of SecureASTCustomizer
what are type checking extensions
how you can rely on type checking extensions to offer proper sandboxing

For example, imagine that you want to offer users the ability to evaluate mathematical expressions. One option would be to implement your own internal DSL, create a parser and eventually an interpreter for those expressions. Obviously this involves a bit of work, but if in the end you need to improve performance, for example by generating bytecode for the expressions instead of evaluating them, or introduce caching of those runtime generated classes, then Groovy is probably a very good option.

There are lots of options available, described in the documentation but the most simple example is just using the Eval class:

Example.java

int sum = (Integer) Eval.me("1+1");

The code 1+1 is parsed, compiled to bytecode, loaded and eventually executed by Groovy at runtime. Of course in this example the code is very simple, and you will want to add parameters, but the idea is that the code which is executed here is arbitrary. This is probably not what you want. For a calculator, you want to allow expressions like:

1+1
x+y
1+(2*x)**y
cos(alpha)*r
v=1+x

but certainly not

println 'Hello'
(0..100).each { println 'Blah' }
Pong p = new Pong()
println(new File('/etc/passwd').text)
System.exit(-1)
Eval.me('System.exit(-1)') // a script within a script!

This is where things start to become complicated, and where we start seeing actually multiple needs:

restricting the grammar of the language to a subset of its capabilities
preventing users from executing unexpected code
preventing users from executing malicious code

The calculator example is pretty simple, but for more complex DSLs, people might actually start writing problematic code without noticing, especially if the DSL is suffiently elegant to be used by non developers.

I was in this situation a few years ago, when I designed an engine that used Groovy "scripts" written by linguists. One of the problems was that they could unintentionally create infinite loops, for example. Code was executing on the server, and then you had a thread eating 100% of the CPU and had no choice but restart the application server so I had to find a way to mitigate that problem without compromising the DSL nor the tooling or the performance of the application.

Actually, lots of people have similar needs. During the past 4 years, I spoke to tons of users who had a similar question: How can I prevent users from doing bad things in Groovy scripts?

Compilation customizers

At that time, I had implemented my own solution, but I knew that other people also had implemented similar ones. In the end, Guillaume Laforge suggested to me that I wrote something that would help fixing those issues and make into Groovy core. This happened in Groovy 1.8.0 with compilation customizers.

Compilation customizers are a set of classes that are aimed at tweaking the compilation of Groovy scripts. You can write your own customizer but Groovy ships with:

an import customizer, which aims at adding imports transparently to scripts, so that users do not have to add "import" statements
an AST (Abstract Syntax Tree) transformation customizer, which allows to add AST transformations transparently to scripts
a secure AST customizer, which aims at restricting the grammar and syntactical constructs of the language

The AST transformation customizer allowed me to solve the infinite loop issue, by applying the @ThreadInterrupt transformation, but the SecureASTCustomizer is probably the one which has been the most misinterpreted of the whole.

I must apologize for this. Back then, I had no better name in mind. The important part of SecureASTCustomizer is AST. It was aimed at restricting access to some features of the AST. The "secure" part is actually not a good name at all, and I will illustrate why. You can even find a blog post from Kohsuke Kawagushi, of Jenkins fame, named Groovy SecureASTCustomizer is harmful[https://kohsuke.org/2012/04/27/groovy-secureastcustomizer-is-harmful/]. It is very true. The SecureASTCustomizer has never been designed with sandboxing in mind. It was designed to restrict the language at compile time, not runtime. So a much better name, in retrospect, would have been GrammarCustomizer. But as you’re certainly aware, there are two hard things in computer science: cache invalidation, naming things and off by one errors.

So imagine that you think of the secure AST customizer as a way of securing your script, and that you want to use this to prevent a user from calling System.exit from a script. The documentation says that you can prevent calls on some specific receivers by defining either a blacklist or a whitelist. In terms of securing something, I would always recommand to use a whitelist, that is to say to list explicitly what is allowed, rather than a blacklist, saying what is disallowed. The reason is that hackers always think of things you don’t, so let’s illustrate this.

Here is how a naive "sandbox" script engine could be configured using the SecureASTCustomizer. I am writing the examples of configuration in Java, even though I could write them in Groovy, just to make the difference between the integration code and the scripts clear.

public class Sandbox {
    public static void main(String[] args)  {
        CompilerConfiguration conf = new CompilerConfiguration();				(1)
        SecureASTCustomizer customizer = new SecureASTCustomizer();				(2)
        customizer.setReceiversBlackList(Arrays.asList(System.class.getName()));		(3)
        conf.addCompilationCustomizers(customizer);						(4)
        GroovyShell shell = new GroovyShell(conf);						(5)
        Object v = shell.evaluate("System.exit(-1)");						(6)
        System.out.println("Result = " +v);							(7)
    }
}

1	create a compiler configuration
2	create a secure AST customizer
3	declare that the `System` class is blacklisted as the receiver of method calls
4	add the customizer to the compiler configuration
5	associate the configuration to the script shell, that is, try to create a sandbox
6	execute a nasty script
7	print the result of the execution of the script

If you run this class, when the script is executed, it will throw an error:

General error during canonicalization: Method calls not allowed on [java.lang.System]
java.lang.SecurityException: Method calls not allowed on [java.lang.System]

This is the result of the application of the secure AST customizer, which prevents the execution of methods on the System class. Success! Now we have secured our script! Oh wait…

SecureASTCustomizer pwned!

Secure you say? So what if I do:

def c = System
c.exit(-1)

Execute again and you will see that the program exits without error and without printing the result. The return code of the process is -1, which indicates that the user script has been executed! What happened? Basically, at compile time, the secure AST customizer is not able to recognize that c.exit is a call on System, because it works at the AST level! It analyzes a method call, and in this case the method call is c.exit(-1), then gets the receiver and checks if the receiver is in the whitelist (or blacklist). In this case, the receiver is c and this variable is declared with def, which is equivalent to declaring it as Object, so it will think that c is Object, not System!

Actually there are many ways to workaround the various configurations that you can make on the secure AST customizer. Just for fun, a few of them:

((Object)System).exit(-1)
Class.forName('java.lang.System').exit(-1)
('java.lang.System' as Class).exit(-1)

import static java.lang.System.exit
exit(-1)

and there are much more options. The dynamic nature of Groovy just makes it impossible to resolve those cases at compile time. There are solutions though. One option is to rely on the JVM standard security manager. However this is a system wide solution which is often considered as a hammer. But it also doesn’t really work for all cases, for example you might not want to prevent creation of files, but only reads for example…

This limitation - or should I say frustration for lots of us - led several people to create a solution based on runtime checks. Runtime checks do not suffer the same problem, because you will have for example the actual receiver type of a message before checking if a method call is allowed or not. In particular, those implementations are of particular interest:

SecureScript by Jim White
Groovy Sandbox by Kohsuke Kawagushi
Groovy Sandbox by Simon Temple

However none of those implementations is totally secure or reliable. For example, the version by Kohsuke relies on hacking the internal implementation of call site caching. The problem is that it is not compatible with the invokedynamic version of Groovy, and those internal classes are going to be removed in future versions of Groovy. The version by Simon, on the other hand, relies on AST transformations but misses a lot of possible hacks.

As a result, with friends of mine Corinne Krych, Fabrice Matrat and Sébastien Blanc, we decided to create a new runtime sandboxing mechanism that would not have the issues of those projects. We started implementing this during a hackathon in Nice, and we gave a talk about this last year at the Greach conference. It relies on AST transformations and heavily rewrites the code in order to perform a check before each method call, property access, increment of variable, binary expression, … The implementation is still incomplete, and not much work has been done because I realized there was still a problem in case of methods or properties called on "implicit this", like in builders for example:

xml {
   cars {				 // cars is a method call on an implicit this: "this".cars(...)
     car(make:'Renault', model: 'Clio')
   }
}

As of today I still didn’t find a way to properly handle this because of the design of the meta-object protocol in Groovy, that here relies on the fact that a receiver throws an exception when the method is not found before trying another receiver. In short, it means that you cannot know the type of the receiver before the method is actually called. And if it is called, it’s already too late…

Until earlier this year I had still no perfect solution to this problem, in case the script being executed is using the dynamic features of the language. But now has come the time to explain how you can significantly improve the situation if you are ready to loose some of the dynamism of the language.

Type checking

Let’s come back to the root problem of the SecureASTCustomizer: it works on the abstract syntax tree and has no knowledge of the concrete types of the receivers of messages. But since Groovy 2, Groovy has optional compilation, and in Groovy 2.1, we added type checking extensions.

Type checking extensions are very powerful: they allow the designer of a Groovy DSL to help the compiler infer types, but it also lets you throw compilation errors when normally it should not. Type checking extensions are even used internally in Groovy to support the static compiler, for example to implement traits or the markup template engine.

What if, instead of relying on the information available after parsing, we could rely on information from the type checker? Take the following code that our hacker tried to write:

((Object)System).exit(-1)

If you activate type checking, this code would not compile:

1 compilation error:

[Static type checking] - Cannot find matching method java.lang.Object#exit(java.lang.Integer). Please check if the declared type is right and if the method exists.

So this code would not compile anymore. But what if the code is:

def c = System
c.exit(-1)

You can verify that this passes type checking by wrapping the code into a method and running the script with the groovy command line tool:

@groovy.transform.TypeChecked // or even @CompileStatic
void foo() {
  def c = System
  c.exit(-1)
}
foo()

Then the type checker will recognize that the exit method is called on the System class and is valid. It will not help us there. But what we know, if this code passes type checking, is that the compiler recognized the call on the System receiver. The idea, then, is to rely on a type checking extension to disallow the call.

A simple type checking extension

Before we dig into the details about sandboxing, let’s try to "secure" our script using a traditional type checking extension. Registering a type checking extension is easy: just set the extensions parameter of the @TypeChecked annotation (or @CompileStatic if you want to use static compilation):

@TypeChecked(extensions=['SecureExtension1.groovy'])
void foo() {
  def c = System
  c.exit(-1)
}
foo()

The extension will be searched on classpath in source form (there’s an option to have precompiled type checking extensions but this is beyond the scope of this blog post):

SecureExtension1.groovy

onMethodSelection { expr, methodNode ->					(1)
   if (methodNode.declaringClass.name=='java.lang.System') {		(2)
      addStaticTypeError("Method call is not allowed!", expr)		(3)
   }
}

1	when the type checker selects the target method of a call
2	then if the selected method belongs to the `System` class
3	make the type checker throw an error

That’s really all needed. Now execute the code again, and you will see that there’s a compile time error!

/home/cchampeau/tmp/securetest.groovy: 6: [Static type checking] - Method call is not allowed!
 @ line 6, column 3.
     c.exit(-1)
     ^

1 error

So this time, thanks to the type checker, c is really recognized as an instance of class System and we can really disallow the call. This is a very simple example, but it doesn’t really go as far as what we can do with the secure AST customizer in terms of configuration. The extension that we wrote has hardcoded checks, but it would probably be nicer if we could configure it. So let’s start working with a bit more complex example.

Imagine that your application computes a score for a document and that you allow the users to customize the score. Then your DSL:

will expose (at least) a variable named score
will allow the user to perform mathematical operations (including calling methods like cos, abs, …)
should disallow all other method calls

An example of user script would be:

abs(cos(1+score))

Such a DSL is easy to setup. It’s a variant of the one we defined earlier:

Sandbox.java

CompilerConfiguration conf = new CompilerConfiguration();
ImportCustomizer customizer = new ImportCustomizer();
customizer.addStaticStars("java.lang.Math");                        (1)
conf.addCompilationCustomizers(customizer);
Binding binding = new Binding();
binding.setVariable("score", 2.0d);                                 (2)
GroovyShell shell = new GroovyShell(binding,conf);
Double userScore = (Double) shell.evaluate("abs(cos(1+score))");    (3)
System.out.println("userScore = " + userScore);

1	add an import customizer that will add `import static java.lang.Math.*` to all scripts
2	make the `score` variable available to the script
3	execute the script

There are options to cache the scripts, instead of parsing and compiling them each time. Please check the documentation for more details.

So far, our script works, but nothing prevents a hacker from executing malicious code. Since we want to use type checking, I would recommand to use the @CompileStatic transformation transparently:

it will activate type checking on the script, and we will be able to perform additional checks thanks to the type checking extension
it will improve the performance of the script

Adding @CompileStatic transparently is easy. We just have to update the compiler configuration:

ASTTransformationCustomizer astcz = new ASTTransformationCustomizer(CompileStatic.class);
conf.addCompilationCustomizers(astcz);

Now if you try to execute the script again, you will face a compile time error:

Script1.groovy: 1: [Static type checking] - The variable [score] is undeclared.
 @ line 1, column 11.
   abs(cos(1+score))
             ^

Script1.groovy: 1: [Static type checking] - Cannot find matching method int#plus(java.lang.Object). Please check if the declared type is right and if the method exists.
 @ line 1, column 9.
   abs(cos(1+score))
           ^

2 errors

What happened? If you read the script from a "compiler" point of view, it doesn’t know anything about the "score" variable. You, as a developer, know that it’s a variable of type double, but the compiler cannot infer it. This is precisely what type checking extensions are designed for: you can provide additional information to the compiler, so that compilation passes. In this case, we will want to indicate that the score variable is of type double.

So we will slightly change the way we transparently add the @CompileStatic annotation:

ASTTransformationCustomizer astcz = new ASTTransformationCustomizer(
        singletonMap("extensions", singletonList("SecureExtension2.groovy")),
        CompileStatic.class);

This will "emulate" code annotated with @CompileStatic(extensions=['SecureExtension2.groovy']). Of course now we need to write the extension which will recognize the score variable:

SecureExtension2.groovy

unresolvedVariable { var ->			(1)
   if (var.name=='score') {			(2)
      return makeDynamic(var, double_TYPE)	(3)
   }
}

1	in case the type checker cannot resolve a variable
2	if the variable name is `score`
3	then instruct the compiler to resolve the variable dynamically, and that the type of the variable is `double`

You can find a complete description of the type checking extension DSL in https://docs.groovy-lang.org/latest/html/documentation/#type_checking_extensions[this section of the documentation], but you have here an example of _mixed mode compilation : the compiler is not able to resolve the score variable. You, as the designer of the DSL, know that the variable is in fact found in the binding, and is of the double, so the makeDynamic call is here to tell the compiler: "ok, don’t worry, I know what I am doing, this variable can be resolved dynamically and it will be of type `double`". That’s it!

First completed "secure" extension

Now it’s time to put this altogether. We wrote a type checking extension which is capable of preventing calls on System on one side, and we wrote another which is able to resolve the score variable on another. So if we combine both, we have a first, complete, securing type checking extension:

SecureExtension3.groovy

// disallow calls on System
onMethodSelection { expr, methodNode ->
    if (methodNode.declaringClass.name=='java.lang.System') {
        addStaticTypeError("Method call is not allowed!", expr)
    }
}

// resolve the score variable
unresolvedVariable { var ->
    if (var.name=='score') {
        return makeDynamic(var, double_TYPE)
    }
}

Don’t forget to update the configuration in your Java class to use the new type checking extension:

ASTTransformationCustomizer astcz = new ASTTransformationCustomizer(
        singletonMap("extensions", singletonList("SecureExtension3.groovy")),
	CompileStatic.class);

Execute the code again and it still works. Now, try to do:

abs(cos(1+score))
System.exit(-1)

And the script compilation will fail with:

Script1.groovy: 1: [Static type checking] - Method call is not allowed!
 @ line 1, column 19.
   abs(cos(1+score));System.exit(-1)
                     ^

1 error

Congratulations, you just wrote your first type checking extension that prevents the execution of malicious code!

Improving configuration of the extension

So far so good, we are able to prevent calls on System, but it is likely that we are going to discover new vulnerabilities, and that we will want to prevent execution of such code. So instead of hardcoding everything in the extension, we will try to make our extension generic and configurable. This is probably the trickiest thing to do, because there’s no direct way to provide context to a type checking extension. Our idea therefore relies on the (ugly) thread locals to pass configuration data to the type checker.

The first thing we’re going to do is to make the variable list configurable. Here is the code on the Java side of things:

Sandbox.java

public class Sandbox {
    public static final String VAR_TYPES = "sandboxing.variable.types";

    public static final ThreadLocal<Map<String, Object>> COMPILE_OPTIONS = new ThreadLocal<>();		(1)

    public static void main(String[] args) {
        CompilerConfiguration conf = new CompilerConfiguration();
        ImportCustomizer customizer = new ImportCustomizer();
        customizer.addStaticStars("java.lang.Math");
        ASTTransformationCustomizer astcz = new ASTTransformationCustomizer(
                singletonMap("extensions", singletonList("SecureExtension4.groovy")),			(2)
                CompileStatic.class);
        conf.addCompilationCustomizers(astcz);
        conf.addCompilationCustomizers(customizer);

        Binding binding = new Binding();
        binding.setVariable("score", 2.0d);
        try {
            Map<String,ClassNode> variableTypes = new HashMap<String, ClassNode>();			(3)
            variableTypes.put("score", ClassHelper.double_TYPE);					(4)
            Map<String,Object> options = new HashMap<String, Object>();					(5)
            options.put(VAR_TYPES, variableTypes);							(6)
            COMPILE_OPTIONS.set(options);								(7)
            GroovyShell shell = new GroovyShell(binding, conf);
            Double userScore = (Double) shell.evaluate("abs(cos(1+score));System.exit(-1)");
            System.out.println("userScore = " + userScore);
        } finally {
            COMPILE_OPTIONS.remove();									(8)
        }
    }
}

1	create a `ThreadLocal` that will hold the contextual configuration of the type checking extension
2	update the extension to `SecureExtension4.groovy`
3	`variableTypes` is a map variable name → variable type
4	so this is where we’re going to add the `score` variable declaration
5	`options` is the map that will store our type checking configuration
6	we set the "variable types" value of this configuration map to the map of variable types
7	and assign it to the thread local
8	eventually, to avoid memory leaks, it is important to remove the configuration from the thread local

And now, here is how the type checking extension can use this:

import static Sandbox.*

def typesOfVariables = COMPILE_OPTIONS.get()[VAR_TYPES]				(1)

unresolvedVariable { var ->
    if (typesOfVariables[var.name]) {						(2)
        return makeDynamic(var, typesOfVariables[var.name])			(3)
    }
}

1	Retrieve the list of variable types from the thread local
2	if an unresolved variable is found in the map of known variables
3	then declare to the type checker that the variable is of the type found in the map

Basically, the type checking extension, because it is executed when the type checker verifies the script, can access the configuration through the thread local. Then, instead of using hard coded names in unresolvedVariable, we can just check that the variable that the type checker doesn’t know about is actually declared in the configuration. If it is, then we can tell it which type it is. Easy!

Now we have to find a way to explicitly declare the list of allowed method calls. It is a bit trickier to find a proper configuration for that, but here is what we came up with.

Configuring a white list of methods

The idea of the whitelist is simple. A method call will be allowed if the method descriptor can be found in the whitelist. This whitelist consists of regular expressions, and the method descriptor consists of the fully-qualified class name of the method, it’s name and parameters. For example, for System.exit, the descriptor would be:

java.lang.System#exit(int)

So let’s see how to update the Java integration part to add this configuration:

public class Sandbox {
    public static final String WHITELIST_PATTERNS = "sandboxing.whitelist.patterns";

    // ...

    public static void main(String[] args) {
        // ...
        try {
            Map<String,ClassNode> variableTypes = new HashMap<String, ClassNode>();
            variableTypes.put("score", ClassHelper.double_TYPE);
            Map<String,Object> options = new HashMap<String, Object>();
            List<String> patterns = new ArrayList<String>();					(1)
            patterns.add("java\\.lang\\.Math#");						(2)
            options.put(VAR_TYPES, variableTypes);
            options.put(WHITELIST_PATTERNS, patterns);						(3)
            COMPILE_OPTIONS.set(options);
            GroovyShell shell = new GroovyShell(binding, conf);
            Double userScore = (Double) shell.evaluate("abs(cos(1+score));System.exit(-1)");
            System.out.println("userScore = " + userScore);
        } finally {
            COMPILE_OPTIONS.remove();
        }
    }
}

1	declare a list of patterns
2	add all methods of `java.lang.Math` as allowed
3	put the whitelist to the type checking options map

Then on the type checking extension side:

import groovy.transform.CompileStatic
import org.codehaus.groovy.ast.ClassNode
import org.codehaus.groovy.ast.MethodNode
import org.codehaus.groovy.ast.Parameter
import org.codehaus.groovy.transform.stc.ExtensionMethodNode

import static Sandbox.*

@CompileStatic
private static String prettyPrint(ClassNode node) {
    node.isArray()?"${prettyPrint(node.componentType)}[]":node.toString(false)
}

@CompileStatic
private static String toMethodDescriptor(MethodNode node) {								(1)
    if (node instanceof ExtensionMethodNode) {
        return toMethodDescriptor(node.extensionMethodNode)
    }
    def sb = new StringBuilder()
    sb.append(node.declaringClass.toString(false))
    sb.append("#")
    sb.append(node.name)
    sb.append('(')
    sb.append(node.parameters.collect { Parameter it ->
        prettyPrint(it.originType)
    }.join(','))
    sb.append(')')
    sb
}
def typesOfVariables = COMPILE_OPTIONS.get()[VAR_TYPES]
def whiteList = COMPILE_OPTIONS.get()[WHITELIST_PATTERNS]								(2)

onMethodSelection { expr, MethodNode methodNode ->
    def descr = toMethodDescriptor(methodNode)										(3)
    if (!whiteList.any { descr =~ it }) {										(4)
        addStaticTypeError("You tried to call a method which is not allowed, what did you expect?: $descr", expr)	(5)
    }
}

unresolvedVariable { var ->
    if (typesOfVariables[var.name]) {
        return makeDynamic(var, typesOfVariables[var.name])
    }
}

1	this method will generate a method descriptor from a `MethodNode`
2	retrieve the whitelist of methods from the thread local option map
3	convert a selected method into a descriptor string
4	if the descriptor doesn’t match any of the whitelist entries, throw an error

So if you execute the code again, you will now have a very cool error:

Script1.groovy: 1: [Static type checking] - You tried to call a method which is not allowed, what did you expect?: java.lang.System#exit(int)
 @ line 1, column 19.
   abs(cos(1+score));System.exit(-1)
                     ^

1 error

There we are! We now have a type checking extension which handles both the types of the variables that you export in the binding and a whitelist of allowed methods. This is still not perfect, but we’re very close to the final solution! It’s not perfect because we only took care of method calls here, but you have to deal with more than that. For example, properties (like foo.text which is implicitly converted into foo.getText()).

Putting it altogether

Dealing with properties is a bit more complicated because the type checker doesn’t have a handler for "property selection" like it does for methods. We can work around that, and if you are interested in seeing the resulting code, check it out below. It’s a type checking extension which is not written exactly as you have seen in this blog post, because it is meant to be precompiled for improved performance. But the idea is exactly the same.

SandboxingTypeCheckingExtension.groovy

import groovy.transform.CompileStatic
import org.codehaus.groovy.ast.ClassCodeVisitorSupport
import org.codehaus.groovy.ast.ClassHelper
import org.codehaus.groovy.ast.ClassNode
import org.codehaus.groovy.ast.MethodNode
import org.codehaus.groovy.ast.Parameter
import org.codehaus.groovy.ast.expr.PropertyExpression
import org.codehaus.groovy.control.SourceUnit
import org.codehaus.groovy.transform.sc.StaticCompilationMetadataKeys
import org.codehaus.groovy.transform.stc.ExtensionMethodNode
import org.codehaus.groovy.transform.stc.GroovyTypeCheckingExtensionSupport
import org.codehaus.groovy.transform.stc.StaticTypeCheckingSupport

import static Sandbox.*

class SandboxingTypeCheckingExtension extends GroovyTypeCheckingExtensionSupport.TypeCheckingDSL {

    @CompileStatic
    private static String prettyPrint(ClassNode node) {
        node.isArray()?"${prettyPrint(node.componentType)}[]":node.toString(false)
    }

    @CompileStatic
    private static String toMethodDescriptor(MethodNode node) {
        if (node instanceof ExtensionMethodNode) {
            return toMethodDescriptor(node.extensionMethodNode)
        }
        def sb = new StringBuilder()
        sb.append(node.declaringClass.toString(false))
        sb.append("#")
        sb.append(node.name)
        sb.append('(')
        sb.append(node.parameters.collect { Parameter it ->
            prettyPrint(it.originType)
        }.join(','))
        sb.append(')')
        sb
    }

    @Override
    Object run() {

        // Fetch white list of regular expressions of authorized method calls
        def whiteList = COMPILE_OPTIONS.get()[WHITELIST_PATTERNS]
        def typesOfVariables = COMPILE_OPTIONS.get()[VAR_TYPES]

        onMethodSelection { expr, MethodNode methodNode ->
            def descr = toMethodDescriptor(methodNode)
            if (!whiteList.any { descr =~ it }) {
                addStaticTypeError("You tried to call a method which is not allowed, what did you expect?: $descr", expr)
            }
        }

        unresolvedVariable { var ->
            if (isDynamic(var) && typesOfVariables[var.name]) {
                storeType(var, typesOfVariables[var.name])
                handled = true
            }
        }

        // handling properties (like foo.text) is harder because the type checking extension
        // does not provide a specific hook for this. Harder, but not impossible!

        afterVisitMethod { methodNode ->
            def visitor = new PropertyExpressionChecker(context.source, whiteList)
            visitor.visitMethod(methodNode)
        }
    }

    private class PropertyExpressionChecker extends ClassCodeVisitorSupport {
        private final SourceUnit unit
        private final List<String> whiteList

        PropertyExpressionChecker(final SourceUnit unit, final List<String> whiteList) {
            this.unit = unit
            this.whiteList = whiteList
        }

        @Override
        protected SourceUnit getSourceUnit() {
            unit
        }

        @Override
        void visitPropertyExpression(final PropertyExpression expression) {
            super.visitPropertyExpression(expression)

            ClassNode owner = expression.objectExpression.getNodeMetaData(StaticCompilationMetadataKeys.PROPERTY_OWNER)
            if (owner) {
                if (expression.spreadSafe && StaticTypeCheckingSupport.implementsInterfaceOrIsSubclassOf(owner, classNodeFor(Collection))) {
                    owner = typeCheckingVisitor.inferComponentType(owner, ClassHelper.int_TYPE)
                }
                def descr = "${prettyPrint(owner)}#${expression.propertyAsString}"
                if (!whiteList.any { descr =~ it }) {
                    addStaticTypeError("Property is not allowed: $descr", expression)
                }
            }
        }
    }
}

And a final version of the sandbox that includes assertions to make sure that we catch all cases:

Sandbox.java

public class Sandbox {
    public static final String WHITELIST_PATTERNS = "sandboxing.whitelist.patterns";
    public static final String VAR_TYPES = "sandboxing.variable.types";

    public static final ThreadLocal<Map<String, Object>> COMPILE_OPTIONS = new ThreadLocal<Map<String, Object>>();

    public static void main(String[] args) {
        CompilerConfiguration conf = new CompilerConfiguration();
        ImportCustomizer customizer = new ImportCustomizer();
        customizer.addStaticStars("java.lang.Math");
        ASTTransformationCustomizer astcz = new ASTTransformationCustomizer(
                singletonMap("extensions", singletonList("SandboxingTypeCheckingExtension.groovy")),
                CompileStatic.class);
        conf.addCompilationCustomizers(astcz);
        conf.addCompilationCustomizers(customizer);

        Binding binding = new Binding();
        binding.setVariable("score", 2.0d);
        try {
            Map<String, ClassNode> variableTypes = new HashMap<String, ClassNode>();
            variableTypes.put("score", ClassHelper.double_TYPE);
            Map<String, Object> options = new HashMap<String, Object>();
            List<String> patterns = new ArrayList<String>();
            // allow method calls on Math
            patterns.add("java\\.lang\\.Math#");
            // allow constructors calls on File
            patterns.add("File#<init>");
            // because we let the user call each/times/...
            patterns.add("org\\.codehaus\\.groovy\\.runtime\\.DefaultGroovyMethods");
            options.put(VAR_TYPES, variableTypes);
            options.put(WHITELIST_PATTERNS, patterns);
            COMPILE_OPTIONS.set(options);
            GroovyShell shell = new GroovyShell(binding, conf);
            Object result;
            try {
                result = shell.evaluate("Eval.me('1')"); // error
                assert false;
            } catch (MultipleCompilationErrorsException e) {
                System.out.println("Successful sandboxing: "+e.getMessage());
            }
            try {
                result = shell.evaluate("System.exit(-1)"); // error
                assert false;
            } catch (MultipleCompilationErrorsException e) {
                System.out.println("Successful sandboxing: "+e.getMessage());
            }
            try {
                result = shell.evaluate("((Object)Eval).me('1')"); // error
                assert false;
            } catch (MultipleCompilationErrorsException e) {
                System.out.println("Successful sandboxing: "+e.getMessage());
            }

            try {
                result = shell.evaluate("new File('/etc/passwd').getText()"); // getText is not allowed
                assert false;
            } catch (MultipleCompilationErrorsException e) {
                System.out.println("Successful sandboxing: "+e.getMessage());
            }

            try {
                result = shell.evaluate("new File('/etc/passwd').text");  // getText is not allowed
                assert false;
            } catch (MultipleCompilationErrorsException e) {
                System.out.println("Successful sandboxing: "+e.getMessage());
            }

            Double userScore = (Double) shell.evaluate("abs(cos(1+score))");
            System.out.println("userScore = " + userScore);
        } finally {
            COMPILE_OPTIONS.remove();
        }
    }
}

Conclusion

This post has explored the interest of using Groovy as a platform for scripting on the JVM. It introduced various mechanisms for integration, and showed that this comes at the price of security. However, we illustrated some concepts like compilation customizers that make it easier to sandbox the environment of execution of scripts. The current list of customizers available in the Groovy distribution, and the currently available sandboxing projects in the wild, are not sufficient to guarantee security of execution of scripts in the general case (dependending, of course, on your users and where the scripts come from).

We then illustrated how you could, if you are ready to pay the price of loosing some of the dynamic features of the language, properly workaround those limitations through type checking extensions. Those type checking extensions are so powerful that you can even introduce your own error messages during the compilation of scripts. Eventually, by doing this and caching your scripts, you will also benefit from dramatic performance improvements in script execution.

Eventually, the sandboxing mechanism that we illustrated is not a replacement for the SecureASTCustomizer. We recommand that you actually use both, because they work on different levels: the secure AST customizer will work on the grammar level, allowing you to restrict some language constructs (for example preventing the creation of closures or classes inside the script), while the type checking extension will work after type inference (allowing you to reason on inferred types rather than declared types).

Last but not least, the solution that I described here is incomplete. It is not available in Groovy core. As I will have less time to work on Groovy, I would be very glad if someone or some people improve this solution and make a pull request so that we can have something!

Who is Groovy?

Wed, 4 Mar 2015 00:00:00 +0100

With all the changes that the Groovy project is seeing since the beginning of the year, I thought it was a good time to make a summary about its history. In particular, with the end of sponsorship from Pivotal, as well as Guillaume Laforge annoucing he is joining Restlet, a lot of people state that Groovy is done. It will be the occasion to talk about the history of the project, both in terms of community and sponsorship.

First of all, Groovy is, and will remain, alive. Groovy is a community project and we are pleased to announce that Groovy has started the process to join the Apache Software Foundation. The community deserved it, and even if it will mean some adaptations on our side, we think that the ASF will be a great fit for the project.

To build the statistics you will read in this post, I have used a Groovy script (of course!) that takes as reference the number of commits. This is far from being a perfect number, because some commits are just for fixing typos, while others are full new features and it also totally misses the fact that patches, before Git, didn’t carry the author information or the problem that we maintain multiple branches of Groovy, requiring a lot of work, but it gives an idea… And because people think that end of sponsorship may be equivalent to death, I separated commits in two categories: sponsored and community. Sponsored commits are commits which are likely to have been made by someone directly paid to contribute on Groovy. We will see how this proportion evolved over time.

If you think Groovy is dead, please read the following carefully. Let’s start our journey back in time!

2003: A dynamic language for the JVM

Total: 476 commits Community: 476 commits (100%)

James Strachan : 374 commits (78%)
Bob McWhirter : 76 commits (15%)
Sam Pullara : 23 commits (4%)
Kasper Nielsen : 2 commits (0%)
Aslak Hellesoy : 1 commits (0%)

2003 is the inception year of Groovy. James Strachan, in August 2003, wanted to create a dynamic language for the JVM, inspired by Ruby, but closer to Java. Groovy was born, and the idea never changed over time: Groovy is the perfect companion for Java, a language which can be very close to Java in terms of syntax but also removes a lot of its boilerplate. Bob McWhirter is a famous name in the JVM world, he is now Director of Polyglot at Red Hat, so you can see that when he started contributing to the language back then, there was already a story for him! Sam Pullara is another very smart guy in the JVM world and he worked for top companies like BEA, Yahoo! or Twitter to name a few. He is a technical reviewer for the JavaOne conference.

2004: Guillaume Laforge joins the project

Total: 871 commits Community: 871 commits (100%)

James Strachan : 495 commits (56%)
Guillaume Laforge : 101 commits (11%)
John Wilson : 70 commits (8%)
Sam Pullara : 66 commits (7%)
Jeremy Rayner : 33 commits (3%)
Chris Poirier : 28 commits (3%)
Bing Ran : 21 commits (2%)
Steve Goetze : 12 commits (1%)
John Stump : 10 commits (1%)
Russel Winder : 8 commits (0%)
Zohar Melamed : 8 commits (0%)
Jochen Theodorou : 7 commits (0%)
Damage Control : 4 commits (0%)
Bob McWhirter : 3 commits (0%)
Christiaan ten Klooster : 3 commits (0%)
Yuri Schimke : 2 commits (0%)

Groovy is hosted at Codehaus (which just has annouced its retirement) and 2004 sees the appearance of famous names of the community. In particular, you can already see Guillaume Laforge and Jochen Theodorou. Both of them still directly work on the project as today. John Wilson started contributing the famous XML support of Groovy, and you can also note names like Russel Winder of Gant and GPars fame. Jeremy Rayer’s work is famous in the Groovy community since he wrote the first versions of the Groovy grammar using Antlr.

2005: Jochen Theodorou joins the project

Total: 934 commits Community: 934 commits (100%)

Jochen Theodorou : 244 commits (26%)
James Strachan : 162 commits (17%)
Pilho Kim : 104 commits (11%)
John Wilson : 79 commits (8%)
Guillaume Laforge : 75 commits (8%)
Dierk Koenig : 70 commits (7%)
Jeremy Rayner : 62 commits (6%)
Christian Stein : 48 commits (5%)
Alan Green : 20 commits (2%)
Russel Winder : 20 commits (2%)
Martin C. Martin : 14 commits (1%)
Sam Pullara : 10 commits (1%)
John Rose : 8 commits (0%)
Hein Meling : 7 commits (0%)
Scott Stirling : 6 commits (0%)
Franck Rasolo : 5 commits (0%)

I would tend to think that 2005 is the year when Jochen Theodorou took the technical lead of Groovy. In 2005, he becomes the most prolific contributor, even beyond the creator of the language himself. Dierk Koenig makes an appearance here: he is known for his work on GPars, but also for the reference book for Groovy: Groovy in Action.

2006: Rise of Paul King

Total: 480 commits Community: 480 commits (100%)

Jochen Theodorou : 221 commits (46%)
John Wilson : 56 commits (11%)
Paul King : 54 commits (11%)
Guillaume Laforge : 47 commits (9%)
Dierk Koenig : 37 commits (7%)
Jeremy Rayner : 23 commits (4%)
Russel Winder : 14 commits (2%)
Guillaume Alleon : 12 commits (2%)
Joachim Baumann : 6 commits (1%)
Martin C. Martin : 4 commits (0%)
Graeme Rocher : 2 commits (0%)
Marc Guillemot : 2 commits (0%)
Christian Stein : 1 commits (0%)
Steve Goetze : 1 commits (0%)

2006 is a very calm year for Groovy in terms of code production. James, the creator of the language, already disappeared from the contributors, and will not contribute anymore. Guillaume Laforge, in agreement with the other contributors, takes the project lead (he is still the lead today).

With half as many commits as in 2007, in retrospect, I would say that this was a critical year: either the project would die, or it would have become what it is today. And my personal feeling is that the person who saved Groovy just appeared in the contributors list: Paul King. Paul is undoubtfully the most active contributor to Groovy. He wrote a lot of the Groovy Development Kit, that is to say the APIs without which a language would be nothing. Having a nice language is one thing, having proper APIs and libraries that unleash its full potential is another. Paul King did it. Look at his ranking here: 3rd place. You will never see him ranked lower than that. And guess what? Paul is not paid to do this. He runs his own business and if you want to work with a Groovy expert, he’s probably the best.

Joachim Baumann is a name some people would recognize: he is still working with Groovy and one of the most regular contributors, with the Windows installer. Joachim takes time, for each Groovy release, to produce a Windows installer, which today we are still not capable of handling automatically.

2007: Groovy 1.0

Paul King : 447 commits (30%)
Jason Dillon : 265 commits (18%)
Jochen Theodorou (Sponsored) : 242 commits (16%)
Danno Ferrin : 101 commits (6%)
Alex Tkachman (Sponsored) : 87 commits (5%)
Graeme Rocher (Sponsored) : 61 commits (4%)
Russel Winder : 46 commits (3%)
Marc Guillemot : 36 commits (2%)
Andres Almiray : 34 commits (2%)
Guillaume Laforge (Sponsored) : 33 commits (2%)
Jeremy Rayner : 26 commits (1%)
Alexandru Popescu : 24 commits (1%)
John Wilson : 22 commits (1%)
Joachim Baumann : 21 commits (1%)
Jeff Brown : 8 commits (0%)
Dierk Koenig : 6 commits (0%)
Martin C. Martin : 6 commits (0%)
Guillaume Alleon : 4 commits (0%)

2007 is an important year in the history of Groovy. On 2nd January, Groovy 1.0 is out. Paul King ranks #1 for the first time, and will remain on top for a long time. This year also sees the creation of G2One, the first company build specifically for Groovy and Grails, by Guillaume Laforge, Graeme Rocher and Alex Tkachman. Both Graeme and Alex make their first appearance in the contributors graph, and both of them made significant contributions to the Groovy ecosystem: Graeme is famous for co-creating the Grails framework, and is still the lead of the project, while Alex is the one who contributed major performance improvements to the Groovy runtime (call site caching) and first experimented with a static compiler for Groovy (Groovy++).

Danno Ferrin contributed what is still one of my personal favorite features of Groovy, AST transformations, and probably one of the reasons I got paid to work on Groovy so thank you Danno! Andrés Almiray, listed here for the first time, is famous for the Griffon framework, a Grails-like framework for desktop applications which is still actively developed. He spent a lot of time improving the Swing support in Groovy.

Starting from 2007, you will see that the sponsored ratio of commits is changing. People who were employed by G2One fall into that category. As you can see, 2007 is more than important for Groovy, it is its second birth. And to conclude that, Groovy won the first prize at JAX 2007 innovation award.

2008: The G2One era

Total: 1069 commits Sponsored: 287 commits (26%) Community: 782 commits (73%)

Paul King : 445 commits (41%)
Danno Ferrin : 176 commits (16%)
Jochen Theodorou (Sponsored) : 126 commits (11%)
Alex Tkachman (Sponsored) : 125 commits (11%)
Guillaume Laforge (Sponsored) : 33 commits (3%)
Jim White : 32 commits (2%)
Russel Winder : 31 commits (2%)
Martin Kempf : 22 commits (2%)
Roshan Dawrani : 19 commits (1%)
Jeremy Rayner : 14 commits (1%)
Martin C. Martin : 12 commits (1%)
Jason Dillon : 9 commits (0%)
Andres Almiray : 8 commits (0%)
Thom Nichols : 5 commits (0%)
Graeme Rocher (Sponsored) : 3 commits (0%)
Jeff Brown : 3 commits (0%)
John Wilson : 3 commits (0%)
James Williams : 1 commits (0%)
Marc Guillemot : 1 commits (0%)
Vladimir Vivien : 1 commits (0%)

In 2008, Paul King still ranks #1 and you can see that the people who were sponsored by G2One were actually not the main contributors. Actually, most of them did consulting to pay salaries, which doesn’t leave much time to contribute to the language. Hopefully, a great project such as Groovy can rely on its community! Guillaume, Graeme and Alex were looking for an opportunity to spend more time on actual development, and it happened in November 2008 when G2One got acquired by SpringSource.

Some of the contributors you see in this list are still actively using Groovy or contributing: Jim White for example is famous for his contributions on the scripting sides of the language. Roshan Dawrani is one of the few guys capable of opening cryptic code and fixing bugs. Jeff Brown is a name you should know, since he is now a key member of the Grails team.

2009: milestones and the inappropriate quote

Total: 835 commits Sponsored: 183 commits (21%) Community: 652 commits (78%)

Paul King : 342 commits (40%)
Roshan Dawrani : 128 commits (15%)
Jochen Theodorou (Sponsored) : 101 commits (12%)
Alex Tkachman (Sponsored) : 41 commits (4%)
Guillaume Laforge (Sponsored) : 40 commits (4%)
Jason Dillon : 31 commits (3%)
Jim White : 31 commits (3%)
Danno Ferrin : 24 commits (2%)
Peter Niederwieser : 23 commits (2%)
Hamlet D’Arcy : 18 commits (2%)
Russel Winder : 14 commits (1%)
Martin C. Martin : 13 commits (1%)
Thom Nichols : 13 commits (1%)
Andres Almiray : 12 commits (1%)
Vladimir Vivien : 3 commits (0%)
Graeme Rocher (Sponsored) : 1 commits (0%)

2009 is another important year concluding with the release of Groovy 1.7, the first version of Groovy supporting inner classes or the famous power asserts from Peter Niederwieser. If you know Groovy, you must know Peter, the father of the famous Spock testing framework which just reached 1.0!

Hamlet D’Arcy contributed a lot in terms of code quality, but also became the first specialist of AST transformations. 2009 is also the year I started to use Groovy, as a user. I never stopped and actually I started contributing back then. At that time, Groovy was still using Subversion (we’re now using Git like all the cool kids), so it was the good old patch way, loosing authorship.

This year is also the year when James Strachan wrote a very famous quote about Groovy. This quote is probably the most innapropriately used quote about Groovy of all time, because it was done by its creator, but remember that James left the project in 2005!

I can honestly say if someone had shown me the Programming in Scala book by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I’d probably have never created Groovy.

— James Strachan
on his blog

First of all James says nothing about the language itself here. He had already left the project and says that if he had known about Scala before, he wouldn’t have created Groovy. I am today very happy that he didn’t know about it, or we would have missed an incredibly powerful language. Groovy today is nothing close to what it was when James left the project, thanks to the lead of Guillaume Laforge and incredibly talented people like Paul King, Jochen Theodorou and all the contributors listed on this page. Groovy and Scala both have their communities, but also different use cases. I wouldn’t sell one for the other…

In the end of 2009, another important milestone occurred for project, with VMware acquiring SpringSource.

2010: DSLs all the way

Total: 894 commits Sponsored: 189 commits (21%) Community: 705 commits (78%)

Paul King : 443 commits (49%)
Roshan Dawrani : 134 commits (14%)
Jochen Theodorou (Sponsored) : 96 commits (10%)
Guillaume Laforge (Sponsored) : 93 commits (10%)
Hamlet D’Arcy : 71 commits (7%)
Alex Tkachman : 28 commits (3%)
Peter Niederwieser : 19 commits (2%)
Andres Almiray : 7 commits (0%)
Jason Dillon : 1 commits (0%)
Russel Winder : 1 commits (0%)
Thom Nichols : 1 commits (0%)

2010 is a pretty stable year for Groovy. Groovy reaches 1.8 in 2010 with important features for its incredible DSL design capabilities. With command chain expressions, native JSON support and performance improvements, Groovy put the bar very high in terms of integration in the Java ecosystem. Today, no other JVM language is as simple as Groovy to integrate with Java. With cross-compilation and by the use of the very same class model, Groovy is at that date the best language for scripting on the JVM. It is so good that a lot of people start to see it as a better Java and want to use it as a first class language. However, being dynamic, Groovy is still a problem for a category of users…

2011: Time to move to GitHub

Total: 841 commits Sponsored: 514 commits (61%) Community: 327 commits (38%)

Cédric Champeau (Sponsored) : 252 commits (29%)
Paul King : 212 commits (25%)
Jochen Theodorou (Sponsored) : 163 commits (19%)
Guillaume Laforge (Sponsored) : 98 commits (11%)
Jochen : 44 commits (5%)
Hamlet D’Arcy : 33 commits (3%)
Roshan Dawrani : 26 commits (3%)
Andres Almiray : 1 commits (0%)
Andrew Eisenberg : 3 commits (0%)
Alex Tkachman : 2 commits (0%)
Bobby Warner : 1 commits (0%)
Colin Harrington : 1 commits (0%)
Dierk Koenig : 1 commits (0%)
Dirk Weber : 1 commits (0%)
John Wagenleitner : 1 commits (0%)
Lari Hotari (Sponsored) : 1 commits (0%)
Peter Niederwieser : 1 commits (0%)

In 2011, I became a committer to the Groovy project. As I said, I had contributed several fixes or features for Groovy 1.8, but for the first time, I became a committer and I started to be able to push changes to the codebase without having to ask permission. So this is basically the first time you see my name on the contributors list, but you can see that I am ranking #1 and I have never lost that ranking since then. It surprised me too, but there is a very good reason for that. In october 2011, in addition to being a committer, I also became paid to work on Groovy. Full-time. I entered the club of lucky people being paid to work on open-source software. It was sincerely a dream, and I will never be enough thankful to Guillaume Laforge for giving me this opportunity. He changed my life and I think I became a better developer thanks to him. VMware was my employer back then, and while I had never worked on a language before, Guillaume trusted my skills and proposed to me to work on something that would dramatically change the language : a static type checker.

I also worked on the infrastructure of the language, starting from the migration to GitHub. It was an important move to make: as you can see, there was a very limited set of committers to Groovy. With GitHub, we had the tool we needed to increase the size of our community and from the numbers that will follow, I think it’s a success.

2012: Groovy 2 and static compilation

Cédric Champeau (Sponsored) : 515 commits (46%)
Paul King : 249 commits (22%)
Jochen Theodorou (Sponsored) : 169 commits (15%)
Guillaume Laforge (Sponsored) : 74 commits (6%)
PascalSchumacher : 12 commits (1%)
Peter Niederwieser : 11 commits (0%)
René Scheibe : 11 commits (0%)
Andre Steingress : 9 commits (0%)
John Wagenleitner : 7 commits (0%)
Peter Ledbrook : 6 commits (0%)
Andres Almiray : 6 commits (0%)
Adrian Nistor : 5 commits (0%)
Tim Yates : 5 commits (0%)
Baruch Sadogursky : 4 commits (0%)
Andrew Eisenberg : 3 commits (0%)
Rich Freedman : 3 commits (0%)
Stephane Maldini : 3 commits (0%)
Andrew Taylor : 2 commits (0%)
Jeff Brown : 2 commits (0%)
Luke Daley : 2 commits (0%)
Tiago Fernandez : 2 commits (0%)
Andrey Bloschetsov : 1 commits (0%)
Johnny Wey : 1 commits (0%)
Kenneth Kousen : 1 commits (0%)
Mathieu Bruyen : 1 commits (0%)
Paul Bakker : 1 commits (0%)
Paulo Poiati : 1 commits (0%)
Sean Flanigan : 1 commits (0%)
Suk-Hyun Cho : 1 commits (0%)
Vladimir Orany : 1 commits (0%)

2012 is one of the most important years for the language. It was the year Groovy 2.0 was released. As you can see, I am still ranking #1 and Paul King, an unpaid contributor, is #2. This tells you the importance of community! Groovy 2 is a major change in the language, because it introduced both optional type checking and static compilation. For the first time, Groovy was able to provide at compile time the same level of feedback that Java would have. Some people wanted to kill me for having introduced that into the language. The truth is that it wasn’t my decision, but in retrospect, I am very happy with what the language is now. Without this, some people would have abandonned Groovy in favor of other JVM languages like Scala, while now in Groovy you can have the same level of performance as Java, with type safety, powerful type inference, extension methods, functional style programming and without the boilerplate. And it’s optional. I don’t know any other language that allows this, especially when you take type checking extensions into account, a feature that allows Groovy to go far beyond what Java and other languages offer in terms of type safety or static compilation.

2012 also sees the appearance of Pascal Schumacher, a silent but very active Groovy committer. Pascal does since 2012 an amazing job in helping us filtering JIRA issues, writing bugfixes, reviewing pull requests and lately writing documentation.

2013: Documentation effort and explosion of contributions

Cédric Champeau (Sponsored) : 244 commits (22%)
Paul King : 188 commits (17%)
PascalSchumacher : 180 commits (16%)
Jochen Theodorou (Sponsored) : 96 commits (8%)
Thibault Kruse : 84 commits (7%)
Guillaume Laforge (Sponsored) : 54 commits (4%)
Andrey Bloschetsov : 43 commits (3%)
Andre Steingress : 36 commits (3%)
Pascal Schumacher : 27 commits (2%)
Tim Yates : 24 commits (2%)
René Scheibe : 12 commits (1%)
kruset : 12 commits (1%)
Martin Hauner : 8 commits (0%)
Andres Almiray : 8 commits (0%)
Larry Jacobson : 4 commits (0%)
John Wagenleitner : 6 commits (0%)
Paolo Di Tommaso : 6 commits (0%)
Jeff Scott Brown (Sponsored) : 5 commits (0%)
Masato Nagai : 5 commits (0%)
Jochen Eddelbüttel : 3 commits (0%)
hbaykuslar : 3 commits (0%)
shalecraig : 3 commits (0%)
Andrew Eisenberg : 2 commits (0%)
Jacopo Cappellato : 2 commits (0%)
Peter Niederwieser : 2 commits (0%)
Rafael Luque : 2 commits (0%)
Vladimir Orany : 1 commits (0%)
saschaklein : 2 commits (0%)
seanjreilly : 2 commits (0%)
upcrob : 2 commits (0%)
Adrian Nistor : 1 commits (0%)
Alan Thompson : 1 commits (0%)
Alessio Stalla : 1 commits (0%)
DJBen : 1 commits (0%)
Eric Dahl : 1 commits (0%)
Ingo Hoffmann : 1 commits (0%)
JBaruch : 1 commits (0%)
Jacob Aae Mikkelsen : 1 commits (0%)
Jim White : 1 commits (0%)
John Engelman : 1 commits (0%)
Jon Schneider : 1 commits (0%)
Karel Piwko : 1 commits (0%)
Kenneth Endfinger : 1 commits (0%)
Kohsuke Kawaguchi : 1 commits (0%)
Luke Kirby : 1 commits (0%)
Michal Mally : 1 commits (0%)
Miro Bezjak : 1 commits (0%)
Olivier Croquette : 1 commits (0%)
Rob Upcraft : 1 commits (0%)
Sergey Egorov : 1 commits (0%)
Stefan Armbruster : 1 commits (0%)
Yasuharu NAKANO : 1 commits (0%)

While continuing to improve Groovy, 2013 was very important for the community. You can start to see the GitHub effect here, with much more contributors than before. It is impressive to see the difference before 2011 and after. The number of contributors is continously growing. In 2013, 63% of commits came from the community!

In February 2013, we also launched a new big project: the documentation and website overhaul. It is incredible to think that this effort is still uncomplete, but if you see that the old wiki has more than a thousand page or contents (often outdated), you can imagine what effort it takes to rewrite the documentation. Hopefully, we’re close to filling the gap now, and with the demise of Codehaus, we officially launched our new website where you can see the result of this job.

I also started working on Android support during 2013, for a first overview in GR8Conf 2014, and continued working on improving the infrastructure, with Bintray, TeamCity and Gradle. And Pivotal was born, out of EMC and VMware. Groovy and Grails, along with the Spring Framework, became part of this new company which is still paying me today to work on Groovy (and I, we, should be very thankful for this).

2014: Towards Android support

Cédric Champeau (Sponsored) : 446 commits (37%)
Paul King : 261 commits (22%)
Jochen Theodorou (Sponsored) : 85 commits (7%)
Guillaume Laforge (Sponsored) : 61 commits (5%)
Thibault Kruse : 54 commits (4%)
Pascal Schumacher : 47 commits (3%)
Jim White : 26 commits (2%)
Yu Kobayashi : 18 commits (1%)
Andre Steingress : 16 commits (1%)
Richard Hightower : 3 commits (0%)
James Northrop : 11 commits (0%)
Kenneth Endfinger : 9 commits (0%)
Tomek Janiszewski : 9 commits (0%)
Matias Bjarland : 8 commits (0%)
Tobia Conforto : 8 commits (0%)
Michael Schuenck : 7 commits (0%)
Sargis Harutyunyan : 7 commits (0%)
Andrey Bloschetsov : 6 commits (0%)
Craig Andrews : 5 commits (0%)
Kent : 5 commits (0%)
Paolo Di Tommaso : 5 commits (0%)
Peter Ledbrook : 5 commits (0%)
Sergey Egorov : 5 commits (0%)
Yasuharu Nakano : 5 commits (0%)
Andrew Hamilton : 4 commits (0%)
Lari Hotari (Sponsored) : 4 commits (0%)
Bloshchetsov Andrey Evgenyevich : 3 commits (0%)
Johannes Link : 3 commits (0%)
Keegan Witt : 3 commits (0%)
Tim Yates : 3 commits (0%)
anto_belgin : 3 commits (0%)
Baruch Sadogursky : 2 commits (0%)
Dan Allen : 2 commits (0%)
Jan Sykora : 2 commits (0%)
John Wagenleitner : 2 commits (0%)
Luke Kirby : 2 commits (0%)
Martin Stockhammer : 2 commits (0%)
UEHARA Junji : 2 commits (0%)
Vihang D : 2 commits (0%)
Andres Almiray : 2 commits (0%)
Andy Hamilton : 1 commits (0%)
Bobby Warner : 1 commits (0%)
Carsten Lenz : 1 commits (0%)
Chris Earle : 1 commits (0%)
David Avenante : 1 commits (0%)
David Nahodil : 1 commits (0%)
David Tiselius : 1 commits (0%)
Dimitar Dimitrov : 1 commits (0%)
Grant McConnaughey : 1 commits (0%)
Jeff Sheets : 1 commits (0%)
Jess Sightler : 1 commits (0%)
Logan Gorence : 1 commits (0%)
Luke Daley : 1 commits (0%)
Manuel Prinz : 1 commits (0%)
Marc Guillemot : 1 commits (0%)
Marcin Grzejszczak : 1 commits (0%)
Nathan Mische : 1 commits (0%)
Peter Swire : 1 commits (0%)
Sagar Sane : 1 commits (0%)
Stephen Mallette : 1 commits (0%)
Tobias Schulte : 1 commits (0%)
Wil Selwood : 1 commits (0%)
davidmichaelkarr : 1 commits (0%)
fintelia : 1 commits (0%)
kruset : 1 commits (0%)
paul-bjorkstrand : 1 commits (0%)

2014 was a difficult year. We had a lot of work to do on the documentation side, new features to deliver (traits) and an important topic we definitely wanted to highlight: Android support. This took longer than expected, but in the end, the new Groovy 2.4. We’re lucky to have half of the commits coming from the community here. Especially, lots of people helped us on the documentation. And it wasn’t easy, because our documentation requires that every snippet of code that appears in the docs belongs to a unit test, to make sure that the documentation is always up-to-date.

Meanwhile, at the end of the year, we learnt from Pivotal that they would end sponsoring our jobs. It means that Guillaume Laforge, Jochen Theodorou and myself, for the Groovy team, plus Graeme Rocher, Jeff Brown and Lari Hotari, for the Grails team, were both loosing their jobs and full time to work on the project at the same time. This wasn’t really a surprise and I am very happy I could work for so long on Groovy, full time, but as I said in a previous post I also wish I will still be able to do that, because you can see from the numbers and features that it matters. If you wonder, we are still discussing with several potential sponsors.

2015: Your story

Total: 178 commits Sponsored: 81 commits (45%) Community: 97 commits (54%)

Cédric Champeau (Sponsored) : 69 commits (38%)
Pascal Schumacher : 59 commits (33%)
Jochen Theodorou (Sponsored) : 12 commits (6%)
Paul King : 12 commits (6%)
JBrownVisualSpection : 7 commits (3%)
Yu Kobayashi : 3 commits (1%)
Christoph Frick : 2 commits (1%)
Kamil Szymanski : 2 commits (1%)
Michael Schuenck : 2 commits (1%)
Sean Gilligan : 2 commits (1%)
Sergey Egorov : 2 commits (1%)
Thibault Kruse : 2 commits (1%)
Andy Wilkinson : 1 commits (0%)
Maksym Stavytskyi : 1 commits (0%)
Mario Garcia : 1 commits (0%)
Radovan Synek : 1 commits (0%)

2015 will be another important year. It’s going to be huge for the community. Guillaume Laforge announced that he was joining Restlet, so for the first time since 2007 he will not be fully employed to work on Groovy, but I don’t expect this to have a big impact on the language development itself: as you can see from the numbers, about half of the commits already come from the community and Guillaume didn’t contribute much code lately. He was instead the lead of the project, the one that took decisions, the one speaking about the project and talking to and leading the community. He was the voice. It was a hard job, a very important one for Groovy. Guillaume is still today the lead of the project, and he will continue to contribute to the language, but I know from him that he wanted to be able to do more code, and put Groovy in action into a new project.

With the end of sponsorship of Pivotal, the demise of Codehaus and Guillaume’s decision, it became even more important to move Groovy to a foundation where it will be able to live with or without us. I have honestly no idea where I will work in a few weeks now. I sincerely hope I will still be able to contribute to the language full time, but let’s be clear: today, it is very unlikely this is going to happen. It makes it very important for the project to be able to develop the community even more. We had more than 4.5 million downloads last year. This is huge. And with Android support, I see a lot of potential, even if we have tough competition with other languages and people being paid to develop them. The Apache Software Foundation is going to help us with securing the future of the language and building a community. I am proud of what you have done, collectively, and this is not over. Groovy is ready for a rebirth under the Apache umbrella!

More than ever, the future of Groovy is you.

To conclude this post, here are the top 10 contributors, in terms of number of commits, of Groovy, for the past 12 years. Congratulations Paul and thanks to our 100+ contributors!

Paul King :Paul King : 2653 commits (23%)
Jochen Theodorou : 1562 commits (13%)
Cédric Champeau : 1526 commits (13%)
James Strachan : 1031 commits (9%)
Guillaume Laforge : 709 commits (6%)
Roshan Dawrani : 307 commits (2%)
Jason Dillon : 306 commits (2%)
Danno Ferrin : 301 commits (2%)
Alex Tkachman : 283 commits (2%)
John Wilson : 230 commits (2%)

Looking for a new job

Mon, 19 Jan 2015 00:00:00 +0100

Will you be the new Groovy sponsor?

There we are. January 19th was a bad day as Pivotal announced that they wouldn’t sponsor the development of the Groovy language, as well as its long time friend the Grails framework, starting from march 31st. As I was paid to work on Groovy, it has a direct consequence for me: starting from April 1st, I am for hire. For you, it probably doesn’t change much as Groovy and Grails existed before Pivotal without corporate funding.

It was hard to keep this secret for weeks, but now that everything is public, it is easier for us. I have been lucky to work full time on the Groovy language for a bit more than 3 years now. Lucky because it was both a great open-source software and my passion. I have implemented major features of the language since Groovy 1.8: static type checking, static compilation, Android support, numerous AST transformations, continous integration… I have given talks to various conferences, including world class ones like JavaOne, Devoxx, GR8Conf or SpringOne2GX and it was a fantastic opportunity that wouldn’t have been possible without the support from VMware then Pivotal.

However I feel that I still have a lot to do. So many things are on the list, so much to improve, like for example adding language support for asynchronous programming, improved Java 8 support, a new meta-object protocol, not forgetting, of course, bugfixes… I am 100% convinced that the language wouldn’t have reached the level of maturity it has without support from a company like Pivotal. And even though Java is considered mature, it doesn’t mean that the language shouldn’t continue to evolve: it is rather the opposite. People are constantly asking for evolutions, because programming models evolve too.

For those reasons, my #1 choice would be to find a new company securing the development of Groovy and Grails, that is to say willing to pay me and my mates to continue working on them. The Groovy community is huge, there are lots of companies using Groovy in a variety of domains, I sincerely hope this is possible. With more than 4 million downloads in 2014, I can’t imagine how many hours of development have been saved thanks to Groovy and Grails… Should you be interested in sponsoring our projects, please write to sponsorship@groovy-lang.org.

About me

If unfortunately we can’t find third parties willing to take over the development of the language and hire us, then I would obviously be available to help you in your projects. For those of you who are looking for someone with a good knowledge of the JVM internals, I am listening. My areas of interest:

R&D: innovation, technical challenges are what drive my motivation
open-source over closed-source: If you are an Open-Source company, by that I mean a company which contributes Open-Source, I think we are on the same line. I love to share what I do and I am convinced that open-source development is the best way to improve the global quality of a project. I have worked on closed-source software in the past, there are good reasons to do so but working on OSS software is really what I prefer. I also enjoy speaking publicly about projects I work on.
JVM: I have spent most of my career on that platform and believe me, it’s not dead :)
back-end over front-end: I love working on tooling, performance tuning, algorithmics… everything that is hidden to the end user. On the other side I am not so good on the front-end part (HTML, CSS, …) so it would probably not be a good idea to make me work on that topic :)
remote working: I live in Saint Hilaire de Loulay, a small countryside town near Nantes, in France. I have now been working from home for more that 3 years, most of the Spring Framework team works that way, and it has been very successful so far. If you are not against the idea, I would love to continue that way.

In any case if you think a profile like mine is interesting for your company, you can contact me or ask for my résumé at cedric.champeau@gmail.com. Meanwhile, rest assured that we are focused and we will soon release Groovy 2.4 to prove that!

10 things your static language can’t do

Mon, 15 Dec 2014 00:00:00 +0100

But maybe mine can…

For those of you who do not know me, and you are likely much more in that category than in the other, I’ve been working on the Groovy language full-time for more than 3 years now. I started as a user several years ago, in the context of DSLs, and eventually became a contributor before getting employed to work on the language. I love static typing, but not at the point of thinking that we should only reason about types. This is why I also love dynamic typing, and why I invested so much in Groovy.

Groovy is primarily a dynamic language. It is probably the most widely used alternative language on the JVM, with use cases ranging from DSLs (scripting Jenkins builds for example) to full blown applications on mobile devices (Android), going through full stack web applications with Grails. Despite being primarily a dynamic language, I spent a lot of time writing a static compiler for Groovy, making it a pretty unique language in the JVM world, but not limited to the JVM world:

Groovy is a language which supports dynamic typing and compile-time type checking. It makes it surprisingly powerful and versatile language, capable of adapting to a great variety of contexts.

When I tell people that I wrote the static compiler for Groovy, I often get a reaction which is "so you admit that dynamic languages are less powerful than static ones", and they see me as the one that made the language right. Oh no, I did not. In fact, I love the dynamic aspects of the language. It is always annoying that I, as a designer of a static compiler, have to defend dynamic languages, but it’s an interesting topic, especially those days where I read lots of articles doing dynamic-bashing.

So in this post, I’m going to illustrate 10 things that a static language (most likely) cannot do. It doesn’t mean that there are only 10 things that a static language cannot do compared to a dynamic one, but it is here to illustrate the fact that this idea that static languages are superior or more scalable just because they are type safe is IMHO stupid. Compare languages between them, but do not compare categories of languages. While static languages will be excellent in making type safety guarantees (errors at compile time), dynamic languages are often far superior in cutting down verbosity. That’s just an example which illustrates that comparing on the sole aspect of type safety is not enough.

In this post I will also illustrate how Groovy is special, because it is capable of handling things in a mixed mode, making it totally unique and incredibly powerful, bringing the best of the two worlds together. Last disclaimer, this post is mostly centered on the JVM world, because this is the one I know the best.

As expected I got lots of comments on various social media. Some are positive, some not, that’s ok, but again, I would like to remember that this is not static languages bashing, nor dynamic languages promotion. Maybe some of you will think "hey, but my static language can do it!" and yes, it is possible, because as the subtitle of this post says, mine can too. But when it does, often, there’s a drawback (like loosing type safety, decreased performance,… ) or you fall on dynamic behavior without even noticing it. When it is the case, I tried to be honest and tell about the possibilities. But I also voluntarily hide some static features of Groovy that make it a very interesting solution (flow typing for example). Last but not least, I am not saying that all dynamic features implement all those items. I am saying that by nature a dynamic language make it possible. An possible doesn’t mean required. So please read carefully before screaming!

Ready?

10. Object-oriented programming done right

This is a very academic point of view. Users of static languages tend to think that their language is object-oriented. Because C++ has a compiler, because Java has a compiler, means that statically typed languages have to be compiled. Object oriented programming does not require languages to be compiled. OOP is about message passing. Object-oriented programming does not imply type safety. It means that an object is something that receives messages. In the Java world, the message is a method with arguments, and the contract is enforced at compile time, but it is an implementation detail which reduces OO programming to a small subset of its capabilities.

The great Alan Kay himself explains it.

Groovy, as a dynamic language, supports commons OOP concepts (and also functional concepts) like class, interface, abstract classes or traits but also has a meta-object protocol. For those of you who did Smalltalk programming, it’s the same idea: the behavior of objects is not determined at compile time, it’s a runtime behavior determined by a meta-object protocol. In Groovy, it translates to the fact that to each class corresponds a meta-class which determines the behavior that an object will have when it receives a message (a method call).

This capability of doing things at runtime instead of compile time is the core of many features of dynamic languages and most of the points illustrated in this blog post derive from it.

I had some comments that the fact that dynamic languages can do OO right wasn’t really interesting. In fact, I insisted on keeping this because this is actually what makes most of the following items possible. So think of 10. as the basement for most of the following items.

9. Multimethods

Java supports overloaded methods. The question whether it is a good or a bad idea is beyond the scope of this post (and believe me it is a very interesting question both in terms of semantics and performance). The idea is that an object can have two methods of the same name accepting different parameters. Let’s illustrate this with Java code:

public static int foo(String o) { return 1; }
public static int foo(Date o) { return 2; }
public static int foo(Object o) { return 3; }

Then you call it like this:

public static void main(String[] args) {
    Object[] array = new Object[] { "a string", new Date(), 666 };
    for (Object o : array) {
        System.out.println(foo(o));
    }
}

What do you think it prints? Well, most beginners will probably answer something that looks natural when you know the contents of the array:

1
2
3

But the correct answer is:

3
3
3

Because the static type of o when the call to foo is made is Object. To say it more clearly, the declared type of o is Object so we are calling foo(Object). The reason for this is that the code is statically compiled so the compiler has to know at compile time which method is going to be called. A dynamic language like Groovy chooses the method at runtime (unless, of course, you use @CompileStatic to enforce static semantics), so the method which is going to be called corresponds to the best fitting arguments. So Groovy, unlike Java, will print the less surprising result:

1
2
3

It is theorically possible for a static language to do the same. But it comes at the price of performance. It would mean that the arguments have to be checked at runtime, and since static languages do not, as far as I know, implement an inlining cache, performance would be lower than those of a well designed dynamic language…

But to add something to a dynamic language, what if you remove the Object version of foo, and remove 666 from the array? As an exercise to the reader, would this Java code compile?

public static int foo(String o) { return 1; }
public static int foo(Date o) { return 2; }

public static void main(String[] args) {
    Object[] array = new Object[] { "a string", new Date() };
    for (Object o : array) {
        System.out.println(foo(o));
    }
}

If not, what do you have to do to make it pass? Yes, dynamic languages are superior here…

8. Duck typing

Duck typing has always been a selling point of dynamic languages. Basically imagine two classes:

class Duck {
   String getName() { 'Duck' }
}
class Cat {
   String getName() { 'Cat' }
}

Those two classes define the same getName method, but it is not defined explicitly in a contract (for example through an interface). There are many reasons why this can happen. For example, you didn’t write those classes, they are in a third party library and for some reason those methods were not intended to be part of the contract. Imagine that you have a list of objects containing either ducks, cats, or anything else definining a getName method. Then a dynamic language will let you call that method:

def list = [cat, dog, human, hal]
list.each { obj ->
   println obj.getName()
}

A static language like Java would force you to have a cast here. But since you don’t have an interface defining getName and implemented by all objects, you cannot cast to that type so you have to consider all types and delegate appropriately like in the following code:

if (obj instanceof Cat) {
   return ((Cat)obj).getName();
}
if (obj instanceof Duck) {
   return ((Duck)obj).getName();
}
if (obj instanceof Human) {
   return ((Human)obj).getName();
}
if (obj instanceof Computer) {
   return ((Computer)obj).getName();
}

The real solution in Java is to define either a common super class or an interface for all those, but again, sometimes you just cannot because you don’t have access to the code! Imagine that the Cat and Dog classes where designed like this for example:

public abstract class Something {} // should define getName, but does not for some obscure reason
public class Cat extends Something {
   public String getName() { return "Cat"; }
}
public class Dog extends Something {
   public String getName() { return "Dog"; }
}

Often the developer didn’t even realize that all objects share a common interface. That’s bad for you, and if you find this code you have no choice but the cascading instanceof solution. There are multiple issues with that code:

it is very repetitive, the only thing which changes is the type used in the test and the cast
it has to be extensive, that is to say that if your list happens to contain an object having a getName method but not in your list of cases to consider, the code is broken. This means that you have to think about changing that method if you add a new type in your list.
in the JVM world, as the number of cases to consider grows, the size of the method will increase to the point where the JIT (just-in-time compiler) decides it’s not worth inlining, potentially dramatically reducing performance.

Of course, one may say "but why the hell didn’t you use an interface". This is of course a good way to solve this in Java, but it is not always possible. Not for example if you don’t have access to the source code (think of the various classes being split in third party libraries). I often faced this problem in the past, and believe me it’s no fun (I look at you, Apache Lucene).

There are actually alternatives for static languages. In Java, you could use a reflective proxy: define an interface, then create a proxy implementing that interface which will delegate to the appropriate getName method. Of course it is overkill: for each object of your list you have a proxy instantiated… Another option, again in Java, is to make the call reflective. But in that case, the call becomes slow and in fact, what you are doing is a dynamic call like a dynamic language would do. A language like Groovy doesn’t have that problem because it implements smart techniques like call site caching and runtime bytecode generation which make it much faster than what a reflective call would do…

An elegant alternative used by other static languages is structural typing. This is for example what the Go language does. In this case, you define an interface, but the object does not have to explicitly implement the interface: the fact that the object defines a method corresponding to the method in the interface is enough to implement it. This is elegant but it changes the semantics of an interface as you define it in Java. Last but not least, this technique cannot be used on a platform like the JVM, because the virtual machine has no way to do it. Well, this is not totally true since now we have the invokedynamic bytecode instruction but guess what? You are relying on a dynamic feature of the VM… Can you hear it?

Some argued that this is very bad design. I must repeat that if you think so, you missed the point. The idea is to workaround poorly designed APIs (or APIs which were "optimized"). When I talked about Lucene it was for a very good reason. I faced the problem. Lucene is a highly optimized piece of code. It makes design decisions which are often based on performance: flattening as much as possible class hierarchies (the HotSpot JIT doesn’t like deep class hierarchies), make classes final, prefer abstract classes over interfaces, … So it is easy to find classes that you want to extend, but you can’t because they are final, or classes that implicitly implement a contract but do not define interfaces. This is a pain to work with, and the ability of a dynamic language to be able to call such methods without having to explicitly declare a contract is a real gain. Some static languages offer similar features through structural typing, but then you have to think about what it means (virtual table lookup?) and how it is implemented depending on the platform (on the JVM, relying on reflection is possible but you loose all type safety and have very bad performance). So everytime I used duck typing, it wasn’t on APIs that I had designed. It was on 3rd party APIs, that for some reason didn’t provide me with a way to call some methods.

7. Respond to non-existing methods

A dynamic language answers to messages (method calls) at runtime. This means that a well designed dynamic language should be able to let you answer any kind of method call, including… non existing methods! This feature is at the core of powerful facilitating frameworks like Grails. In Grails, you can define a domain class like this:

class Person {
   String firstName
   String lastName
   int age
}

The Person class does not define any method, nor does it have any explicit relation to a datastore, an ORM or SQL server. However, you can write code like this:

def adults = Person.findByAge { it>= 18 }

I will not dig into the details about how this is done, but the idea is to intercept the fact that the findByAge method does not exist, then parse the name of the method and build a query based on the method name and the rest of the arguments (here, a closure, an open block of code). Queries can be as complex as you wish, like findByLastNameAndAge or whatever you can think of. Of course Grails does some smart things here, like generating a new method at runtime, so that the next time this method is hit, it is not an unknown method anymore, and can be invoked faster! Only a dynamic language would let you do that. Say bye to infamous DAOs that you have to change everytime you have a new query, it is not necessary. One could say that they prefer safety at compile time rather than the ability to do this, but Grails also offers that possibility of checking that the syntax is correct at compile time, while still leveraging the dynamic runtime to make this work… It’s all about boilerplate removed, code verbosity and productivity…

The ability to react to arbitrary messages is actually at the core of many DSLs (domain specific languages) written in Groovy. They are at the core of builders for example, which will let you write code like:

catalog {
   book {
   	isbn 123
	name 'Awesome dynamic languages'
        price 11.5
        tags {
	   dynamic,
	   groovy,
	   awesome
	}
   }
}

Instead of the less readable Java 8 version (for the reader’s mental sanity, I will not write the Java 7 version):

builder.catalog( (el) -> {
  el.book ( (p) -> {
     p.setISBN("123");
     p.setName("Awesome dynamic languages");
     p.setPrice(11.5);
     p.setTags("dynamic","groovy","awesome");
  })
});

6. Mocking and monkey-patching

Mocking is at the core of many unit testing strategies. Most of static languages make use of an external library to do this. Why this can be true of dynamic languages too, this is often not strictly necessary. For example Groovy offers built-in stubbing/mocking capabilities, very easily thanks to its dynamic nature. Monkey patching rely on the very same behavior but is easier to explain so I will illustrate this concept here. Imagine that you use a closed-source library (I won’t judge you, I promise) or an open-source library for which you don’t want to/don’t have time to contribute to, but you have found a serious security issue in a method:

public class VulnerableService {
   public void vulnerableMethod() {
      FileUtils.recurseDeleteWithRootPrivileges("/");
   }
}

You know how to fix it, but you have to wait for the maintainer to upgrade the library. Unfortunately, you can’t wait because attackers are already leveraging the vulnerability on your production server (yeah, they like to). One option that a dynamic language can let you do is redefine the method at runtime. For example, in Groovy, you could write:

VulnerableService.metaClass.vulnerableMethod = {
   println "Well tried, but you have been logged to Santa's naughty guys list!"
}

Then a caller that would call the vulnerableMethod would call the monkey-patched version instead of the original one. Of course in a language like Groovy, this would only be true if the callee is dynamically compiled: if you use @CompileStatic to behave like a static compiler, you’re out of luck, because the method which will be invoked is selected at compile time, so you will be vulnerable even if you try to monkey patch… Groovy provides other extension mechanisms to work around this, but it’s not the topic here ;-)

5. Dynamic objects

Most dynamic languages let you create… dynamic objects. It is basically an object for which you attach methods and properties at runtime. Not that I am a big fan of it but there are some valid use cases (serialization, languages like Golo not supporting classes, prototype based construction, …). It can also be convenient if you want to rapidly prototype a class.

As an example, let’s see how you could create an arbitrary object to represent a person, without actually leveraging on a class, using the Groovy language:

def p = new Expando()
p.name = 'Cédric'
p.sayHello = { println "Hello $name" }

p.sayHello()

The code is totally dynamic here. It lets you create an arbitrary object, attach new methods to it, data, …, without relying on strong typing. Of course it is interesting when you see that the sayHello method is capable of referencing "pseudo-fields" which are themselves dynamic!

4. Scripting

Static languages can do scripting. But it is definitely not what I would call scripting. Having to write types is not natural in a script. I even worked in the past in a context where people who wrote scripts where not programmers. They didn’t even know what a type is, and they don’t care. The most popular scripting technologies like Bash do not have types, and it’s not a problem, so imagine the following. You arrive late at your office, your boss is very angry about that and shouts to you: "you have 5 minutes, not more, to give me the total number of followers of users who have submitted an accepted pull request on the Groovy repo recently". It’s a weird query, most probably your boss is going into social networking madness but you have no choice otherwise you’re fired.

In that case, most developers would think of:

using a Bash script combining curl, grep, regular expressions and hoping that man works
using a tool they know like Java, but since they have so little time, they will probably rely on a regular expression to parse the JSON feed until they realize they have to do a second HTTP query for each user
quiting their job

In Groovy, you would do:

import groovy.json.JsonSlurper

def json = new JsonSlurper().parse('https://api.github.com/repos/groovy/groovy-core/issues?state=closed'.toURL())
json.user.collectEntries { u ->
   // second query to fetch the nb of followers
   def followers = new JsonSlurper().parse(u.followers_url.toURL())
   [u.login,followers.size()]
}.values().sum()

What you can see here is that we use a facility, JSonSlurper which actually parses the JSON result. It is much more reliable that what you would have done with a quick hack like a regex, but not only:

all data is accessible in a path-like fashion (json.user.address.city.postalCode)
you don’t need a single type here

Even if you use a smart JSON parser with your static language, you would still have to write a collection of classes to unmarshall the JSON structure into beans/classes. For such a simple use case, you really don’t care. You just want things done, easily, quickly. You don’t need type safety. You don’t need it to be super clean and tolerant to future changes of the JSON format. Get. Things. Done. (and boss happy).

3. Runtime coercions

Another thing that dynamic languages are particularily good at is runtime coercions. In general static languages users know about one type of conversion, which is casting. Some are lucky enough to know about coercion (like the use of implicit in Scala), the others rely on the adapter pattern. In a dynamic language, runtime coercions are often easy to implement. A coercion differs from a cast in the sense that you want to convert an object of class A to an object of class B, but a B cannot be assigned to an A.

Groovy provides "natural" conversions for some widely used types: lists to objects, and maps to object, like in the example here:

Point p = [1,2] // coercion of a list literal into an object of class Point thanks to constructor injection
Point p = [x:1, y:2] // coercion of a map literal into an object of class Point thanks to setter injection

But if it happened to be that you cannot use maps or lists but really want to convert one type to another, you can just declare a converter:

class A {
   Object asType(Class<?> clazz) { new B(...) }
}

I can see you raising an eyebrow here, because I wrote the conversion code directly in class A, but remember it’s a dynamic language with a meta-object protocol, so nothing prevents you from writing this conversion code outside of the class A itself, through its metaclass, which would let you add conversion algorithms for classes which are beyond your control. It’s a win!

2. Dynamic binding

Dynamic binding is linked to DSL evaluation and scripting. Imagine the following script:

a+b

In this script, variables a and b are unbound. They are not known from a compiler, so if you tried to statically compile this with a compiler like Java (or C++, or Scala) it would definitely blow up. Not if you compile this with Groovy. Because it’s dynamic, it’s able to know that those variables will be eventually bound, when the script is executed. Groovy provides means to inject those variables when you need them. It is some kind of late binding, but it is the core of expression languages, and it is no surprise that products like ElasticSearch uses Groovy as the default scripting language: it allows it to be both compilable and late bound. But there is more, if you think you have an issue with not being able to resolve a and b at compile time and that you fear to write code which might fail at runtime…

1. Mixed mode compilation

The last thing that a dynamic language like Groovy is capable of doing is leveraging mixed mode compilation. Behind this curious term is a unique concept in programming languages: Groovy is able of mixing static code with dynamic code, but more, you can instruct the compiler how to do so. So if you design a DSL like in ElasticSearch where you know that some variables will be bound, that the number, names and types of those variables are fixed and known in advance, then you can instruct the compiler and switch to a statically compilable mode! This means that if the user uses an unknown variable, compilation will fail.

This technique is already used in Groovy itself, in the powerful Markup Template Engine. It is a template engine which is capable of generating markup-like contents with a very nice builder-like syntax, but all templates are statically compiled even if the code seems to be full of unresolved method calls or variables!

For those who are interested in this, I invite them to take an eye at my blog posts describing how you can do this.

Conclusion

In conclusion, I have highlighted some points where dynamic languages can do what static languages cannot. Users of the most widely used dynamic language, Javascript, probably have lots of ideas too. The point for me is not to tell which one is better than the other because I don’t care. In general, I am not much into the war behind those, because I really enjoy both. I do static typing most of time, but I really enjoy the dynamic nature of the language too because often I don’t want to be slowed down just to make a compiler happy. I, as a developer, should be happy. Making a compiler happy is secondary and often not necessary. Last but not least, you might have thought, reading this post, that your static language can do this or that. I won’t blame you here, because mine can too. The idea here is more to show that it is totally unnatural for a static language or it often comes with horrible drawbacks like verbosity, performance issues or simply difficult to implement.

So what about using a language that brings the best of the two worlds together?

Fixing Bluetooth debugging on Android Wear

Mon, 20 Oct 2014 00:00:00 +0200

I have been working on making Groovy work on Android for several months now. In the last weeks, I even showed at SpringOne2GX an example of an application written in Groovy that worked on Android Wear. However, that code worked in an emulator. Recently, I got a real device, a Moto 360, so I wanted to see that application running on a real device. For some very obscure (understand buggy Android SDK) reason, it was far from being that easy…

I litteraly spent hours trying to figure out what was wrong, so I thought it would be interesting for those of you who face the same problem to have a blog post that explains how to deal with it.

The problem

If you follow the instructions on the Android documentation about how to enable bluetooth debugging, it’s in the end pretty simple. Basically, it’s about enabling USB debugging on your physical handheld, then enable bluetooth debugging on your wearable, and in the end enable bluetooth debugging in the Android Wear companion app.

The guide says:

In the Android Wear companion app, you should see the status change to:

Host: connected Target: connected

However, whatever I did, the Target: connected line never appeared for me. It was always Target: disconnected, so if I continued with the instructions:

adb forward tcp:4444 localabstract:/adb-hub; adb connect localhost:4444

Doing

adb devices

Only showed my wearable as offline:

List of devices attached

dcfbbafd	device
localhost:4444	offline

I have searched for an answer in a lot of pages, including Stackoverflow where I tried very unlikely answers like in this page: execute both commands separately instead of doing them in the same line. I thought that maybe there was a timing issue and that the fact of separating both commands would give the toolkit a chance, but no, wasn’t that easy.

In the end, I was totally convinced that the problem was because I had previously associated my handheld with an emulator. I was convinced of it because even if I had now associated it with a real device (the Moto 360), in the Android Wear companion app, the device was recognized as an "emulator"… mmm… So I tried uninstalling the Android Wear app, clear its cache, but no matter what I did, after reinstalling, the settings were kept, and the Moto recognized as an emulator. So sad…

The solution

So you were looking at a solution, and here it is. Basically, the problem is that the Android companion app doesn’t store its settings under its cache. They are stored in the Google Play Services space, so here is the procedure that worked for me, and I sincerely hope it will do for you. On your handheld:

open the applications settings, search for the Android Wear application, then force stop it
clear its data and cache
now search for Google Play Services (depending on your language settings, it can appear with a different name, on m y device it is "Services Google Play".
click on Manage space. You will see that there’s a section for connected devices. I tried to clear data here, but it didn’t help, so you have to click on delete all data.
reboot your phone
reboot your Moto 360

I am unsure that the two last steps are really necessary, but I did it because I wanted to make sure that force stopping and clearing data did not introduce some weird behavior after that. When you reopen the Android Wear companion app, it should now be as if it was the first time you opened it and ask you to associate it with your watch. Do it, and now, you should be able to follow the normal procedure described in the Android documentation and…

Success, now you can debug your application on a real device (which includes deploying it…).

Hey, finally got it working! I need to blog about how I arrived the insane emulator issue... #groovylang pic.twitter.com/XINccF3V7k
— Cédric Champeau (@CedricChampeau) 19 Octobre 2014

Tip of the day: reversed git bisect

Wed, 23 Jul 2014 00:00:00 +0200

I had an interesting use case for git bisect today and as my blog also consistutes a good archive for things I don’t want to loose, let’s take advantage of this to share the trick with you!

Normally, git bisect is used to find what commit introduced a regression in the codebase. For example, if you know that current HEAD is buggy but that at least, RELEASE_1_0 was good, then you can write:

git bisect start             (1)
git bisect bad               (2)
git bisect good RELEASE_1_0  (3)

1	start bisecting
2	tells that `HEAD` contains the regression
3	tells that `RELEASE_1_0` is a tag corresponding to a version known not to have the bug

Git will checkout a revision that you can test, and you issue a list of git bisect good or git bisect bad commands until it determines what commit introduced the regression.

This is a very practical way to find a regression. In Groovy, I’ve used this more than once, it’s very useful.

Reversing the logic

But today, I wanted to reverse the logic. Actually, we had a bug report and we found out that the bug was already fixed, but we didn’t know in which version it was fixed. So actually, I didn’t want to find a regression, but a fix commit.

The idea to do this is to reverse the meaning of bad and good in bisect:

bad becomes "doesn’t produce a compile error"
good becomes "produces a compile error"

And since the range of revisions to test was pretty big (we know that the error was reported on Groovy 2.2.1, but master is 2.4.0), then I also took advantage of the git bisect run command, which automatically continues bisecting based on a command line return status code.

So basically, here’s what I wrote:

git bisect start                (1)
git bisect bad master           (2)
git bisect good GROOVY_2_2_1    (3)
git bisect run ./bisect.sh      (4)

1	start bisecting
2	master is known to have the fix, so we say bad is master
3	GROOVY_2_2_1 is known to have the bug, so we say good is GROOVY_2_2_1
4	start automatic bisecting thanks to the ./bisect.sh script

And what does bisect.sh consist of? Here you go:

bisect.sh

#!/bin/bash
export GROOVY_HOME=/tmp/testversion                                                                     (1)
./gradlew clean -x javadoc -x groovydoc -x javadocAll -x groovydocAll -PskipIndy=true installGroovy     (2)
/tmp/testversion/bin/groovy bisect.groovy || exit 0                                                     (3)
exit 1                                                                                                  (4)

1	tells where the local build version of Groovy will be installed
2	builds Groovy and installs it locally
3	executes the test script and if the test fails, return a success exit code
4	return a failure exit code

The trick is to reverse the exit codes in the script too: if the script compiles, then it means that the bug was fixed. Since we reverse the logic, we then need the script to return a bad exit code! In case the script fails, we will return a success (0) error code, because it means that the revision doesn’t have the fix. Easy, but needs some mental contorsion :-)

You will note that this script uses GROOVY_HOME and a local installation path. You can configure it using the $HOME/.gradle/gradle.properties file, and adding the following line in it:

gradle.properties

groovy_installPath=/tmp/testversion

Eventually, here is the groovy script which served as a test case (almost copied directly from the JIRA issue):

bisect.groovy

abstract  class Base<A> {
    abstract  void foo(A[] a)
}

class X {}

class Inheritor extends Base<X>{
    @Override
    void foo(X[] a) {}
//Groovyc: Can't have an abstract method in a non-abstract class.
//The class 'B' must be declared abstract
//or the method 'void foo([Ljava.lang.Object;)' must be implemented.
}

Inheritor

Note that each revision took around ~1 min 30s to test, even skipping the javadoc/groovydoc and indy versions of Groovy, so you can imagine what benefit you have here in using automatic bisecting.

In the end, after around 20 minutes of automatic processing, I received this nice message:

74d991f9f8c39d2730a054431bf28e6516e61735 is the first bad commit
commit 74d991f9f8c39d2730a054431bf28e6516e61735
Author: Cedric Champeau <cedric.champeau@gmail.com>
Date:   Sun Apr 27 18:06:24 2014 +0200

    GROOVY-6722: Compiler doesn't handle generic array covariant

:040000 040000 b61b92399ac86246c157f948b3232bf5ab0cf04f 5d0c56413a621c7693e7985aff0e4c84eb08889f M	src
bisect run success

What? "bad commit"? Yes, remember that the logic is reversed, so "bad" means actually "fixed it". So it says that the first commit which fixed the bug was actually 74d991f. And here we go, issue closed ;) One improvement I can see is to use a local clone of my repository instead of working directly in the same repository, so that I can continue working on my copy while bisecting is in progress.

The new Groovy website

Fri, 18 Jul 2014 00:00:00 +0200

Last week, we revealed the beta of a brand new website for the Groovy language. This new website is open sourced and already received a few contributions. In order to make it even easier and as it a fully statically generated site that makes use of Groovy I wanted to give more technical details on the toolchain and how it is generated.

A static website

One of the first questions which arised was: why not use Grails/Spring Boot/Ratpack? In fact, the new Groovy website is fully statically generated. It offers multiple advantages:

hosting is much easier, as it only consists of static pages and assets
maintenance is simplified, no database to backup for example
everything is self contained, pages and data, into a single repository
no need for authentication
all content is public

Last but not least, we didn’t have any requirement for storing anything in a database, or that would require dynamic generation. Pull requests are enough so far. Eventually, we’re thinking about a blog, but even that can be statically generated even if you want to allow users to comment on articles (this blog is a perfect example). So in short, this decision was motivated by one mantra: the right tool for the right job.

The documentation, that you can find on this page, had already started migrating from the aging wiki to . It is generated independently of the website and integrated into it using iframes (we’re thinking about source integration though).

Structure of the project

Generator and site

The project is built using Gradle 2 and consists of 2 subprojects:

the generator project contains, as the name says, the static generator. It makes use of a template engine and provides the classes used in the model of the templates.
the site project contains the templates and data. If you’re looking into contributing contents, this is likely the place to look at.

Building and testing the site is easy:

git clone https://github.com/groovy/groovy-website.git          (1)
cd groovy-website
./gradlew generate                                              (2)

1	clones the repository
2	generates the website from templates

The output will be visible in <project directory>/site/build/site. There’s also a checkDeadlinks tasks that we will use once we get out of the beta phase to ensure that the generated pages do not contain any dead link.

Internally, we use our CI server to deploy changes to the master branch live. So any commit which is pushed onto the master branch is automatically published (in general, takes less than 2 minutes).

Adding contents

Even if the site is statically generated, we still have data. In this project, there’s an important file, named sitemap.groovy which handles a lot of the contents of the website. It is our "low cost" database and as you can see, it’s a DSL describing the contents of the website.

For example, you can see the menu section which looks like this:

menu {
    group('Groovy') {
        item 'Learn',                       'learn.html'
        item 'Documentation',               'documentation.html'
        item 'Download',                    'download.html'
        item 'Community',                   'community.html'
        item 'Ecosystem',                   'ecosystem.html'
    }

    group('About') {
        item 'Contributing',                'contribute.html'
        item 'Source code',                 'https://github.com/groovy/groovy-core'
        item 'Books',                       'learn.html#books'
        item 'Sponsors',                    'sponsors.html'
        item 'FAQ',                         'faq.html'
        item 'Search',                      'search.html'
    }

    // ...
}

It is a purely declarative description of the site menus. Actually, the "Groups" are used in the footer of the page, while the main Groovy group is used to generate the top navigation bar. Using a simple descriptive DSL is very interesting, because it decouples templates from the contents of the menu. We make sure that those templates do not contain any element which is hardcoded and reduce the risks of forgetting to update the footer, for example, if a section is added.

The same file is used to describe the list of downloads:

downloads {
    // ...
    distribution('Groovy 2.3') {
        description {
            yield 'Groovy 2.3 is our latest official '
            a(href: 'versioning.html', 'version')
            yield ' of Groovy.'
        }

        version('2.3.4') {
            stable true
            releaseNotes 'https://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=10242&version=20432'
            windowsInstaller 'https://dist.codehaus.org/groovy/distributions/installers/windows/nsis/groovy-2.3.4-installer.exe'
        }
    }
    // ...
}

or the books which are listed on the learn page:

books {
    book('Groovy in Action, 2nd edition') {
        authors "Dierk König, Guillaume Laforge, Paul King, Cédric Champeau, Hamlet D'Arcy, Erik Pragt, and Jon Skeet"
        cover 'img/books/regina.png'
        url 'https://www.manning.com/koenig2/'
        description 'The undisputed definitive reference on the Groovy programming language, authored by core members of the development team.'
    }
    // ...
}

The same is done for all contents that need regular updates: user groups, events, projects of the ecosystem, … I think this DSL provides a very nice way to add contents to the website without caring about where it has to be done. You can really think of it as a small database, but making use of a Groovy DSL.

In addition, this file also declares the mapping between pages in the documentation section and the documentation page. Last but not least, it lists the individual pages that the website contain. Those pages make use of the markup template engine.

Eat your own dog food

In Groovy 2.3, we introduced a new markup template engine. We decided that the new web site was an excellent showcase of this template engine, and a real life use case. This template engine has several remarkable features, like static compilation of templates (even if the model is dynamic), layouts and of course a human readable builder like syntax:

html {
   head {
    title 'Groovy markup template engine in action!'
   }
   body {
    ul {
        features.each { f-> li(f.name) }
    }
   }
}

It has already been integrated into Spring Boot and Ratpack will use it in the next version (to be released on August 1st). A hint about its performance can be found here. If you are interested in details about how it works, you can find the documentation here and you can read my blog posts about it.

The website subproject is therefore organized accordingly. Inside the main source tree, you’ll find the following directories:

assets: contains static assets, like Javascript, CSS, images, …
html: contains pure HTML files which are easier to embed as is than using a markup syntax
includes: contains elements of code which are shared among multiple templates
layouts: contains template layouts, as defined in the documentation
pages: contains the main pages of the website

In general, consider pages as the entry point. A page generally makes use of one layout. As an example, let’s see how the Ecosystem page is generated. The source file consists of this:

ecosystem.groovy

layout 'layouts/main.groovy', true,                                             (1)
    pageTitle: 'The Groovy programming language - Ecosystem',                   (2)
    mainContent: contents {                                                     (3)
      div(id: 'content', class: 'page-1') {
        section(class: 'row') {
          div(class: 'row-fluid') {
            // ... snip side menu ...
            div(class: 'col-lg-8 col-lg-pull-0') {
              include template: 'includes/contribute-button.groovy'             (4)
              h1 {
                i(class: 'fa fa-leaf') {}
                yield ' Ecosystem'
              }
              p {
                yield '''
                    Beside the language and its API, Groovy gave birth   ...
                    on various themes such as web frameworks, desktop    ...
                    In this section, we will highlight a few of the most ...
                    which leverage Groovy at their core.
              '''
              }
              hr(class: 'divider')

              ecosys.eachWithIndex { e, index ->                                (5)
                def (name, item) = [e.key, e.value]
                article {
                  a(name: "${name}") {}
                  div(class:"content-heading clearfix media") {
                    div {
                      if (item.logo) {
                        img class: "pull-${(index % 2 == 0) ? 'left' : 'right'}",
                          src: item.logo, alt: name, hspace: '20px'
                      } else {
                        h2(name)
                      }
                      p(item.description)
                    }
                    a(href: item.url, target:'_blank', "Learn more...")
                  }
                }
                hr(class: 'divider')
              }
              // ...
            }
          }
        }
      }
    }

1	make use of the `main` layout
2	the layout requires a `pageTitle` variable
3	as well as a `mainContent` section corresponding to the main page contents
4	example of use of an `include`
5	iterates over the `ecosys` variable which contains the list of ecosystem projects as found in the sitemap

As you can see, this template format has the advantage of taking care of generating markup for you. You won’t hit your head again on the wall to find an unclosed tag. Everything is embedded, readable and concise.

Lessons learnt

Using the markup template engine for this project was interesting, because it was probably the first "real life" project to use it intensively. And as such, we discovered usability issues, but also bugs. Hopefully, none of those bugs or usability features were critical, and everything could be worked around, but expect some fixes in Groovy 2.3.5. It is also the reason why the project initially used Gradle 2: it comes with Groovy 2.3.2 which embeds the markup template engine, so it was possible to use it without organizing the project into separate modules like we have. In fact, the early versions of the site didn’t use subprojects. It’s only when we wanted to leverage improvements from Groovy 2.3.4 that we had to switch to that architecture.

A team work

In the end, I can’t finish this blog post without mentionning the team work it implied. In particular:

Damien Vitrac designed the website and produced HTML sketches. If you think the new site looks good, thank this guy!
Guillaume Laforge designed the site architecture, wrote contents, tweaked the CSS, that is to say produced almost all contents. He spent countless hours fixing responsiveness issues and digging into front-end dev.
I streamlined the process by setting up the Gradle project, designing the sitemap DSL, the integration of the markup template engine, CI integration, … that is to say pretty much all the "backend" stuff.
You, as a community, provided awesome pull requests within hours. Keep them coming, we love it!

Each of us have different skills. Guillaume is far better than I am in any kind of web design, styling issues, etc for example, so in the end, I think the combination works quite good and that the site as it is now is already pretty usable.

Let us know what you think, and don’t forget that you can contribute, it’s easy!

Contributing for the Groovy language: setting up IntelliJ IDEA

Tue, 24 Jun 2014 00:00:00 +0200

Often people ask us how you can setup a development environment to contribute on the Groovy project. If you use IntelliJ IDEA, it’s actually very easy, so I decided to make a video for it. It’s my first screencast ever, I think I will try to do more in the future, so please excuse my hesitations and enjoy! You’ll notice that apparently Google Hangouts dislikes my face, but hopefully, the screensharing stuff worked ;-)

Here’s the video:

Of course, you can ask me if you have issues with the setup, but it should really be straightforward!

Groovy on Android, technical details

Tue, 10 Jun 2014 00:00:00 +0200

In my previous post, I have introduced how we could now use the Groovy language to develop Android applications. In this post, I will give you more details about how it works internally, giving you more hints about what makes it possible.

Compiling Groovy

The Groovy language is a JVM language which compiles to bytecode. Even if it has scripting capabilities, it’s always compiled. This means that in Groovy, you have two options: either a class is compiled to a .class file and used like any other class file on the JVM, using the groovyc compiler instead of javac, or a class can be compiled at runtime, for example if you rely on scripting. In the last case, the source script (or any Groovy source in general, not necessarily scripts) is available at runtime, and you rely on APIs that Groovy provide to compile those sources and execute them while your application is running. This is typical of scripting languages, but you must be aware that in any case, Groovy is not an interpreted language: everything is turned into bytecode.

A classic Android application

Since the Dalvik VM doesn’t use the same bytecode as the JVM, Android requires a bit of work in order to compile and execute Java programs. A special tool, called dex, is responsible for converting the JVM bytecode into Dalvik bytecode, and compile it into a classes.dex file. This is illustrated in the following schema:

In our case, we’re using the Gradle build tool, which is now the default for Android projects, so Gradle is responsible for this chain. Java sources are compiled into .class files, then those classes are processed by the dex tool, which converts bytecode for all classes and packages everything into a classes.dex file. Some additional steps can be done (such as calling ProGuard to reduce the size of the package), but in the end, an APK is produced, which is the deliverable application. When deployed on the device, there’s nothing else to do than loading the classes and executing the application.

Let’s see how the process differs in case of a Groovy application.

An application written in Groovy

In this case, we have .groovy files, corresponding to Groovy sources, but we may also have .java files. In the end, the process looks very similar:

This schema exactly illustrates how the GR8Conf Agenda application is compiled. All is done at compile time, and nothing more at runtime. Groovy sources are compiled into JVM bytecode, which is in turn converted into Dalvik bytecode using dex. There’s absolutely no difference with Java, apart from the compiler, which is able to process both Groovy and Java sources!

One noticeable difference that some people have already reported to me is that since we embed the Groovy runtime, the number of method descriptors used in the classes.dex file is significantly higher than with an application written in pure Java. From my early tests, a Groovy application would consume around 8k methods, without optimizations. The classes.dex file has a limit of 65536 methods, so this is something you have to keep in mind. Anyway, I am not an Android specialist, but there seem to be workarounds available, like the multi-dex option.

In the end, Groovy is not different from any other library you would embed in your application, it’s "just" an additional jar. I also mentionned the fact that I recommended to use @CompileStatic on your classes. There’s a good reason for that. If you don’t, the classes that you will compile will use the dynamic runtime of Groovy. This is unlikely what you want on a mobile application, especially because on Android, it would use reflection, implying a major performance drop. On a normal JVM, Groovy would use call site caching techniques, like generation of classes at runtime or invokedynamic to improve performance. This is not possible on Android, so places where you use dynamic features of Groovy should be limited to small parts of the application, called not frequently. A good example would be the use of a builder for the UI. Builders are a very nice feature of Groovy, and for a UI, it would only be called once when the activity is loaded.

Using @CompileStatic, you will ensure that your classes are statically compiled, meaning that all method calls are resolved at compile time, leading to dramatically improved performance, very close, if not equal, to that of Java (depending on how you write your code, as usual, because idiomatic Groovy might not always be the fastest implementation).

In any case, you must recall that the first application that I wrote does not use a single class generated at runtime. Even if you remove all @CompileStatic annotations, it would use the dynamic runtime, but without creating classes at runtime.

Runtime scripting

One of the difficulties of adapting the Groovy language is that, as we said, Groovy is a highly dynamic language. One of its capabilities is executing scripts at runtime. So what if we wanted to type a Groovy script on the device, and execute it directly on it? Is it possible? In fact, yes, it is possible, given that you follow this process:

You can have an application, written in Groovy or not, but in the end, once the application is running, you have Groovy source code that you want to execute. Then it needs to compile the classes, call dex directly on the device, package the result into a jar file on the local filesystem, then load it using a special classloader. So why this is possible, the process is very complex, not straightforward, but more importantly, it is dramatically slow.

This process is demonstrated in this application, which replicates the behavior of the well-known GroovyShell but directly on an Android device. To give you an idea, on my own device, a Samsung Galaxy Note 3, starting the application, written in Groovy, is blazing fast. Then if you hit the "execute" button, the first time, compilation of the little script will take around 3s.

Subsequent compilations take much less time (around ~500ms), but the fact of having to dex files and write them to the filesystem is a performance killer. In anycase, this shows that it is possible, and even that you could rely on it for an application that would handle scripts. It would be possible, for example, to cache the jar files in order to avoid recompiling them…

Conclusion

In this post, I gave further details on how Groovy gets running onto an Android device. In a future post, I will give you more details about how you can get started with your own project. I will maybe give Lazybones, a project bootstraping tool, more love in the next days ;-)

Groovy on Android

Thu, 5 Jun 2014 00:00:00 +0200

Yesterday ended GR8Conf Europe, a Groovy language conference in Copenhagen that was once again a successful event. I want to thank the crew again for this, and of course the Groovy language community which is so friendly and always helpful. This year turned out to be very special for me, because just two days after the announcement by Apple of the Swift language, I was talking about running Groovy on Android! As Guillaume Laforge noticed, there are a lot of similarities between the Swift language and Groovy language.

The timing is almost perfect, because Android users will want to have a language which is as modern as Swift is, but running on Android. And I see no better candidate than Groovy language here.

The presentation

Should you be interested in the slides, you can find them below. Since it was presented at GR8Conf Europe, it gives a bit of history of the changes needed to have the runtime working on Android too:

For those of you who are new to Groovy, you have to know that this language is derived from Java, meaning that the learning curve is almost 0, but it also removes a lot of its verbosity and adds a lot of features to it, such as closures (similar to lambdas in Java 8, but Java 8 is not available for Android developers), builders, runtime and compile-time metaprogramming.

As an example of how Groovylang can be used to reduce the verbosity of Java on Android, I will take a simple example: asynchronous tasks. Asynchronous tasks are required as soon as a task takes too much time to be executed on the UI thread. That is the case, by default, for any network based operations, in order to guarantee that the UI remains snappy even if network is slow or unavailable. The problem is that those asynchronous tasks are incredibly verbose. You have to write a lot of code, that I would tend to name "inner class hell", just for this. Let’s imagine that you need to parse a JSON feed, then update the UI accordingly. Then you would have to write something like this (no kidding):

public class FeedActivity {
    TextView mTextView;

    ...

    void updateFeed() {
    	new FeedTask().execute("https://path/to/feed");
    }

    class FeedTask extends AsyncTask<String, Void, String> {
        protected String doInBackground(String... params) {
            DefaultHttpClient httpclient = new DefaultHttpClient(new BasicHttpParams());
            HttpPost httppost = new HttpPost(params[0]);

            InputStream inputStream = null;
            String result = null;
            try {
                HttpResponse response = httpclient.execute(httppost);
                HttpEntity entity = response.getEntity();

                inputStream = entity.getContent();
                // json is UTF-8 by default
                BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"), 8);
                StringBuilder sb = new StringBuilder();

                String line = null;
                while ((line = reader.readLine()) != null) {
                    sb.append(line).append("\n");
                }
                result = sb.toString();
            } catch (Exception e) {
                // Oops
            } finally {
                try {
                    if (inputStream != null) {
                        inputStream.close();
                    }
                } catch (Exception squish) {
                }
            }
            StringBuilder speakers = null;
            try {
                JSONObject jObject = new JSONObject(result);
                JSONArray jArray = jObject.getJSONArray("speakers");
                speakers = new StringBuilder();
                for (int i = 0; i < jArray.length(); i++) {
                    speakers.append(jArray.getString(i));
                    speakers.append(" ");
                }
            } catch (JSONException e) {
                // do something?
            }
            return speakers.toString();
        }

        @Override
        protected void onPostExecute(String s) {
            mTextView.setText(s);
        }
    }
}

So now, let’s see the equivalent in Groovy (no kidding either):

public class FeedActivity {
    TextView mTextView

    ...

    void updateFeed() {
    	Fluent.async {
            def json = new JsonSlurper().parse([:], new URL('https://path/to/feed'), 'utf-8')
            json.speakers.join(' ')
        } then {
	    mTextView.text = it
        }
    }
}

I think you can start to see the advantage of using Groovy in your own Android projects. Of course, the Fluent class that I’m using here is a support class which I implemented in my first Android project (which is open sourced btw), but it’s really simple and gives an example of how Android users could benefit from Groovification of their APIs.

Feeling the pain

This is actually a key point of my talk: I hadn’t written any Android application before this talk, and I definitely wanted to be able to write an application in Groovy on Android. Why? Because it’s been some time already that I use Groovy everyday, and there’s no turning back. I wanted to feel the pain of the Java developers on Android, so that I can write better tools for them. And that’s actually where you, as a user, come in action: there are so many Groovy libraries out there whose sole objective is to ease the pain, make things that shouldn’t be complicated a breeze. This is exactly the point. My example with Fluent is one example of simplification of usages of Android APIs, but you have many more to invent, especially because you must have been as annoyed as I did by all those asynchronous tasks, XML files (think of builders!), callback hell, etc…

The beginning of a new era

In upcoming posts, I will try to demonstrate that we’re just at the beginning. Some people are already asking for a {swift} alternative for Android. It’s there guys, you have it, so spread the word and let’s make it happen! I am convinced that it is the beginning of a new era for Android and Groovy. Google already switched their main build system to Gradle, which is, by the way, using Groovy, so I think it’s time to move over and show your love!

All you need to get started is explained in the slide deck above, and you can find the source code of the sample android application on GitHub:

https://github.com/melix/gr8confagenda

Update 3: it’s even simpler now

Groovy 2.4.0 has been released now and building an Android application in Groovy is even simpler. All you have to do is to apply a Gradle plugin. Instructions can be found here. You absolutely don’t have to build Groovy from sources or add tasks to your Gradle build as described originally in this post! Just use the plugin!

Information below is outdated. Please use the Gradle plugin !.

Update 2: build instructions with the Gradle plugin

You can now use a Gradle plugin to integrate Groovy with Android. The plugin can be found here: groovy-android-gradle-plugin. As of version 0.2, the plugin supports the Android plugin 0.10+.

Update: build instructions

If you want to try it by yourself, here’s how you can do it. First of all, official support for Android will be in Groovy 2.4. Before the first beta, you’ll have to build it from sources, and here is the quickest way:

git clone https://github.com/melix/groovy-core.git --branch master
cd groovy-core
./gradlew -PskipIndy=true install

Then you can clone the sample application:

cd ..
git clone https://github.com/melix/gr8confagenda.git

This contains a project that you can open using Android Studio.

If you want to use the Groovy language in your own Android project, a requirement is that it is using Gradle. If so, you can update your build.gradle file as is:

build.gradle

android {
   ...
   packagingOptions {
        // workaround for https://stackoverflow.com/questions/20673625/android-gradle-plugin-0-7-0-duplicate-files-during-packaging-of-apk
        exclude 'META-INF/LICENSE.txt'
        exclude 'META-INF/groovy-release-info.properties'
    }
}

repositories {
    mavenLocal()
    jcenter()
}

dependencies {
    compile 'org.codehaus.groovy:groovy:2.4.0-SNAPSHOT:grooid'
    // the following dependency is necessary if you want JSON support
    compile ('org.codehaus.groovy:groovy-json:2.4.0-SNAPSHOT') {
        transitive = false
    }
}

// add support for Groovy to existing configurations
android.applicationVariants.all {
    task "groovy${name}Compile"(type: GroovyCompile) {
        source = javaCompile.source + fileTree('src/main/java').include('**/*.groovy')
        destinationDir = javaCompile.destinationDir
        classpath = javaCompile.classpath
        groovyClasspath = classpath
        sourceCompatibility = '1.6'
        targetCompatibility = '1.6'
        doFirst {
            def runtimeJars = plugins.findPlugin(com.android.build.gradle.AppPlugin).runtimeJars
            classpath = files(runtimeJars) + classpath
        }
    }
    javaCompile.dependsOn("groovy${name}Compile")
    javaCompile.enabled = false
}

And that’s all! Now, one option for you is to write support libraries and make them available to the community. Enjoy!

A fast markup template engine for Groovy (part 2 of 2)

Tue, 18 Feb 2014 00:00:00 +0100

In this post, I will discuss the implementations details for the markup template engine I have described in a previous post.

Compiling templates

The MarkupTemplateEngine class

Even if my first implementation of the markup template engine was relying on StreamingMarkupBuilder, the technique used to compile templates into bytecode is actually the same after all optimizations. It relies on:

a GroovyClassLoader to create and cache template classes
a CompilerConfiguration to customize compilation

The various template engines that Groovy provide extend the TemplateEngine class. All template engines must implement the createTemplate method which returns an instance of Template. My first idea, here, was therefore to have a template engine which holds a GroovyClassLoader, and compiles scripts as Template instances. For thread safety reasons and to avoid compiling the same template again and again, I instead chose a slightly different approach, which is to compile the scripts, and cache the resulting class into a field of the StreamingMarkupBuilderTemplate class:

    private class StreamingMarkupBuilderTemplate implements Template {
        final Class<BaseTemplate> templateClass;							(1)

        public StreamingMarkupBuilderTemplate(final Reader reader) {
            templateClass = groovyClassLoader.parseClass(
		new GroovyCodeSource(reader, "GeneratedMarkupTemplate" + counter.getAndIncrement(), ""));
        }

        public StreamingMarkupBuilderTemplate(final URL resource) throws IOException {
            templateClass = groovyClassLoader.parseClass(new GroovyCodeSource(resource));
        }

        public Writable make() {
            return make(Collections.emptyMap());							(2)
        }

        public Writable make(final Map binding) {
            return DefaultGroovyMethods.newInstance(templateClass,
		new Object[]{MarkupTemplateEngine.this, binding, templateConfiguration});		(3)
        }
    }

1	`templateClass` corresponds to the user script, compiled as a template class
2	`make` binds a model to the template
3	`make` instantiates a new template and binds the model to it

The bind method is very important. It returns a Writable which will trigger template rendering when writeTo is called. Therefore, nothing is rendered until the Writable#writeTo method is called. The only thing that bind does is instantiating a new template. As we are using a cached class, there’s no compilation involved anymore, so the template is compiled once for all.

The BaseTemplate class

As you can see, the user script is compiled into a class which extends BaseTemplate. This means that the following script:

page.tpl

html {
   body {
      p("I'm a template")
   }
}

compiles to a class extending BaseTemplate. If you are familiar with GroovyClassLoader or GroovyShell, you should actually know that normally, a script compiles to a class extending Script. In our case, we don’t want to extend Script, because it does things that we don’t want, such as overriding getProperty or using a Binding class, or even having the semantics of a Groovy script. Furthermore, it doesn’t allow us to have a custom constructor to have private final fields. So the first step of our compilation process is actually to change the super class of the compiled script. The next step is to create a constructor that takes our model and the template configuration as parameters, as seen in the make method. Last but not least, since the script being compiled defines a run method (which is abstract in Script) corresponding to the script body, we will perform additional code transformations on this specific method.

@Override
    public void call(final SourceUnit source, final GeneratorContext context, final ClassNode classNode) throws CompilationFailedException {
        if (classNode.isScriptBody()) {							(1)
            classNode.setSuperClass(MarkupTemplateEngine.BASETEMPLATE_CLASSNODE);	(2)
            createConstructor(classNode);						(3)
            transformRunMethod(classNode, source);					(4)
        }
    }

1	a Groovy script may contain multiple class, so we need to check if the current class is actually the script body
2	we change the super class from `Script` to `BaseTemplate`
3	we create a new constructor
4	we perform code modifications on the script body

Implementing the builder

The transformRunMethod method is actually an example of how to implement a Groovy DSL using AST transformations. The goal of this method is to alter the AST (abstract syntax tree), so that some method calls in source code, for example, are actually rewritten. It is also the starting point of performance optimizations. This is actually very important. For example, one of the transformations will change:

p(text)

into:

p(getModel().get("text"))

Technically speaking, there is no need to do such a change: we could rely on BaseTemplate implementing propertyMissing to resolve missing variables (here, text) and delegate the call to its internal model field. However, this can be particularily slow, especially in builders where there are lots of nested closures, which involve a very long call chain. By doing this change, we transform a so-called DynamicVariable (text) into something that can be resolved statically (getModel is declared in BaseTemplate and is of type Map). Slowly, we’re making a switch towards statically compilable code… but we’re not there yet.

In classic builder code, compiling this would work and eventually, when the p method is called, the meta-object protocol goes into action and eventually calls the methodMissing method on the BaseTemplate class if it is defined. So to make our code work, all we have to do is to write that method:

BaseTemplate.java

    public Object methodMissing(String tagName, Object args) throws IOException {
        Object o = model.get(tagName);
        if (args instanceof Object[]) {
            final Writer wrt = out;
            TagData tagData = new TagData(args).invoke();
            Object body = tagData.getBody();
            writeIndent();
            wrt.write('<');
            wrt.write(tagName);
            writeAttributes(tagData.getAttributes());
            if (body != null) {
                wrt.write('>');
                writeBody(body);
                writeIndent();
                wrt.write("</");
                wrt.write(tagName);
                wrt.write('>');
            } else {
                if (configuration.isExpandEmptyElements()) {
                    wrt.write("></");
                    wrt.write(tagName);
                    wrt.write('>');
                } else {
                    wrt.write("/>");
                }
            }
        }
        return this;
    }

We can test that this code works by rendering a simple template:

import groovy.text.markup.MarkupTemplateEngine
import groovy.text.markup.TemplateConfiguration

def tplConf = new TemplateConfiguration()
MarkupTemplateEngine engine = new MarkupTemplateEngine(this.class.classLoader, tplConf)

def mkpTemplate = engine.createTemplate '''
html {
    body {
	p(text)
    }
}
'''
def model = [text:'It works!']
mkpTemplate.make(model).writeTo(new PrintWriter(System.out))

What is nice is that we can also rely on the template configuration to perform different transformations. For example, there’s an optional autoEscape flag which tells if variables read from the model should be automatically escaped. If the flag is set to false, the following code:

text

is transformed into:

getModel().get("text")

but if the flag is set to true, the generated code is:

yield(getModel().get("text"))

where yield is the method which will escape contents… So it’s a very flexible way to perform parametrized transformations of templates! The same technique is used to:

transform include (template|escaped|unescaped):'path/to/template' into `include(Groovy|Escaped|Unescaped)('/path/to/template')
transform unescaped.foo into getModel().get("foo")
transform ':XXX'() method calls into methodMissing('XXX', …). This gives a way to render tags which have the same name as helper methods like yield. In that case, the user can write ':yield'() to create a tag <yield> instead of calling the yield method for example.

Statically compiling templates

Oops, I did it again!

Going further towards statically compilable code requires additional trickery. In the previous example, we still have a call (p(…)) which is unresolved, goes through the MOP and eventually calls methodMissing. The same way we converted the text variable into a dynamic call, we can make it statically compilable. Since the method which would eventually be called would be methodMissing, instead of going through the MOP, since we know that this particular method will always be called in our case, we can directly make the change, and hardwire it. The resulting code would look like this:

methodMissing("p", new Object[]{getModel().get("text")})

This change can be made in our transformer, like we did the getModel change. However, we will see that we have a serious problem with that. Meanwhile, let’s show how we can trigger static compilation of templates. This can be done easily by injecting the @CompileStatic annotation through CompilerConfiguration:

compilerConfiguration.addCompilationCustomizers(new TemplateASTTransformer(tplConfig)); 		(1)
compilerConfiguration.addCompilationCustomizers(
                new ASTTransformationCustomizer(CompileStatic.class));					(2)

1	apply the AST transformations to rewrite unresolved variables and method missing
2	apply `@CompileStatic` to the template

We can try the template engine using the same code as before:

import groovy.text.markup.MarkupTemplateEngine
import groovy.text.markup.TemplateConfiguration

def tplConf = new TemplateConfiguration()
MarkupTemplateEngine engine = new MarkupTemplateEngine(this.class.classLoader, tplConf)

def mkpTemplate = engine.createTemplate '''
html {
    body {
    p(text)
    }
}
'''
def model = [text:'It works!']
mkpTemplate.make(model).writeTo(new PrintWriter(System.out))

And it works! So what is the problem? Actually, there are multiple issues. The first one is that you can’t call existing methods anymore! For example, we can’t call the yield method because it has been converted too:

yield 'Some text'

gets converted into:

methodMissing("yield", new Object[] {"Some <text to escape>"})

and eventually generates this:

<yield>Some <text to escape></yield>

Is it the end of the story? Can’t we really statically compile our templates and make them super fast? Well, no, of course!

This is where all the magic begins. Solving this problem requires being able to make a difference between calls to methods which exist (like yield) and calls to methods which are not defined (like html). And guess what, Groovy has a very nice tool whose responsability is exactly this: static type checking and by extension, static compilation.

So let’s start by removing the code which transforms the method calls into methodMissing calls, and try to compile the following template:

import groovy.text.markup.MarkupTemplateEngine
import groovy.text.markup.TemplateConfiguration

def tplConf = new TemplateConfiguration()
MarkupTemplateEngine engine = new MarkupTemplateEngine(this.class.classLoader, tplConf)

def mkpTemplate = engine.createTemplate '''
html {
    body {
        yield text
    }
}
'''
def model = [text:'Text <to be escaped>']
mkpTemplate.make(model).writeTo(new PrintWriter(System.out))

Since we kept static compilation, it will fail, but it will give us interesting information:

[Static type checking] - Cannot find matching method GeneratedMarkupTemplate6#html(groovy.lang.Closure). Please check if the declared type is right and if the method exists.
 at line: 2, column: 1

[Static type checking] - Cannot find matching method GeneratedMarkupTemplate6#body(groovy.lang.Closure). Please check if the declared type is right and if the method exists.
 at line: 3, column: 5

What we see is that the two errors are precisely the methods that we want to directly wire to methodMissing. The yield method has been recognized, so the type checker did the job for us. It is telling us: "ok guys, there are two method calls I know nothing about. Those are html and body. Please do something or I can’t compile it.".

That’s nice, but how can I help the compiler?

Type checking extensions

Well, this is precisely why type checking extensions were added in Groovy 2.1. They allow the programmer to help the compiler when he knows about a method call that the type checker isn’t able to resolve. You can give hints and tell "ok, this method exists, and it returns an object of type Foo". In Groovy 2.2, this mechanism was extended to static compilation, which opens another chapter in the incredible extensibility that Groovy has to offer.

Writing type checking extensions require a bit of knowledge of the Groovy AST (abstract syntax tree), so anyone who ever wrote an AST transformation in Groovy should be capable of writing a type checking extension. Actually, it is even easier, and the process is described here. In our case, we will start leveraging a feature of Groovy 2.2 will allows us to mix dynamic code with statically compiled code. That is to say that the only thing that our extension is going to do is saying "when you don’t know what a method call does, perform a dynamic invocation".

When we added this to Groovy 2.2, we really didn’t want to make "mixed" mode a first class citizen in static compilation, because it defeats the idea of catching typos, which is one of the things people expect most from a type checking system. So this "mixed" mode is actually activated by type checking extensions. This means that the only method calls which will be made dynamic will be those that the programmer knows about, and really wants to convert to dynamic calls. It is an important difference, because we want the developper to be aware of what he is doing.

That said, how can we implement that? It’s actually pretty easy. The first thing to do is to write the code which will help the compiler:

methodNotFound { receiver, name, argList, argTypes, call ->	(1)
    if (call.lineNumber > 0) {					(2)
        if (call.implicitThis) {				(3)
            return makeDynamic(call, OBJECT_TYPE)		(4)
        }
    }
}

1	react to the `methodNotFound` event, thrown by the type checker
2	make sure the event is called on user code, that is to say code for which there’s an associated line number
3	make sure that only calls which are on an "implicit this" are made dynamic (see below)
4	instruct the compiler to perform a dynamic call here

Point 3 is actually important. We said that we wanted the developper to be aware of which calls he want to make dynamic. This is the kind of guards that you might want to add. In builder-style code, only method calls for which there’s no explicit receiver should be considered as method creating tags. For example, if you write this.foo, there is an explicit this receiver, and we don’t want to convert that call. Instead, we want to let the compiler report an error.

Now that the extension is written, we still have to load it. There are two ways of loading type checking extensions: using scripts (Groovy 2.1) or precompiled type checking extensions (Groovy 2.2+). In my case, I wanted to use precompiled type checking extensions, to avoid paying the cost of compiling the type checking extension at runtime. This can be done by wrapping the extension script into a class extending GroovyTypeCheckingExtensionSupport.TypeCheckingDSL:

class MarkupTemplateTypeCheckingExtension extends GroovyTypeCheckingExtensionSupport.TypeCheckingDSL {

    @Override
    Object run() {
	methodNotFound { receiver, name, argList, argTypes, call ->
		...
	}
    }
}

Then the extension can be loaded by slightly changing the way we activate @CompileStatic:

compilerConfiguration.addCompilationCustomizers(
        new ASTTransformationCustomizer(
            Collections.singletonMap("extensions","groovy.text.markup.MarkupTemplateTypeCheckingExtension"), (1)
            CompileStatic.class));									     (2)

1	create the map of arguments for the `@CompileStatic` AST transformation
2	apply `@CompileStatic`

Improving performance

With the type checking extension, we’ve now instructed the compiler to perform dynamic calls when it finds tag methods. All other calls, which were resolved statically, are made static. This is nice, but we are still paying the price of the meta-object protocol here, and there’s no reason to go through a dynamic path were we want the target method to be methodMissing in any case. So, how can we solve that?

Before going further, you have to be warned. What I am going to show you is things that I wouldn’t recommand for beginners in AST transformations. We’re going to update the AST just before it is going to generate bytecode. This is very late in the compilation process, meaning that you are walking along a thin rope without net! Traditional AST transformations run much earlier in the compilation process, and the compiler will do a lot of things for you (like resolving methods, variables, …). Here, it is so late that all those things have already been done, so you have to do it all yourself!

Now let the fun begin! The idea is quite simple actually. Instead of relying on makeDynamic, we will transform the calls into direct calls to methodMissing. The type checking extension API doesn’t let you do this (it’s not meant to transform the AST), so you have to do it yourself. This involves multiple steps:

when we start visiting a method, create an empty list of method calls that will need to be transformed
when an unresolved call is found and that it matches our criteria, put that call into the list
after the method has been visited, trigger a transformer which will transform all calls in the list

The first step requires an extra block:

beforeVisitMethod {		(1)
   newScope {			(2)
      builderCalls = []		(3)
   }
}

1	we’re entering a new method body
2	`newScope` pushes a "type checking scope" on stack, where you can put user data
3	add the `builderCalls` method to this scope

Then you can add the method calls to be transformed this way:

methodNotFound { receiver, name, argList, argTypes, call ->
    if (call.lineNumber > 0) {
        if (call.implicitThis) {
            currentScope.builderCalls << call
            return makeDynamic(call, OBJECT_TYPE)
        }
    }
}

And triggering the AST transformation can be done in an afterVisitMethod block:

afterVisitMethod { mn ->							(1)
   scopeExit {									(2)
      new BuilderMethodReplacer(context.source, builderCalls).visitMethod(mn)	(3)
   }
}

1	when we exit a method body
2	pop the current scope from stack
3	trigger an AST transformation which will visit this method knowing which calls need to be transformed

Of course, we still miss the transformation code. For that, we need a class which extends ClassCodeExpressionTransformer :

    private static class BuilderMethodReplacer extends ClassCodeExpressionTransformer {

        private static final MethodNode METHOD_MISSING = ClassHelper.make(BaseTemplate).getMethods('methodMissing')[0]		(1)

        private final SourceUnit unit;
        private final Set<MethodCallExpression> callsToBeReplaced;

        BuilderMethodReplacer(SourceUnit unit, Collection<MethodCallExpression> calls) {
            this.unit = unit
            this.callsToBeReplaced = calls as Set;
        }

        @Override
        protected SourceUnit getSourceUnit() {
            unit
        }

        @Override
        void visitClosureExpression(final ClosureExpression expression) {
            super.visitClosureExpression(expression)
        }

        @Override
        public Expression transform(final Expression exp) {
            if (callsToBeReplaced.contains(exp)) {									(2)
                def args = exp.arguments instanceof TupleExpression ? exp.arguments.expressions : [exp.arguments]
                args*.visit(this)
                // replace with direct call to methodMissing
                def call = new MethodCallExpression(									(3)
                        new VariableExpression("this"),
                        "methodMissing",
                        new ArgumentListExpression(
                                new ConstantExpression(exp.getMethodAsString()),
                                new ArrayExpression(
                                        OBJECT_TYPE,
                                        [* args]
                                )
                        )
                )
                call.implicitThis = true
                call.safe = exp.safe
                call.spreadSafe = exp.spreadSafe
                call.methodTarget = METHOD_MISSING									(4)
                call													(5)
            } else if (exp instanceof ClosureExpression) {
                exp.code.visit(this)
                super.transform(exp)
            } else {
                super.transform(exp)
            }
        }
    }

1	find the method which will eventually be called, `methodMissing` and keep a handle on it
2	when an expression is visited, we check if it is a method call which should be replaced
3	create a new method call
4	link the call to its target method (very important if you don’t want to crash the compiler!)
5	return the new method call

And we’re done! Of course, I didn’t say it was trivial nor easy, yet, it is possible, and now, all methods supposed to create a tag are directly wired to methodMissing, meaning that they are now statically compiled!

More things to fix…

You may think that all problems are solved, but in fact, there are still issues with this code. Imagine the following template:

p(text.toUpperCase())

If you compile it, it will fail with:

Cannot find matching method java.lang.Object#toUpperCase(). Please check if the declared type is right and if the method exists.

The reason is that we compile the template statically. While we instructed the compiler that text is in fact getModel().get("text"), it is still unable to know what is the return type of this call. Then, it assumes that it returns an Object, and if you try to call toUpperCase on an Object, the method doesn’t exist… This can easily be solved, by making all unresolved method calls dynamic. This means that the template compilation will never throw such errors anymore, but it will instead make a dynamic call. Problem solved.

Well, almost.

What if I do this?

p(((String)text).toUpperCase())

Then, by adding a cast, the static compiler is able to resolve the method call. Instead of doing a dynamic call, like it would with our extension, it will perform a direct method call, which will be faster! This means that if you add types, by casting, to your template, rendering can be made faster.

This is an interesting idea, but it is not really user friendly. So the last thing I added to the type checking extension is actually an optional, "type checked" mode. If this mode is activated, then the programmer is supposed to tell which are the types of the elements found into the binding. Here, the developper would have to declare that text is of type String:

import groovy.text.markup.MarkupTemplateEngine
import groovy.text.markup.TemplateConfiguration

def tplConf = new TemplateConfiguration()
MarkupTemplateEngine engine = new MarkupTemplateEngine(this.class.classLoader, tplConf)

def mkpTemplate = engine.createTypeCheckedModelTemplate '''				(1)
p {
   yield text.toUpperCase()								(2)
}
''', [text:'String']
def model = [text:'Text <to be escaped>']						(3)
mkpTemplate.make(model).writeTo(new PrintWriter(System.out))

1	use `createTypeCheckedModelTemplate` instead of `createTemplate`
2	you can use `text.toUpperCase()` without an explicit cast
3	because the model was declared using a simple map

This mode is actually very interesting if you want to report template errors at template compilation time. Instead of having an error when the template is rendered, the error will occur at compile time. So, for example, if we change the model declaration from:

[text:'String']

[text:'Integer']

template compilation will now fail with:

[Static type checking] - Cannot find matching method java.lang.Integer#toUpperCase(). Please check if the declared type is right and if the method exists.

What is really interesting is that you can declare "complex" models, like:

[persons:'List<Person>', posts:[List<Post>]]

and have the template statically compiled! The implementation details of that mode are a bit complex, but you can take a look at the commit if you want to have some hint (don’t hesitate to ask me if you want me to explain how it works).

And more features!

Last but not least, the template engine implements automatic indent and automatic new lines. The first one is quite easy to implement, as it only requires wrapping the supplied Writer into an IndentWriter. But adding automatic new lines is a bit trickier, because we want to rely on the layout of the source code to actually add behavior! Let me explain that again with examples. If you have:

div {
  p('text')
}

we want to render:

<div>
    <p>Text</p>
</div>

But if we have:

div {  p('text') }

We want to render:

<div><p>Text</p></div>

The problem is that our templates are actually Groovy code… And new lines are not significant. They are not even visible in the AST, so how can we implement such a feature?! The answer relies in each AST node… They all carry line and column number information. So, by comparing, in a ClosureExpression, the line number of the closure itself with the line number of its first code statement, we can determine if there was a new line in source code! The same way, we can check if the last line number of the closure is greater than the line number of the last statement, and if so, introduce a new line… So, in the first example, the code is actually transformed into:

div {
  newLine()
  p('text')
  newLine()
}

And that’s it!

Conclusion

In this (long) blog post, I have demonstrated various techniques that allowed me to transform a slow, dynamic builder based template engine into a fast, statically compiled, template engine with optional user model type checking and unique features like automatic new line insertions. It goes far beyond what the StreamingMarkupBuilder has to offer and demonstrates that compile time metaprogramming can be used in Groovy to provide advanced features. Of course, no one would expect you to create such code from the beginning. If you take a look at the branch commits, you will definitely see that I went step by step. And eventually, I will issue a pull request when I think that the code is ready for prime time. I am not sure yet this should make into core groovy, or instead if it should go into an external project. Ideas are welcome!

I still have to make some changes, like not reporting errors if the type checking mode is not active (and always going through dynamic mode in that case) and probably write more benchmarks, but I’m really looking forward to read what you think. Oh yes, one last thing: congratulations if you read that post throughfully!

A fast markup template engine for Groovy (part 1 of 2)

Sun, 16 Feb 2014 00:00:00 +0100

Recently, I was surprised no-one had the idea of developping a template engine relying on the Groovy MarkupBuilder. Working on JBake made me take a look at existing template engines for Java again, something I hadn’t done for years, and even if new technologies like Thymeleaf or Handlebars exist, not of them are as practical to use as the markup builder.

An interesting experiment

For that reason, I started working on a new template engine leveraging Groovy. The primary focus of this template engine would be XML-like markup (XHTML, HTML5, XML, …), but in theory, it can be used to render anything. The basic idea is that you can write a template which looks like this:

html {
    body {
        yield message			(1)
    }
}

1	'message' is a template variable

which renders to:

<html><body>It works!</body></html>

My first implementation was done quickly done and was leveraging the StreamingMarkupBuilder. It wasn’t very difficult to write, but it had a problem: in general, builders are quite slow, so not very suitable for templating engines, where you should render a page as fast as possible. To check this, I wrote a simple micro-benchmark, which compared my template engine with Freemarker:

@Grab(group='org.gperfutils', module='gbench', version='0.4.2-groovy-2.1')
@Grab('org.freemarker:freemarker:2.3.9')
import groovy.text.markup.MarkupTemplateEngine
import freemarker.template.*

MarkupTemplateEngine engine = new MarkupTemplateEngine()
def mkpTemplate1 = engine.createTemplate '''
html {
    body('It works!')
}
'''
def mkpTemplate2 = engine.createTemplate '''
html {
    body(text)
}
'''

def mkpTemplate3 = engine.createTemplate '''
html {
    body(text.toUpperCase())
}
'''

def cfg = new Configuration()
def ftlTemplate1 = new Template("name", new StringReader('''<html><body>It works!</body></html>'''), cfg);
def ftlTemplate2 = new Template("name", new StringReader('''<html><body>${text}</body></html>'''), cfg);
def ftlTemplate3 = new Template("name", new StringReader('''<html><body>${text?upper_case}</body></html>'''), cfg);
def r = benchmark {
    'MarkupTemplateEngine (simple, no binding)' { mkpTemplate1.make([:]).writeTo(new StringWriter()) }
    'Freemarker (simple, no binding)' { ftlTemplate1.process([:], new StringWriter()) }
    'MarkupTemplateEngine (simple binding)' { mkpTemplate2.make([text:'Hello']).writeTo(new StringWriter()) }
    'Freemarker (simple binding)' { ftlTemplate2.process([text:'Hello'], new StringWriter()) }
    'MarkupTemplateEngine (simple toUpper)' { mkpTemplate3.make([text:'Hello']).writeTo(new StringWriter()) }
    'Freemarker (simple toUpper)' { ftlTemplate3.process([text:'Hello'], new StringWriter()) }
}

r.prettyPrint()

I expected my template engine to be slower, yet I didn’t expect it to be that slow:

                                            user  system    cpu   real

MarkupTemplateEngine (simple, no binding)  23038       0  23038  23059
Freemarker (simple, no binding)              385       0    385    392
MarkupTemplateEngine (simple binding)      26221       0  26221  26244
Freemarker (simple binding)                 1037       1   1038   1046
MarkupTemplateEngine (simple toUpper)      26895       0  26895  26929
Freemarker (simple toUpper)                 1161       0   1161   1171

As wisdom says, make it work, make it right, then make it fast. I already had it working and it did what I wanted, so the next logical step would therefore be optimizing… But before I show you the results of that steps, let’s take a look at the features I have implemented so far.

Features

I am quite happy with the current feature set, which I think is good enough for a public review. Here is a quick list.

builder syntax

html {
   head {
      meta(charset:'utf-8')
      title('Page title')
   }
}

Renders to:

<html><head><meta charset="utf-8"><title>Page title</title></head></body>

For that, the code to initialize the template engine is quite simple (note that you can use it in Java too):

import groovy.text.markup.MarkupTemplateEngine
import groovy.text.markup.TemplateConfiguration

def tplConf = new TemplateConfiguration()
MarkupTemplateEngine engine = new MarkupTemplateEngine(this.class.classLoader, tplConf)

def mkpTemplate1 = engine.createTemplate '''
html {
   head {
      meta(charset:'utf-8')
      title('Page title')
   }
}
'''

def model = [:]

mkpTemplate1.make(model).writeTo(new PrintWriter(System.out))

Groovy goodness

ul {
   persons.each { p ->
   	li(p.name)
   }
}

With the following model:

def model = [persons: [new Person(name:'Bob'), new Person(name:'Alice')]]

Renders to:

<ul><li>Bob</li><li>Alice</li></ul>

template inclusion

inclusion of another template

html {
   include template: 'includes/header.tpl'
   include template: 'includes/body.tpl'
}

inclusion of escaped text

html {
   div(class:'post') {
   	include escaped: 'content/text.txt'
   }
}

inclusion of unescaped text

html {
   div(class:'post') {
   	include unescaped: 'content/raw.txt'
   }
}

automatic escaping of user input

By default, variables read from the model will be escaped. For example, given the following model:

def model = [text:'<html>']

and the following template:

p(text)

Then the result will be:

<p>&lt;html&gt;</p>

It is possible to avoid escaping by prefixing the variable name with unescaped.:

p(unescaped.text)

automatic indentation

Automatic indentation allows the programmer to get rid of indent instructions. For the following template:

html {
   newLine()
   body {
newLine()
p('Auto indent in action!')
newLine()
   }
   newLine()
}

The result without automatic indent is:

<html>
<body>
<p>Auto indent in action!</p>
</body>
</html>

If auto-indent is activated, then the output is changed to:

<html>
    <body>
        <p>Auto indent in action!</p>
    </body>
</html>

automatic new lines

The previous example was nice, but you have a lot of newLine method calls. Activating auto-newline will use the open blocks to introduce newLines automatically. That means that you can change the template to:

html {
   body {
p('Auto indent in action!')
   }
}

which is much more readable! To understand how it works, let’s slightly change the template, by moving the body block to the same line as html:

html {   body {
p('Auto indent in action!')
   }
}

in that case, the output is modified accordingly: -

<html><body>
        <p>Auto indent in action!</p>
    </body>
</html>

This means that new lines are really added where you have them in source code!

optional type checking of model

Imagine the following model:

class User {
   Long id
   String name
}

def model = [user: new User(id: 123, name: 'Cedric')]

and the following code to create a template:

def template = engine.createTemplate '''
p "User id: $user.id Name: $user.name Age: $user.age"
'''

The template is compiled fine, but if you try to render it, it will fail:

template.make(model).writeTo(new PrintWriter(System.out)) 	(1)

1	fails with groovy.lang.MissingPropertyException: No such property: age for class: User

To avoid failing at render time, you can use a type checked mode:

def template = engine.createTypeCheckedModelTemplate '''	(1)
p "User id: $user.id Name: $user.name Age: $user.age"
''', [user: 'User']						(2)

1	use createTypeCheckedModelTemplate instead of createTemplate
2	provide model type information

And this time, compilation of the template will fail directly:

[Static type checking] - No such property: age for class: User

This means that it won’t wait for an actual template to be rendered for failing, which is a feature some people like (yes, I’m looking at you, Play! Framework ;)). Now that we’ve showed a quick list of those features (there are more, like helper methods in model, …), what about performance? Is it worth it? I thought Groovy, and builders, were slow?

Statically compiled templates

I had already played with static builders in the past, and we also have this very nice feature of static type checking extensions in Groovy 2.1. In Groovy 2.2, type checking extensions were extended to static compilation… So my first idea was to get rid of the StreamingMarkupBuilder, which is good but not really suited for optimizations. And before I explain the implementation details, here is the result of a more complete benchmark, using my latest implementation:

@Grab(group='org.gperfutils', module='gbench', version='0.4.2-groovy-2.1')
@Grab('org.freemarker:freemarker:2.3.9')
import groovy.text.markup.MarkupTemplateEngine
import freemarker.template.*

MarkupTemplateEngine engine = new MarkupTemplateEngine()
def mkpTemplate1 = engine.createTemplate '''
html {
    body('It works!')
}
'''
def mkpTemplate2 = engine.createTemplate '''
html {
    body(text)
}
'''

def mkpTemplate3 = engine.createTemplate '''
html {
    body(text.toUpperCase())
}
'''

def mkpTemplate3_typed = engine.createTemplate '''
html {
    body(((String)text).toUpperCase())
}
'''

def mkpTemplate4 = engine.createTemplate '''
html {
    body {
        ul {
            persons.each {
                li("$text $it")
            }
        }
    }
}
'''

def mkpTemplate4_typed = engine.createTemplate '''
List<String> pList = (List<String>) persons
String txt = text
html {
    body {
        ul {
            for (String p: pList) {
                li("$txt $p")
            }
        }
    }
}
'''

def model = [text:'Hello', persons:['Cedric','Guillaume','Jochen','Pascal','Paul']]

def cfg = new Configuration()
def ftlTemplate1 = new Template("name", new StringReader('''<html><body>It works!</body></html>'''), cfg);
def ftlTemplate2 = new Template("name", new StringReader('''<html><body>${text}</body></html>'''), cfg);
def ftlTemplate3 = new Template("name", new StringReader('''<html><body>${text?upper_case}</body></html>'''), cfg);
def ftlTemplate4 = new Template("name", new StringReader('''<html><body><ul><#list persons as person><li>${text} ${person}</#list></ul></body></html>'''), cfg);

def r = benchmark {
    'MarkupTemplateEngine (simple, no binding)' { mkpTemplate1.make([:]).writeTo(new StringWriter()) }
    'Freemarker (simple, no binding)' { ftlTemplate1.process([:], new StringWriter()) }
    'MarkupTemplateEngine (simple binding)' { mkpTemplate2.make(model).writeTo(new StringWriter()) }
    'Freemarker (simple binding)' { ftlTemplate2.process(model, new StringWriter()) }
    'MarkupTemplateEngine (simple toUpper)' { mkpTemplate3.make(model).writeTo(new StringWriter()) }
    'Freemarker (simple toUpper)' { ftlTemplate3.process(model, new StringWriter()) }
    'MarkupTemplateEngine (typed toUpper)' { mkpTemplate3_typed.make(model).writeTo(new StringWriter()) }
    'MarkupTemplateEngine loop' { mkpTemplate4.make(model).writeTo(new StringWriter()) }
    'MarkupTemplateEngine typed for loop' { mkpTemplate4_typed.make(model).writeTo(new StringWriter()) }
    'FreeMarker loop' {  ftlTemplate4.process(model, new StringWriter()) }
}

r.prettyPrint()

                                           user  system   cpu  real

MarkupTemplateEngine (simple, no binding)   609       0   609   615
Freemarker (simple, no binding)             400       0   400   410
MarkupTemplateEngine (simple binding)       628       3   631   631
Freemarker (simple binding)                 864       0   864   877
MarkupTemplateEngine (simple toUpper)       681       0   681   690
Freemarker (simple toUpper)                 972       0   972   986
MarkupTemplateEngine (typed toUpper)        696       3   699   706
MarkupTemplateEngine loop                  2927      12  2939  2949
MarkupTemplateEngine typed for loop        2579       0  2579  2615
FreeMarker loop                            2862       0  2862  2894

As you can see, there is a huge difference. For non trivial templates, the markup template engine is even faster than Freemarker! To acheive such performance, all templates are compiled into bytecode, but also make use of static compilation, type checking extensions, AST transformations, …

In the next part, I will describe the techniques I used to implement this engine. Meanwhile, let me know what you think of it. If you feel brave, you can test it by checking out the sources on my fork:

git clone -b markup-template-engine https://github.com/melix/groovy-core.git
./gradlew console	(1)

1	will compile Groovy and open a Groovy console where you can test the template engine

Update: In the next part, we discuss the technical details.

Authoring your blog on GitHub with JBake and Gradle

Mon, 3 Feb 2014 00:00:00 +0100

A few weeks ago, I started this new blog and decided to host it on GitHub. I already explained how I migrated contents from JRoller, but I didn’t really the tool behind publishing. In this post, I will show you how easy it is, now that I have improved my toolchain with Gradle!

JBake

JBake is a static site generator. When I chose this tool for my blog, what attracted me is actually that it runs on the JVM, so I already had the idea of automating stuff with Gradle.

Since then, I have submitted several pull requests to the project, and we’re actually thinking of using it to build the next Groovy website. JBake has good chances to become the "Jekyll of the JVM" and already supports multiple markup languages (markdown, asciidoc) as well as multiple template engines (FreeMarker in 2.2.1, Groovy and Thymeleaf in 2.3.0).

The idea behind JBake is very simple: contents is written using a markup language and "baked" with template engines into actual HTML. Everything is generated statically and you can upload the generated site wherever you want.

On the other side, GitHub offers a nice tool to publish pages: GitHub Pages, so we can have both the sources of our blog and the output of bakery in the same repository. So the initial process looked like this:

create a new page in content directory
run jbake command, check the result, eventually make changes
git add page.adoc to add the new page to the sources
git commit -a to commit the page to the master branch, corresponding to the sources of the blog
git checkout gh-pages to switch to the GitHub pages branch
cp -R output/* . to copy the contents of the output directory at the root of the GitHub pages
git add page.html to add the new page to the GitHub pages branch
git commit -a to commit the contents to GitHub pages
git push origin master to push the sources to GitHub
git push origin gh-pages to push the generated pages to GitHub pages

I published my first blog posts like this, and it’s not that complicated, but it involves a lot of manual steps, so I came up with a Gradle plugin to make this much easier, that is, focusing on contents, period!

Gradle to the rescue!

Prerequisites

We’re going to setup a project that will allow you to publish to GitHub pages in one step, using Gradle. For that, we will need:

an empty blog repository that you created on GitHub
JBake to setup the initial JBake project structure
Gradle 1.10 (previous versions should work too)
The JBake plugin for Gradle, will be downloaded automatically
The Git plugin for Gradle, will be downloaded automatically

Building JBake

This step will become optional when JBake 2.3.0 will be out (or a snapshot of JBake will be published). Meanwhile, you need to install a local version of JBake into your local Maven repository. Don’t worry, this will be easy, you will only need Maven (hey, too bad JBake doesn’t use Gradle ;)):

git clone https://github.com/jbake-org/jbake.git
cd jbake
mvn install

That’s all!

Setting up a project

First step consists in creating the initial project layout:

git init                                                            (1)
git remote add origin <your github remote url>                      (2)
mkdir -p src/jbake                                                  (3)
gradle init                                                         (4)
git add build.gradle gradle gradlew gradlew.bat settings.gradle src (5)
cd src/jbake
jbake -i                                                            (6)

1	initializes the git repository
2	add your remote
3	we will put JBake sources into that directory
4	installs the Gradle wrapper into the repository
5	adds Gradle wrapper and initial structure into Git
6	initializes the default JBake directory layout

Using Gradle to render pages

At this point, your src/jbake directory contains the classic JBake folder contents:

src
 |-- jbake
       |-- assets    : static assets (images, css, ...)
       |-- content   : blog posts, ...
       |-- templates : HTML templates (by default, uses FreeMarker)

Replace the contents of build.gradle with the following:

build.gradle

buildscript {
  repositories {
      mavenLocal() // needed to use the local JBake 2.3.0 build
      jcenter()
  }

  dependencies {
    classpath 'me.champeau.gradle:jbake-gradle-plugin:0.1-SNAPSHOT' (1)
    classpath 'org.asciidoctor:asciidoctor-java-integration:0.1.4'  (2)
    classpath 'org.freemarker:freemarker:2.3.19'                    (3)
  }
}

apply plugin: 'jbake'                                               (4)

1	import the JBake plugin
2	if you write pages using the Asciidoctor format, this is necessary
3	if you use the Freemarker template engine, this is necessary
4	then apply the plugin

By default, the plugin will look for JBake sources in src/jbake and generate the site into build/jbake.

Generating the output

Now that this is applied, you can generate the site by running the following command:

./gradlew -i jbake

after the rendering step, you should now have a new directory:

build
  |-- jbake

into which you will find the generated HTML contents. At this point, you could choose to upload those files to an FTP server, for example, but since we want to host our pages on GitHub, we need to add some configuration to our build.

Publishing to GitHub Pages

Create a new file named publish.gradle with the following contents:

publish.gradle

buildscript {
  repositories {
      jcenter()
  }

  dependencies {
    classpath 'org.ajoberstar:gradle-git:0.6.3'     (1)
  }
}

apply plugin: 'github-pages'                        (2)

githubPages {
  repoUri = 'git@github.com:youtname/blog.git'      (3)
  pages {
     from(file('build/jbake')) {
     	into '.'                                    (4)
     }
  }
}

1	use the gradle-git plugin
2	apply the plugin
3	configure your GitHub repository URL
4	tells the plugin to upload the contents of the `build/jbake` directory

Then modify the main build.gradle file by adding the following task:

task publish(type: GradleBuild) {           (1)
    buildFile = 'publish.gradle'
    tasks = ['publishGhPages']
}

1	we’re using a separate, sub-build to avoid a classpath issue with JNA

Now if you do:

./gradlew publish

Gradle will pick the contents of the output/jbake directory, add it to the gh-pages branch and push it to GitHub, all in one step! A few seconds later, you should see the result on github.io!

One last thing: this configuration does not add the sources to your git repository. It only deals with the output, so you still need to add the source files, commit them and push them to GitHub. This is of course optional if you don’t want to publish the sources… If you decide to make them public, then you can also use the git plugin to do it automatically!

Conclusion

In this post, we’ve shown you how you can leverage JBake and Gradle to publish a blog on GitHub. The setup is really easy, so I hope more people will follow that way and contribute to Open Source too!

Of course, this post has been published using the procedure described on this page, so I used JBake and Gradle to blog about how to publish a blog on GitHub using JBake and Gradle! Like my friend Andres Almiray would say, that’s a toolception!

JLangDetect 0.4 released and moved to GitHub

Wed, 22 Jan 2014 00:00:00 +0100

A new era

Back in 2011, I initiated a project called JLangDetect, a language detection library for the JVM. At that time, I was working for a company that made NLP tools and I started this as a pet project on my free time. I made it available as open source software and people started to use it, so I was pretty happy at that time to see a project I initiated being actually used in production!

While I don’t have the need for such a library now that I work on the Groovy language, some people are still using it. At the first place, a friend of mine, Freddy Rabiller, took time to upgrade it. A new version has been released (0.4) and the project is now available on GitHub.

A big thank to Freddy, who will probably take the lead of development on this project now!

Closure parameter type inference for Groovy

Tue, 7 Jan 2014 00:00:00 +0100

Let’s start 2014 with a new blog post about a long standing request since we introduced static type checking in Groovy 2: closure parameter type inference. Before we start, let me wish you a happy new year and a lot of open source contributions!

Type checking closure parameter types

To illustrate the problem, let’s start with this very simple, standard, Groovy code:

void test() {
    assert ['foo','bar','baz'].collect { it.toUpperCase() } == ['FOO','BAR','BAZ']
}
test()

This code compiles and runs perfectly fine. Now if you want the test method to be type safe, you can annotate it with @TypeChecked:

import groovy.transform.TypeChecked

@TypeChecked
void test() {
    assert ['foo','bar','baz'].collect { it.toUpperCase() } == ['FOO','BAR','BAZ']
}
test()

If you compile this, you will notice that the compiler fails with an error:

[Static type checking] - Cannot find matching method java.lang.Object#toUpperCase(). Please check if the declared type is right and if the method exists.
 at line: 5, column: 42

Fixing this requires an explicit closure parameter type:

import groovy.transform.TypeChecked

@TypeChecked
void test() {
    assert ['foo','bar','baz'].collect { String it -> it.toUpperCase() } == ['FOO','BAR','BAZ']
}
test()

The problem comes from the collect method, which accepts a Closure. In Groovy, a Closure is a block of code which may capture variables, but it is also represented as an object of the class Closure. This is a different concept from Java 8 lambdas, which have no Lambda class, for example. A Java 8 lambda can be considered as purely syntactic sugar, which is interpreted as an interface implementation at compile time, although a Closure can be manipulated. To illustrate this, let’s compare the signatures of collect (in Groovy) and Map (in Java 8) which correspond to the same concept:

public static <T,U> List<U> collect(List<T> source, Closure<U> closure) (1)

items of the source list are of type T and converted to type U using the closure

<R> Stream<R> map(Function<? super T, ? extends R> mapper); (1)

Function is an interface, the lambda expression will be converted into this target interface

The Java 8 equivalent would therefore be:

list.stream().map((it)->it.toUpperCase()).collect(Collectors.toList()); (1)

we’re not using the smarter method reference notation here, to illustrate the concept

As you can see, Java allows the same thing as Groovy but doesn’t require an explicit type. The reason is that for Java, there’s no ambiguity: it makes use of target typing. Since a lambda is targetting an interface, the type of the parameters can be inferred from the interface type. In Groovy, we can’t do this, because Closure is not an interface. It is a class which can be manipulated. At this point, you may wonder why we don’t do the same as in Java, and there are several reasons:

historical reason first, Closure was one of the key features of the language, 10 years ago!
a single class, Closure, is enough for all usages of an open block. We don’t need Function, Consumer, BiFunction, … So we can dramatically reduce the amount of "design interfaces"
last but not least, Closures support various delegation strategies. This is something Java (or even Scala) is totally unable to do. Closure can be cloned, curried, trampolined, … and it always returns an instance of another Closure. This closure can change the delegate, which is the key for nice builder like DSLs. The delegate is used whenever a method call in a closure doesn’t have an explicit receiver. For example:

mail {
   from 'austin.powers@groovy.baby'
   to 'mini.me@evil.com'
   subject 'Attention please!'
   body '...'
}

In this DSL, the from, to, subject and body method calls are done on the delegate. Being able to set the delegate absolutely requires a Closure class. The implementation of the mail method may have something like:

class EmailSpecification {
    void from(String sender) { ... }
    void to(String to) { ... }
    void subject(String subject) { ... }
    void body(String body) { ... }
    void mail(Closure mail) {
       def mailSpec = mail.clone()
       mailSpec.delegate = this
       mailSpec()
    }
}

The problem with this approach is that if the closure requires parameters, like in the collect case, the Java type system, as well as the Groovy type system (which is the same), isn’t expressive enough to let you define them:

public static <T,U> List<U> collect(List<T> source, Closure<U> closure) (1)

We could like to say that Closure returns a U, but also that it consumes a T

Of course the first option that was studied was defining lots of Closure interfaces, corresponding to the various number of arguments (up to some arbitrary limit):

public static <T,U> List<U> collect(List<T> source, Closure1<T,U> closure) (1)

Closure1 is a kind of closure which accepts a single argument and returns a value

While this works, it has several drawbacks:

it requires a lot of arbitrary, totally useless in a dynamic context, number of interfaces/classes to define closures
it doesn’t solve the case of polymorphic closures

Polymorphic closures

Polywhat? In Groovy, closures can be polymorphic. To illustrate the concept, let’s take a look at a common method that iterates on map entries:

def map = [key1: 'value 1', key2: 'value2']
map.each { k,v -> println "Key is $k, value is $v" } (1)
map.each { e -> println "Key is ${e.key], value is ${e.value}" } (2)
map.each { println "Key is ${it.key], value is ${it.value}" } (3)

version where the map entry is automatically converted into a key and value arguments
version where the closure takes a single, Map.Entry argument
version with an implicit argument, it, is a Map.Entry

In all cases, it is always the same method which is called, that is to say each(Closure) on a Map. The signature of this method is:

public static <K,V> each(Map<K,V> self, Closure<?> onEachEntry)

Of course, the return type of the closure doesn’t help here, and just reading that signature, you have absolutely no way to guess that the closure will accept either a Map.Entry or a pair of K,V. Nor does the compiler. At best, your IDE knows it, and it does because it is hardcoded! This is exactly why the compiler fails, and also why so many people think it’s a bug.

Not convinced? Let’s make the same signature more cryptic:

public static <Dead,Pool> magneto(Map<Dead,Pool> self, Closure<?> professorX)

Now can you guess what professorX accepts as parameters? ;)

Tweaking the type system

We have discussed several options and we took the time to think about it, and after the last Groovy DevCon, which took place just before the Groovy and Grails eXchange 2013 in London, I decided to work on an implementation. For Groovy 2.1, we had introduced @DelegatesTo for closures, to be able to help the compiler in the case we described above (hinting at the delegate type) but we were still missing parameter type inference. My guess was that it was possible to do something similar to what @DelegatesTo does, but for parameter types.

Annotating closures

The idea is to annotate closures so that the compiler can fetch the information and infer the argument types from the context. In the case of a simple method accepting a closure, a simple annotation could do:

void doSomething(String src, @ClosureParams(String.class) Closure cl) { ... }

The @ClosureParams annotation is here to instruct the compiler that the closure will accept either an implicit or explicit parameter of type String:

doSomething {
   it.toUpperCase()
}

When the compiler determines that the method which will be called is doSomething (remember that this is only possible if type checking is activated), then an additional lookup on the doSomething signature can be done, and we can retrieve the list of expected parameter types from the closure annotation. Success!

Well, not really:

we still don’t support polymorphic closures
generics, GENERICS, aaahhhh, GENERICS!

Introducing… generics!

To make things a bit more complicated, we have generics. Don’t get me wrong. From a user perspective, generics are very good because they make the code more readable and help reduce the amount of boilerplate (think of vectors/maps before Java 1.5…). The typical case is the collect example that we used initially:

public static <T,U> List<U> collect(List<T> source, Closure<U> closure) (1)

In this case, we want to say that the closure:

is monomorphic
accepts a single parameter of type T

and the problem is… how to express this? One might think that you could write:

public static <T,U> List<U> collect(List<T> source, @ClosureParams(T) Closure<U> closure) (1)

but the truth is that the JVM doesn’t support placeholders as annotation values, nor does it support parametrized types (like @Foo(List<T>)). This tells us that the simple strategy doesn’t work.

The solution

The solution we propose is to decouple the declaration of the parameter types from the type itself. In other words, instead of declaring the types in the annotation, we will declare an object which is used as a hint to compute the types at compile time. In the case of collect, we end up with this:

public static <T,E> List<T> collect(List<E> self, @ClosureParams(FirstParam.FirstGenericType.class) Closure<? extends T> transform)

In this case, FirstParam.FirstGenericType doesn’t represent the type itself. It’s a hint used by the compiler, which says "the type of the argument is the type of the first generic type of the first parameter". In this case, the first parameter is List<E>, so the first generic type is E. This means that if you call the method with a List<String>, now the compiler can infer that E is a String.

Type hints

At this point, you may actually think that this "solution" is a bit complex. However, you have to remember that this kind of work is only necessary if you want to support type inference, so it is really only necessary if you use type checking. This makes this a tool primarily aimed at framework builders. In particular, lots of frameworks are written in Java (including Groovy itself), so the syntax has to be compatible with Java. Second, there’s no need to define one FirstParam.FirstGenericType class per method. The same class can be reused for all cases where it makes sense. Remember that it doesn’t represent the type of the parameters but a way to fetch the type (one level of indirection).

To make things easier for framework writers, the candidate implementation provides a set of predefined hint classes that should fit most of the use cases. Let’s go through the list:

FirstParam

FirstParam is a hint that says that the expected parameter type corresponds to the first parameter of the method call, like in:

public static void downto(BigInteger self, Number to, @ClosureParams(FirstParam.class) Closure closure)

The closure accepts a single parameter of type BigInteger.

FirstParam.FirstGenericType

This hint is used when the type to use is not the type of the parameter, but the type of the first generic type of the first argument, like in:

public static <T,E> Collection<T> collect(Collection<E> self, Collection<T> collector, @ClosureParams(FirstParam.FirstGenericType.class) Closure<? extends T> transform)

Note that if you have a Collection defined like this:

class PersonList extends LinkedList<Person> {}

and that you call collect:

list.collect { it.name }

the compiler will be capable of inferring that the type of the first generic type is actually a Person.

FirstParam also supports SecondGenericType and ThirdGenericType. You can also find SecondParam and ThirdParam which follow the same structure.

MapEntryOrKeyValue

This hint is used for cases where the closure may accept a Map.Entry or a key,value pair, which is quite common in the Groovy GDK, like each on maps:

public static <K, V> Map<K, V> each(Map<K, V> self, @ClosureParams(MapEntryOrKeyValue.class) Closure closure)

It is an example of polymorphic closure. This hint does all the job of telling that the parameter types may be a K,V pair or a Map.Entry<K,V>. For that, it expects the map to be the first parameter of the method.

SimpleType

Simple type can be used for monomorphic closures, in the cases the closure accepts parameters of a non-parametrized type. In this case, you need to use an option to specify the fully qualified name, like in this example:

public static void eachByte(InputStream is, @ClosureParams(value=SimpleType.class, options="byte") Closure closure)

In this example, the closure accepts a single parameter of type byte. For a non primitive type, you need the fully qualified name:

public static Writable filterLine(InputStream self, @ClosureParams(value=SimpleType.class, options="java.lang.String") Closure predicate)

If the closure accepts multiple arguments then you need options to be an array:

public static <T> T withObjectStreams(Socket socket, @ClosureParams(value=SimpleType.class, options={"java.io.ObjectInputStream","java.io.ObjectOutputStream"}) Closure<T> closure)

FromString

The last predefined hint can be used whenever none of the previous hints is suitable. A good example is the sort method on a collection, which takes a closure which either accepts a single parameter of type T (where T is the component type) or two parameters of type T in which case we have a comparator-style closure:

public static <T> List<T> sort(Collection<T> self, @ClosureParams(value=FromString.class, options={"T","T,T"} Closure c)

As you can see, in this example, the options map defines two possible signatures. The string literal are used at compile time to match those of the method signature. Since it involves much more work for the compiler, it is not recommanded to use FromString if other options are available, because it would be slower at compile time.

Future work

The candidate implementation is available on GitHub. It works pretty well, and honestly, I couldn’t come with any better idea. One very good point of this implementation is that it is Java friendly. You can annotate classes written in pure Java and the Groovy compiler would be able to use the extra information. In the future, we could probably support a nicer syntax for Groovy, but it would require a grammar change, which is not planned until Groovy 3. For example, we could write this:

public static <T> List<T> sort(Collection<T> self, Closure<T or T,T -> ?> c)

Which would totally avoid the "ugliness" of the annotation, while using the same backing tool.

Last thing, do not hesitate to comment on this blog about the solution we found. Of course, it took some time, and the discussions can be found here:

Thanks to everybody who participated in the discussion, and, of course, thank you for your comments if any: this is still a candidate solution, so if you come with any better idea, I’m open!

Performance of coercion vs closures in Groovy 2.2

Fri, 22 Nov 2013 00:00:00 +0100

To inaugurate this new blog, I will discuss the topic of coercion performance in Groovy. Especially, you might now that Groovy 2.2 introduced implicit closure coercion. If you don’t know what closure coercion is, or just what coercion alone is, let’s start with a reminder.

Casting vs coercion

Casting

In an object oriented language like Groovy, variables are typed. Even if Groovy is a dynamic language, each variable has a type at runtime. Even if Groovy shares the same typing model as Java, there’s almost no need for casting in Groovy.

Casting in Java is necessary because it’s a statically typed language, so if you want to call a method defined on the Person class on an object which is declared as type Object, you have to do a cast:

String pretty(Object o) {
    if (o instanceof Person) {
	return ((Person)o).getName(); // (1)
    }
}

(Person) is an explicit cast

in Groovy, casting is not necessary because we rely on runtime types and dynamic invocation. This means that this code is equivalent to this in Groovy:

String pretty(o) {
    o.name // (1)
}

casting isn’t required

Casting is only possible within the type hierarchy. That is, you can cast any object to any subtype (or interface) and it’s you’re responsability to make sure (for example using instanceof) that the runtime type will be correct. If you don’t, you may have the famous ClassCastException error at runtime.

Coercion

For types which are not in the same hierarchy, Groovy provides an alternative mechanism called coercion. Coercion is very handy because it basically allows you to convert an object of some type into an object of another, in general incompatible, type.

A good example is converting a File to a String[] corresponding to the lines of a text file. In groovy, you can write:

def lines = file as String[]

Obviously, if you had written:

def lines = (String[]) file

then it would have produced a ClassCastException. Basically, a cast is (almost) a no-op, while coercion involves any kind of treatment. It is also possible to implement your own coercion rules, by implementing the asType method:

class Person {
   String name
   int age
   def asType(Class target) {
      if (List==target) {
         [name,age]
      }
   }
}
def p = new Person(name:'Austin Powers', age:50)
assert p as List == ['Austin Powers', 50]

Closure coercion

One of the most widely used features of Groovy is closure coercion. It’s an easy way to implement interfaces. For example, giving the following interface:

interface Predicate {
    boolean apply(Object target)
}

You can implement it using coercion:

Predicate filter = { it.length() > 3 } as Predicate

This is especially interesting when the interface is used as a method call parameter:

List filter(List source, Predicate predicate) {
   source.findAll { predicate.apply(it) } // (1)
}

note that this example doesn’t really make sense since it’s the role of findAll to apply a closure as predicate!

So you can call the method without having to create an anonymous abstract class:

def items = filter(['foo','bar', 'foobar'], {
    it.length()>3
} as Predicate)
assert items == ['foobar']

Implicit closure coercion in Groovy 2.2

With the release of Groovy 2.2, closure coercion can be implicit, in case the target is a SAM (single abstract method) type. That is to say that the target type must have a single abstract method, which is the case for many functional interfaces (like Predicate here) and abstract classes. So the example can be further simplified:

def items = filter(source) { it.length()>3 } // (1)
assert items == ['foobar']

note that as Predicate is not needed anymore!

Can it be easier? Probably not, but maybe you noticed that this is close to what Java 8 will allow with lambdas:

List<String> items = filter(source, String str -> str.length()>3)

Now you must be aware of some subtle differences with Java 8. One is that closures are not lambdas but instances of the Closure class (a subclass of Closure, to be precise), while lambdas are converted at compile time and can be directly implemented, for example, as methods (simple case) or anonymous inner classes. This difference implies that if you have:

def method(Closure c) { ... }
def method(SAMType arg) { ... }

Then if you pass a closure as argument:

method { ...do something... }

then the method which is chosen is the version which accepts a Closure, not the version accepting a SAMType. But since Closure implements Runnable and Callable, the same is true for those two interfaces:

def method(Runnable c) { ... }
def method(SAMType arg) { ... }
method { ...do something... } // will call method(Runnable)

This means that if you want to call the SAMType version, you still have to use explicit coercion:

method { ...do something... } as SAMType

Now that we exposed the basics of closure coercion, let’s come to the topic that gave its name to this blog post: performance.

Performance of coercion vs closure

GBench

We will discuss here the impact of using closure coercion and compare the cost of implicit/explicit closure coercion as compared with calling a method which directly accepts a closure. For that, let’s start with the tool we’re going to use: GBench.

GBench is a project I really like and that I use a lot. It’s meant for micro-benchmarking. We know that micro-benchmarks are bad, but in some cases, they are useful. GBench makes them a little better by providing a framework that does all the boring stuff that you have to do when micro-benchmarking:

setting up timers
warm up
repeat the execution of the same code N times
generation of a report

All this using a nice DSL. If you want to write benchmarks, time execution of some process in your Groovy program, make sure to use it, it’s just the perfect tool.

Measurements

Now let’s proceed with the measurements. We want to compute the cost of:

directly accepting a closure as an argument
coercing the closure to a SAM type then calling

For that, we’re just defining a very simple SAM type and two helper methods:

interface SAMType {
    void apply()
}

@groovy.transform.CompileStatic
void direct(Closure c) { c.call() }

@groovy.transform.CompileStatic
void coercion(SAMType s) { s.apply() }

The two methods that will be called are compiled statically so that we made direct method calls inside the method body. This allows us to measure precisely the cost of calling the method, rather than the cost of dynamic dispatch. The measurements are made using this code:

Closure cachedClosure = { 'do something' }
SAMType cachedSAMType = { 'do something' }

@Grab(group='org.gperfutils', module='gbench', version='0.4.2-groovy-2.1')
def r = benchmark {
      'explicit coercion' {
          coercion { 'do something' } as SAMType
      }
      'implicit coercion' {
          coercion { 'do something' }
      }
      'direct closure' {
          direct { 'do something' }
      }
      'cached SAM type' {
          coercion cachedSAMType
      }
      'cached closure' {
          direct cachedClosure
      }
  }
  r.prettyPrint()

You can see that we are testing 5 cases here:

explicit coercion calls the method accepting a SAMType with explicit coercion of a closure into a SAMType
implicit coercion does the same, without as SAMType
direct closure calls the method accepting a Closure. This means that this version will not involve any conversion.
cached SAM type calls the SAMType version of the method with a coerced closure which is defined outside of the scope of the benchmark method
cached closure calls the Closure version of the method with a closure which is defined outside of the scope of the benchmark method

The last two versions are interesting because as I explained before, GBench automatically repeats the execution of the code N times. This means that this code:

SAMType cachedSAMType = { 'do something' }
// ...
'cached SAM type' {
    coercion cachedSAMType
}

is more or less equivalent to:

SAMType cachedSAMType = { 'do something' }
// ...
10000.times {
    coercion cachedSAMType
}

So here is the result of the execution of this benchmark:

Environment
===========
* Groovy: 2.2.0-rc-3
* JVM: Java HotSpot(TM) 64-Bit Server VM (23.5-b02, Oracle Corporation)
    * JRE: 1.7.0_09
    * Total Memory: 679.4375 MB
    * Maximum Memory: 1765.375 MB
* OS: Linux (3.8.0-22-generic, amd64)

Options
=======
* Warm Up: Auto (- 60 sec)
* CPU Time Measurement: On

                   user  system   cpu  real

explicit coercion  1258       0  1258  1259
implicit coercion  1102      12  1114  1115
direct closure      318       5   323   324
cached SAM type     263       0   263   265
cached closure      259       0   259   261

What you can see from those results is that:

using implicit closure coercion is slightly faster than explicit closure coercion
having a method which accepts directly a closure can significantly improve performance (almost 4x faster dispatch!)
using a cached closure or a cached SAM type is fast in any case

Note that using cached closures is not something that is specific to Groovy: it would be true for any Java code too, if you consider a coerced closure as an anonymous inner class. Each time the method is called, you create a new instance of the closure (or, in Java, the anonymous inner class). So moving the definition of the closure (or anonymous inner class) outside the loop and you will reuse the same instance, dramatically improving performance.

We must explain what performance we’re talking about here: the closure does nothing special here, just returning a dummy string. So the cost of the treatment is almost null. What if the code actually does something? Would the differences be so important? To check that, we will modify the code:

interface SAMType {
    void apply()
}

@groovy.transform.CompileStatic
void coercion(SAMType s) { s.apply() }

@groovy.transform.CompileStatic
void direct(Closure c) { c.call() }

void doSomething() {
   Thread.sleep(100)
}

Closure cachedClosure = { doSomething() }
SAMType cachedSAMType = { doSomething() }

@Grab(group='org.gperfutils', module='gbench', version='0.4.2-groovy-2.1')
def r = benchmark {
      'explicit coercion' {
          coercion { doSomething() } as SAMType
      }
      'implicit coercion' {
          coercion { doSomething() }
      }
      'direct closure' {
          direct { doSomething() }
      }
      'cached SAM type' {
          coercion cachedSAMType
      }
      'cached closure' {
          direct cachedClosure
      }
  }
  r.prettyPrint()

In this version, we simulate a long running process with Thread.sleep(100). The results are shown below:

                     user  system     cpu       real

explicit coercion  248621       0  248621  100329258
implicit coercion  208407       0  208407  100273428
direct closure          0  166932  166932  100238245
cached SAM type         0  157406  157406  100232334
cached closure          0  160848  160848  100214197

Note that it’s better to look at the real column here, since Thread.sleep doesn’t consume any CPU. What is interesting here is that now, there’s almost no difference between each version. This is simply explained: the cost of the treatment exceeds the cost of instantiating a closure and coercing it.

Conclusion

So given those figures, what can we conclude? First of all, one of the interests of implicit closure coercion is that previously (before Groovy 2.2), if you wanted users to avoid explicit coercion, you had to write a method accepting a closure:

// real method
void addListener(Listener listener) { ... }
// convenience method to avoid explicit coercion from user code
void addListener(Closure cl) { addListener(cl as Listener) }

The problem is that you double the number of methods here, so implicit closure coercion is a big bonus here. But our figures showed that calling a method accepting a closure is much faster, so you have a dilemn here: should you keep the closure version or not? The second benchmark gives a first answer: you shouldn’t remove the Closure version only if you know that the treatment in the closure is very fast. As soon as business code in the closure is a bit complex, it’s not worth it and you can remove the Closure version. This means that in the vast majority of cases, you can remove it without problem.

In fact, there’s one more case where you’d want to keep the Closure version: if you manipulate the closure before calling it, like changing the delegate:

void doSomething(SAMType arg) { ... }
void doSomething(Closure cl) {
   def clone = cl.rehydrate(delegate,this,this)
   doSomething(clone as SAMType)

Hope things are clearer for you now!

A new home!

Thu, 21 Nov 2013 00:00:00 +0100

Goodbye JRoller, welcome GitHub! Eventually, I migrated my blog to GitHub pages. It is a now a statically generated website. What is interesting here is how I migrated contents from my old blog. I used a Groovy script which downloaded the old pages, converted them into Asciidoctor format so that they can be statically processed by JBake.

Should you need something similar, I pushed this little toy here: https://github.com/melix/jroller-export

Now the export is quite raw, there are probably minor rendering issues to fix, or broken links, and I still have to integrate:

twitter/google+ share buttons
comments

But this is another story!

Last but not least, this blog is also published on GitHub and all posts are published under Creative Commons by-nb-sa.

deck2pdf: Exporting HTML5 slide decks to PDF

Tue, 30 Jul 2013 00:00:00 +0200

Exporting your awesome HTML5 presentation to PDF

For some time now, I’ve started using deck.js to write my talks. One of the reasons I do this is that it saves me lots of time when I have to copy and paste code, because I can rely on javascript code highlighting libraries to do the job. I can focus on contents instead of rendering. If I had a better knowledge of CSS, too, I could certainly write impressive presentations (but unfortunately, no, I’m not good at CSS).

The problem with HTML5 slideshows is that you are often asked to upload your slides as PDF. For example, SpeakerDeck, SlideShare or Parleys all require you to upload slides as PDF. None of them supports HTML5 (which is understandable because there are many frameworks available). While I did find some libraries that did the job (often in Perl or Ruby), I never managed to find one that actually worked properly.

Deck2pdf

That’s where it all started. I knew that JavaFX2 provided a WebView, which allowed rendering HTML pages with WebKit. What if I could use it to render my slide deck and export it to PDF? deck2pdf was born!. I started with something that was able to export my deck.js presentations, but I figured out very quickly that it could support other HTML5 presentation libraries quite easily. As of today, deck2pdf supports:

But more importantly, if your favorite HTML5 slideware is not supported, adding custom profiles is supported and fairly easy! Adding support for another library is as simple as adding a descriptor file with the javascript commands inside. For example, here’s how the deck.js profile is written:

totalSlides=$.deck('getSlides').length
nextSlide=$.deck('next')

That’s all! Support for more complex interactions is also provided using Groovy profiles, like for example in impress.js:

setup = {
    js 'var api = impress();'
    js '''var $$ = function ( selector, context ) {
        context = context || document;
        return context.querySelectorAll(selector);
    };'''
    js '''var byId = function ( id ) {
        return document.getElementById(id);
    };'''
}

nextSlide = {
    js('api.next()')
}

totalSlides = {
    js (/$$(".step", byId('impress')).length/)
}

// longer pause because of transitions
pause = 2000

Of course, deck2pdf is open source and licensed under APL2. Feel free to fork it and contribute new profiles! The home page of the project has documentation explaining how you can create your own. I’m waiting for your pull requests!

GR8Conf Europe 2013 is over

Mon, 27 May 2013 00:00:00 +0200

GR8Conf Europe 2013

Last week I spoke at GR8Conf Europe for the second time. As last year, the content of the conference was really amazing and highly technical. While the conference started with a University day on May, 22, we took advantage of the conference to organize a Groovy DevCon the day before. A Groovy DevCon is basically a physical meeting for the Groovy Core developers, where we share ideas and eventually make decisions with regards to the future of Groovy.

This year, several well-known Groovy gurus participated in the meeting:

Guillaume Laforge, the Groovy project lead
https://[Jochen ``Blackdrag'' Theodorou], the Groovy technical lead
Dierk Koenig, Groovy committer and main author of the famous Groovy in action book
Andrés Almiray, the Griffon lead
myself, Groovy Core committer

But we had a special guest, M. Juergen Hoeller of Spring fame himself!

The meeting was really useful. On my side, I was particularily happy with the discussions regarding Groovy and Java 8, where we eventually took the (long awaited for me) decision to provide some automatic coercion of closures to interfaces. Basically, this means that you would no longer have to write:

    interface Callback {
        void execute()
    }
    void onEvent(Callback c) { c.execute() }

    onEvent({println 'hello'} as Callback)

But instead, you will be able to write directly:

    interface Callback {
        void execute()
    }
    void onEvent(Callback c) { c.execute() }

    onEvent {println 'hello'}

This will lead to easier to read and maintain code. We discussed some limitations for the feature, but for a large number of cases, this would work directly. You can expect this feature to appear in Groovy 2.2.

AST transformations unleashed

The University day was the occasion for me to give a workshop about AST transformations. For GR8Conf Europe, I had the chance to do it with Andrés Almiray, the Griffon lead, but also one of the most talented programmer of the Java community. Andrés started with a small introduction about AST transformations, then I continued the workshop with a series of exercises (that you can take for yourself). Unfortunately, two hours and a half didn’t give us enough time to complete all exercises but I was quite happy with the session. I really hope this gave ideas to some people in the room, and that it provided the necessary bits to get started. Note that if you missed it, I will give the same workshop at GR8Conf US in Minneapolis, so register now!

Type checking your DSLs

I gave my second talk on Friday, last day of the conference. It was named Type checking your DSLs. While I already gave this talk 3 times before (G&G exchange, Greach and Devoxx France), I always like to change some contents to adapt to the audience and have some novelties inside. Unfortunately, for technical reasons, this talk went really bad for me. I assume people felt it, but I was quite disconcerted. The reason is that most of my talk relies on commenting code, but that I wasn’t able to show it properly! Even if the projectors were supposed to support full-HD through HDMI, for some obscure reason, my laptop wasn’t able to ``discover'' the HDMI output, and I was stuck to a 1024x768 VGA output. Worse than that, on screen, the borders were cut. This was the first time this ever happened to me, and to be honest, it completely troubled me. I wasn’t able to show code properly, so I wasn’t able to keep track of what I wanted to say. This was a horrible experience, but I hope people get the idea, as I only managed to give a rough idea of what I wanted to say :-( You know, that feeling when you try to explain something, and that people look at you as if you were an extraterrestrial…

I will give the same talk at Minneapolis later this year, so I sincerely hope things will go better, especially because I really like this talk, which gives a lot of hints on the power of the language, combining several features available in Groovy 2.1. All the (uncut) slides are available on GitHub.

Conclusion

Going to GR8Conf for the second time was like last year an incredible experience. I had the chance to talk to lots of talented and famous developers of the Groovy community, as well as sharing with users, which is worth the trip. This gave me for example the occasion to talk to two GPars brains for the first time (Dierk Koenig and Vaclav Pech). I’d really like to thank Soren for organizing this, as well as the crew for all the goodness that we can see here. I’m really looking forward to go to the next edition!

Null-safe invocation and primitive types

Wed, 17 Apr 2013 00:00:00 +0200

Null-safe method invocation

Yesterday, I worked on a bug reported on the static compiler of Groovy. It appeared that the underlying problem was related to how null-safe invocations are handled when the expected return type is a primitive. Let’s take an example:

class Person {
   String name
   int age
}

Person getPerson() { null }

Person p = getPerson()

def result = p?.age

Now the question is, what is the value of result? If you run this in the Groovy console, the result would be null. This is compatible with the assumption that the null-safe invocation operator (?.) always return null if the receiver of the message is null (here, p is null, so result is null). The definition is pretty easy, but it gets more complicated if you slightly modify the code:

int result = p?.age

By explicitely setting the type to int, executing this would throw an error, stating that you cannot convert null to int. It makes sense knowing that the null-safe invoker is supposed to return null if the receiver is null, but it starts getting strange as if p is not null, the assignment is perfectly valid since getAge() is expected to return a primitive type… This means that unlike the ``normal'' invoker which is guaranteed to keep the method return type untouched, the null-safe invoker does not honor the method signature.

The dynamic world

What if the null-safe invoker was aware of the return type? In null-safe'', you must think about what the safe'' part stands for. It’s definitely the receiver, because you want the invocation to be safe (not failing) if the receiver is null. If the receiver is null, then return null. This means that because p is null, it chooses to return null, independently of the method that was supposed to be called. Here, the method was getAge(), which is supposed to return a primitive type. As null is not a primitive, I would expect the null-safe invoker to return a default value compatible with the primitive type. This means that here, I would expect the null-safe invoker to return 0. Now, what is the problem with returning a default value? First of all, we’re in a dynamic world. This means that when p?.age is executed, the target method hasn’t been chosen, because you need to know the runtime type of p to determine what method will eventually be called. As p is null, the dynamic runtime doesn’t know the type of p, so has no idea that calling age would return a primitive type. Conclusion, in a dynamic world, the null-safe invoker must always return null, even if the expected method would return a primitive…

The static world

Now what if we’re in a pure static world? In that case, the method to be called is chosen at compile time, given the inferred type of p. It means that unlike the dynamic runtime, the static compiler, at this point, knows that p.age is p.getAge() which returns a primitive type. So it is capable of handling the null-safe invocation with what I think is better, semantically speaking: null-safe invocation only checks the receiver, and returns a value which depends on the return type of the method being invoked. So a static compiler is able to return 0 instead of the non-pritive null. What is funny is that I asked, on Twitter, what people expected from the result of p?.getAge() if p is null. Everybody answered null. So it’s clear that my way of thinking is not mainstream, but I’m ok with that. I just find it awkward that an operator which is supposed to act on invocation is also capable of altering the return type…

Conclusion

Anyway, even if it’s possible for the static compiler to return a default value, the fix I pushed doesn’t do that. It will always return null. The main reason for doing that is not that it was easier to fix (it’s quite the opposite), but that it keeps the semantics of statically compiled Groovy equal to those of dynamic Groovy. As it’s not possible for the runtime to know what the method would return, always returning null is fine, even if some situations (static compilation), you know a bit more what would happen :)

Les écoles sont-elles le problème ?

Tue, 26 Mar 2013 00:00:00 +0100

Comme beaucoup, j’ai appris aujourd’hui la création d’une école de développement lancée par Xavier Niel. Mais dans cette annonce, j’ai été choqué par plusieurs choses, à commencer par cette citation:

Aujourd’hui, le système français ne marche pas. Il est coincé entre d’une part l’université, qui propose une formation pas toujours adaptée aux besoins des entreprises mais qui est gratuite et accessible au plus grand nombre, et d’autre part les écoles privées, chères, dont la formation est assez qualitative mais laisse sur le côté de la route le plus grand nombre de talents, voire de génies, que nous pourrions trouver en France.

Qu’est-ce qu’essaie de dire Xavier Niel ici ? Que seule les écoles privées sont capables de fournir une formation de qualité ? Que parce que l’Université est ouverte au plus grand nombre, la qualité n’est pas au rendez-vous ? Mais quid des IUT, quid des écoles d’ingénieur publiques (INSA, réseau Polytech’ dont je suis issu, …) ? Je ne doute pas que la citation ci-dessus ait volontairement omis ces organismes publics, qui sont j’ose le croire capables de formations modernes, techniques et dont les professeurs sont au moins aussi dévoués que ceux du privé !

Il ne s’agit pas non plus d’un problème de sélection de talents à l’entrée, car ici, contrairement à la Faculté qui se doit d’accueillir quiconque s’inscrit, les IUT et autres écoles d’ingénieur publics ont un processus de sélection à l’entrée. Et sur ce point, j’ai une petite anecdote à vous raconter. Il y a deux ans, lorsque j’ai changé d’entreprise, j’ai eu besoin de récupérer mon diplôme d’ingénieur (oui, parce qu’on me demandait de prouver que j’étais ingénieur) et je suis donc retourné à mon ancienne école, Polytech’Nantes, où j’ai croisé un ancien professeur. Au détour de la conversation, j’ai appris qu’il y avait moins de candidats à l’entrée de l’école qu’il y a 10 ans. Visiblement, les nouvelles technologies'' n’attirent plus autant qu’avant. Ils avaient donc, à l’époque, un problème de sélection à l’entrée pour trouver des étudiants, tout simplement, sans parler de trouver des talents. A ce propos, je rejoins quand même Xavier Niel sur un point (du moins, si je lis entre les lignes). Les écoles d’ingénieur privées en France, on les choisit surtout parce que c’est marketing, ça fait bien sur un CV de sortir d’Epitech. Au passage, si tu es étudiant et que tu lis ces lignes, méfie toi des écoles qui ne donnent pas droit à un diplôme d’Etat en sortie, car comme tu l’as constaté, c’est utile… Ces écoles, on y entre souvent parce qu’on a les moyens de payer. Bref, c’est plus un investissement qu’un problème de qualité de formation… En pratique, il reste à démontrer qu’un ingénieur sorti d’une école privée est meilleur'' qu’un autre…

Alors pour rire un peu, lisons le programme de l’école 42: Merise, XWindow, Corba, J2ME, que des technologies toutes neuves et incroyablement répandues dans l’industrie… j’avais les mêmes cours lorsque je suis rentré à l’IRESTE en 1999 ! Alors, certes, il y a du HTML5 au programme, de l’iOS et Android. En revanche, je cherche encore Spring, le framework alternatif à JEE le plus répandu dans l’industrie, Scala (ou encore Groovy, rêvons un peu). En clair, ce programme existe aussi dans n’importe quelle autre formation, y compris dans le public. Je vous conseille d’ailleurs, au sujet d’iOS, cet ">excellent post de Julien Ponge sur l’abandon de l’enseignement d’iOS à l’INSA Lyon (Ecole d’ingénieur publique, faut-il le rappeler). En clair, ce programme, si intéressant soit-il, n’a rien de révolutionnaire. C’est un classique qu’on trouve dans n’importe quelle école. Ce à quoi on pourrait me dire que ça n’est pas dans le programme mais dans la façon de l’enseigner que se trouve la clé, mais là, je demande à voir. Il y a des cours, il faut bien les suivre. Si c’est faire des projets annexes pour bidouiller, etc, c’est sympa, mais ça, on le fait depuis longtemps ailleurs aussi. Non, je ne vois pas trop…

Le problème n’est donc certainement pas dans les programmes, ni dans la sélection. Ce qui m’amène à parler des raisons pour lesquelles les gens comme moi en viennent à choisir l’informatique. Je les divise en deux catégories:

ceux qui sont passionnés, tombés dedans quand ils étaient petits: j’en fais partie. J’ai commencé à coder à 8 ans, je n’ai jamais arrêté depuis. C’est une passion autant qu’un boulot et rares sont ceux qui comme moi parviennent à combiner les deux.
ceux qui y vont parce qu’il y a du travail : on ne cesse de le répéter, on manque de main d’oeuvre dans l’informatique. Ca n’est (pour l’instant) pas trop mal payé, et donc on y va parce qu’on peut bien manger. J’en ai croisé beaucoup et j’en croise encore.

Dans la première catégorie, je classe les professeurs. S’il existe un vrai problème, ça n’est certainement pas l’enseigement, ni les programmes. Non, l’informatique souffre d’un problème de reconnaissance, et il existe à de multiples niveaux. Le premier, c’est celui de l’enseignement. Lorsque j’étais étudiant, j’ai commencé par faire un DEUG, puis j’ai choisi de ne pas suivre la voie de la recherche (doctorat) et de poursuivre en école d’ingénieur. Etait-ce le programme qui était en cause ? Non, certainement pas. Etait-ce le problème des classes surchargées et du fait de ne pas être choyé ? Non plus: j’étais extrêmement heureux de l’autonomie qu’ont les étudiants de Fac, tout comme j’étais totalement dans l’esprit de devoir travailler fortement à la maison pour compenser le manque de travaux individualisés''. Non, le vrai problème, c’est le salaire. Le calcul était simple: en sortie d’école, je toucherai autant, voire plus, avec un diplôme d’ingénieur que mon prof à l’Université. Et lui était incontestablement plus compétent que moi. Alors, être professeur en France, aujourd’hui, c’est de la dévotion pure et simple. Je l’ai déjà dis en privé à certains, je le confirme : je vous admire les gars. Alors j’ai choisi l’école d’ingénieur (publique, pas question d’aller dans le privé). Là encore, je caricature un peu, l’idée de l’école était aussi qu’elle était plus proche du monde de l’entreprise et que la voie universitaire'' avec carrière définie à l’avance, les mutations et autres ne m’enchantait guère, mais ça c’est un autre débat…

Bref, pour entrer, sélection à l’entrée. Il fallait montrer patte blanche, passer un entretien avec un professeur et un chef d’entreprise, avoir de bons résultats et faire partie des heureux élus (la sélection comptait 1/3 de l’université, 1/3 de l’IUT, 1/3 de prépas). Une fois entré, le plus dur est fait. On passe 3 ans à suivre ses cours, à s’améliorer, mais je regretterai toujours l’ambiance de la fac, les révisions autour d’un café, les impressions de polys, … Il est clair que les moyens de l’école étaient plus élevés, par rapport à la fac, et qu’on est entre de bonnes mains. A moins de faire l’idiot, difficile de ne pas en sortir avec un diplôme. Mon attachement à la fac et au monde de la recherche ne s’est pas arrêté à l’entrée à la fac, non, puisque j’ai décidé de faire en parallèle un DEA en informatique, possibilité qui était offerte par l’école à l’époque. On devait alors suivre des cours supplémentaires à la Fac. J’y ai appris beaucoup, de personnes brillantes. Alors, à l’école, j’ai aussi eu des professeurs moins ``bons'' que d’autres, mais franchement, même en entreprise, on trouve de sacrés boulets, ça n’a rien à voir avec le fait qu’on soit dans une école publique.

Dans le cadre de mon double diplôme ingénieur/DEA, j’ai eu la chance de participer à la création d’une entreprise issue d’un projet de recherche, qui m’a embauchée à ma sortie d’école… Je n’aurais pas choisi un autre type d’entreprise. En particulier, je fuis les SSII et je cherche plutôt des éditeurs de logiciels.

Pourquoi ce choix ? Parce que je pense que les SSII font partie du problème en France. La formation technique à l’école n’était pas un problème mais il est clair qu’on nous met dans un état d’esprit plus que douteux allant directement dans le sens des SSII. Des cours de management, c’est intéressant, mais pourquoi s’enfoncer dans ce modèle détestable qui consiste à croire qu’au bout de 2 ans, on n’est plus capable de développer et qu’il faut passer chef de projet ? Alors certes, la catégorie 2 des personnes dont j’ai parlé plus haut s’en accomodent très bien, mais il est stupide de croire qu’un bon ingénieur ferait un bon chef de projet, tout comme il est stupide de ne proposer des augmentations de salaires qu’en changeant de poste et en passant à un poste de management. Si les entreprises, SSII en tête, prenaient moins les ingénieurs pour de la chair à commercial, on s’en sortirait déjà beaucoup mieux (nous les ingénieurs, l’entreprise, elle, c’est moins sûr). Même après 2 ans d’expérience, on est très loin d’avoir un niveau de compétence suffisant pour égaler la productivité d’un senior. Les experts, ça existe, et il faut les respecter. Le vrai problème est là: le respect du développeur. Le choix de devenir développeur senior est très difficile en France, de part cet aspect social assez négatif (non mais allô quoi ? T’as 30 ans et t’es développeur ? Non mais allô !) et le fait qu’avec l’âge, on aimerait bien s’offrir une maison quand même…

Au sujet des SSII, j’ai une petite anecdote. Il y a quelques années, l’entreprise pour laquelle je travaillais se portait assez mal, et donc par précaution j’ai commencé à passer des entretiens d’embauche. J’ai mis mon CV sur Les Jeudis'' et à peine quelques heures plus tard, j’avais déjà décroché un entretien… dans une SSII. Comme il faut bien manger et que les offres des éditeurs ne courent pas les rues (à mon grand désespoir), je suis allé à cet entretien en le prenant comme un entraînement. Et heureusement. Pour l’entretien, j’ai été confronté à un potentiel futur supérieur'', un commercial en fait qui m’a présenté cette grande SSII leader en France, n’embauchant que des dieux'' (je caricature à peine). D’une part, se dire qu’un ingénieur se trouve hiérarchiquement lié à un commercial me pose un sérieux problème. C’est qu’on vend un profil avant de vendre mes compétences. Il n’y avait aucun travail à me proposer. On m’a convonqué pour avoir un CV dans une base et le vendre à des clients. Ca n’est donc pas parce qu’il y avait un projet qu’on m’a convoqué, mais parce que potentiellement, je suis vendable. Alors, comme j’ai la chance d’être dans un domaine où il y a encore (un peu) de choix dans les offres d’emploi, non merci. Mais le plus choquant dans cet entretien, pour moi, ce fut la question finale, faite par ce commercial: pourriez-vous me résumer ce qu’on a dit dans cet entretien''. J’ai trouvé cette question humiliante. Déjà, j’étais venu en me disant qu’on avait un projet à me proposer, mais non, je commercial ne savait même pas que je n’avais pas postulé, mais qu’on m’avait cordialement invité à passer un entretien. Ensuite, mon égo en a pris un coup par cette question, qui sous-entend que je ne suis pas assez professionnel pour avoir compris le discours élitiste et commercial que l’on m’avait longuement rabaché dans cet entretien. Le lissage et la lobotomisation avait déjà commencé. Evidemment, on ne m’a pas rappelé et je ne m’en porte pas plus mal.

Ne soyons pas caricatural non plus, je suis certain d’être ``mal tombé'', mais au final, j’avais mon opinion sur les SSII et cet entretien n’a fait que la conforter. Il existe sûrement des sociétés plus petites où on n’est pas uniquement un CV dans une base…

Ce qui m’amène à un dernier point. Je suis un amoureux de l’esprit open source. Aujourd’hui, j’ai la chance d’en vivre et j’aimerais que beaucoup plus de développeurs puissent le faire. Ne serait-ce que pour redécouvrir leur métier. J’ai toujours rêvé de travailler dans une entreprise faisant de l’open-source, et quand je dis faisant de l’open-source, je ne veux pas dire utilisant des projets open source, non, je veux dire qui contribue directement à des projets OSS. Cet esprit me vient probablement de tout petit, lorsqu’à 10 ans, j’achetais ces revues dont j’ai oublié le nom, où je pouvais recopier le code source d’un jeu sur mon CPC, et après de longues heures de recopie, être émerveillé par ce que je venais d’accomplir.

Alors, oui, à l’époque, je n’inventais pas grand chose, mais je voulais et je pouvais bidouiller. Et je pense qu’il n’y a rien de plus instructif, en informatique, que la bidouille. Triturer du code, recopier, modifier, lorsqu’on est enfant, il n’y a rien de plus simple et on apprend énormément (aaaahhh, et donc là, je peux faire de l’overscan !). Il n’empêche qu’à une époque, j’ai eu besoin de structurer tout celà et que c’est à l’Université que ça s’est fait (comprendre les concepts qu’on manipule).

Au quotidien, l’open source tel que je le pratique à de nombreux avantages. D’abord, je suis consommateur. Je ne comprends pas ce que fait une fonction, j’ouvre le code, je le lis. Je peux aussi profiter du code et de l’expérience des autres (si tant est que la licence le permette). D’un autre côté, je contribue. Et en tant que développeur, je trouve que c’est extrêmement formateur. Lorsque j’ai commencé à développé sur Groovy, je pensais ne pas être mauvais (comme 90% des développeurs, je pense être meilleur que ma moyenne ;-)), pour autant, lorsque je lisais le code de nombreux projets open source, je me trouvais petit. Et encore aujourd’hui, je le suis, mais j’apprend. Faire de l’open-source, c’est par définition montrer son code à tout le monde, et en pratique, on n’aime pas spécialement montrer une mauvaise image de soi. Etre scruté par des milliers, voire millions d’yeux, c’est très formateur et ça pousse à s’améliorer. Ca ne veut pas dire qu’on ne fait pas de la m…, mais en tout cas, on la limite et si on en produit, on est vite rappelé à l’ordre… J’aime ce métier parce que je vois directement les résultats de ce que je produis : des utilisateurs contents (ou pas) et je n’ai rien à leur cacher. Bref, les qualités intrinsèques de mon ``produit'' passent avant toute chose. L’un des bons côtés de mon métier, c’est se balader dans une conférence et croiser quelqu’un qui vous dit que ce que vous avez fait lui a sauvé sa journée. C’est peut de chose, mais ça montre que le dev open source est avant tout tourné vers l’autre. Si je ne pouvais lire le code des autres, je ne mesurerais pas la marge de progression que j’ai et j’apprendrais beaucoup moins, ce qui signifie, au bout du compte, une qualité moindre et une dévalorisation générale du métier…

Alors non, clairement, je ne pense pas que le problème, en France, ce soit les écoles. Je ne crois pas non plus que le problème soit les enseignants, ou le public vs le privé. Et je ne crois pas qu’on perde des talents à cause d’un problème de sélection. Non, le problème, c’est la reconnaissance du métier. Faire que les gens soient fiers de leur travail et puissent en vivre dignement. En particulier, le problème du salaire des profs est à mon humble avis critique et explique que la France laisse disparaître des talents dans les méandres des inter-contrats. Non, commençons par définir des projets, faciliter l’innovation, arrêter de penser franco-français (le financement public du ``Cloud Français'' est un pur gaspillage, à mon humble avis) et défendons au contraire ce qui permet de faire émerger les talents: ouverture, neutralité du net, reconnaissance du travail. Arrêtez de demander des stagiaires avec 5 ans d’expérience. Bref, faites nous rêver !

Nous faire rêver, effectivement, Xavier Niel sait le faire, on peut lui reconnaître ça. En pratique, je ne suis pas convaincu de la solution, parce que je ne suis pas convaincu que le problème soit là. A moins que de cette école n’émerge une technique nous permettant de lire des vidéos YouTube sans saccade :-)

Static builders inception

Wed, 13 Feb 2013 00:00:00 +0100

Documentation is good for everybody

As part of the Groovy documentation effort, I started writing documentation for several features of Groovy, including compiler configuration, type checking extensions, and yesterday the @DelegatesTo annotation.

What is interesting when you do so, apart from the fact that this should have been done much sooner, is that it is interesting for our users, of course, but also for us. In particular, when I wrote the statically compiled builder example in the @DelegatesTo documentation, I found myself thinking there was a lot of repetitive code in there… And that gave me a new idea…

In the following post, I will show you how I combined several features of Groovy 2.1 to provide an implementation of a statically compiled HTML builder: static compilation, @DelegatesTo, meta-annotations, custom type checking extensions and of course AST transformations, without having to write all the boilerplate necessary to make it type safe! Nothing less…

Statically compiled markup builder

What if we could annotate a class as a markup builder, describe its schema, then, if a user wants to statically compile the builder usage, have a type-safe, statically compiled builder? That means, if you write this:

@groovy.transform.CompileStatic
String build() {
    def builder = new HTMLBuilder()
    builder.html {
        body {
            p {
                out << "Hello, this is "
                a(href:'https://groovy.codehaus.org') { out << 'Groovy' }
            }
        }
    }
}

Then you want compile-time errors if any of the tags used in the builder is not recognized or at the wrong place. As you know, builders are normally dynamic in Groovy, but the documentation for @DelegatesTo documentation showed that it was possible to create a statically checked (and statically compiled) builder.

There were two problems in the approach from the documentation:

unnecessary repetitive/verbose code (inner classes)
no ability to perform checks on the allowed attributes of a tag at compile-time

Here comes the statically compiled markup builder experiment!

The night hacker

So last night, I started an experiment to see if I could make things easier and after two hours of coding, I had indeed a first working implementation, which allowed me to define a schema for my builder, but didn’t check attributes yet.

Today, I spent several hours fixing bugs for Groovy 2.1.1, including some that prevented me from going further (including one with nasty classloading issues that prevented Gradle from loading some classes from my AST transformation…). After that, I could eventually fix my prototype and I now have a fully working implementation…

Checking out the sources

First of all, all the code is available on GitHub. It makes uses of the gradle wrapper, so all you need is to run:

    $ ./gradlew test

The builder

Creating a statically checked builder is easy:

    import groovyx.transform.StaticMarkupBuilder

    @StaticMarkupBuilder
    class HTMLBuilder {
        static schema = {
            html {
                head { title() }
                body {
                   p()
                   a(attributes:['href', 'target'])
                }
            }
        }
    }

For examples of several builders, you can take a look at the unit test.

What is important here is that using the @StaticMarkupBuilder, we are not using a builder, we are defining one:

a statically compiled builder
which accepts a predefined schema

Here, the schema is very simple. It says that at the top level, we can find the html tag. This tag can include either a head or a body tag. Below body, you can find either p or a, and if it’s an a, then the only allowed attributes are href and target.

It’s a very simple schema that of course doesn’t matches what you can do in HTML5, but remember that it’s a prototype aimed at showing off the amazing capabilities of Groovy 2.1.

To use it, you can do:

    @CompileStatic
    void test() { // using a method to ensure that the builder will be statically compiled!
        def out = new ByteArrayOutputStream()
        def builder = new Builder3(out)
        builder.html {
            body {
                p 'Hello, Groovy!'
            }
        }
        println out.toString()
    }

What we’re doing here is creating a builder inside a statically compiled portion of code (so that you can make sure that the builder usage is indeed statically compiled). If you run this code, it will show:

    Hello, Groovy!

But what is really interesting is showing what would happen if you use a wrong tag:

    @CompileStatic
    void test() { // using a method to ensure that the builder will be statically compiled!
        def out = new ByteArrayOutputStream()
        def builder = new Builder3(out)
        builder.html {
            bodyp { // compile-time error!
                p 'Hello, Groovy!'
            }
        }
        println out.toString()
    }

Here, Groovy will fail at compile-time!

    Groovyc: [Static type checking] - Cannot find matching method groovyx.transform.StaticMarkupBuilderTest#bodyp(groovy.lang.Closure). Please check if the declared type is right and if the method exists.
    Groovyc: [Static type checking] - Cannot find matching method groovyx.transform.StaticMarkupBuilderTest#p(java.lang.String). Please check if the declared type is right and if the method exists.

Of course, the error message can be improved (there are ways to do this), but what is interesting is that you really fail before the test gets executed!

Behind the magic

To acheive this first example, there are two things in action:

a meta-annotation, called @StaticMarkupBuilder that will expand itself into @CompileStatic and StaticMarkupBuilderGenerator: this way, using a single annotation, I am saying that the builder will be statically compiled without having to use @CompileStatic explicitely. The second annotation is an AST transformation:
an AST transformation that transforms the schema definition into a set of inner-classes, close to what the documentation says

The AST transformation is very powerful. Say you have this schema:

    static schema = {
        html {}
    }

Then the AST transformation will create an inner class called HTMLBuilder$HtmlTag, then also create, in the HTMLBuilder class, a method which name is html and uses a closure… Of course, it will add the @DelegatesTo annotation transparently, so the generated method signature will look like this:

    void html(@DelegatesTo(HTMLBuilder$HtmlTag) Closure body) { ... }

For sub-tags, the principle is the same, but instead of generating the method in the HTMLBuilder class, it is added to the appropriate inner tag class… Of course, the AST transform does a bit more work:

make the builder and each generated inner class extend a base class (AbstractTag), this class defining how a tag should be rendered.
generate methods for string only arguments (p `text') or empty tags
generate a method for tags accepting attributes

The latter is very important as it will help us demonstrate how we can also check that when the builder is used, the attributes being used are checked against a list of valid tags.

The attribute checking problem

Here, we have an interesting problem: our schema allows us to say that the a tag only accepts href and target attributes. But can we prevent the user from using other attributes at compile-time?

    builder {
        ...
        a(notGood:'invalid attribute') { out << 'Link text' }
    }

Let’s see what happens if we don’t perform additional magic. When the builder is checked by the type checker, it will find a method named a, which accepts a map and a closure as arguments. This method exists, so it passes compilation. This is bad, because we wanted a compile-time error here. We could have, when we generated the builder class, added a check into the a method body that verifies that the map keys are in the authorized list, but it’s a runtime check and here we want a compile-time one…

So we have a new challenge:

how to make the authorized attributes list known when the builder usage is compiled?
how to make the static compiler check that the user only picks attributes from that list

For the first part, my choice was to add an annotation in the map parameter of the generated method:

    void a(@CheckedAttributes(['href','target']) Map attributes, @DelegatesTo(...) Closure code) { ... }

Alright, so now we have the information available at runtime, when the compiler will select this method, the method will have an annotation that we can reflect to retrieve the list of attributes… Still, the compiler won’t use that information, so we’re doomed, right?

Not really, because Groovy 2.1 includes type checking extensions. A type checking extension will allow us to plug into the type checking system and perform additional checks that the compiler normally doesn’t do. Here, if it chooses this method, we will ask it to check the annotation and check that the actual arguments, those provided by the user, are using keys allowed by the builder!

The code of this type checking extension can be found here.

We’re almost done! The only problem with this solution is that one has to annotate every method that uses our builder with @CompileStatic(extension=’groovyx/transform/StaticBuilderExtension.groovy’). This is not as cool as seeing the extension being automatically applied…

Global AST transformations to the rescue

There’s one solution to this. We will use a global AST transformation here. Unlike local AST transformations that are triggered by seeing an annotation in the code, global AST transformations are automatically loaded by the compiler and applied to every class being compiled. Thanks to this feature, we will be able to scan classes being compiled and if we find something annotated with @CompileStatic, then change it to @CompileStatic(extensions=’…’). That’s all!

The code for such an annotation can be found here.

You must remember that using global AST transformations have a significant impact on compilation times, since they are applied on every class, so always make sure you don’t use unnecessary transforms.

Conclusion

The main focus of this blog post wasn’t to describe in details how the AST transformation works, but rather show you how a smart combination of the features of Groovy 2.1 can allow you to perform tasks that would seem impossible, like type checking at compile-time the usage of a builder, statically compiling the code and eventually, performing checks on things that are normally unchecked by the compiler (arguments of a call).

In the end, this example is not complete, of course:

the definition of the schema is insufficient (what about tags that allow arbitrary tags, …)
allowText attribute isn’t used yet
the error messages can be improved (yes, it is possible to replace them with tags!)
the API can be improved for more fluency

But what is more important is that using such techniques, I am pretty sure that someone even crazier than me could write a more complete implementation that would accept, say… a real schema (think of xsd)!

Using Groovy to play with invokedynamic

Thu, 31 Jan 2013 00:00:00 +0100

InvokeDynamic and Groovy 2

JVM language implementors, especially those working on dynamic languages like Groovy, welcome the arrival of invokedynamic in Java 7. InvokeDynamic is a new JVM instruction that was introduced to make our life easier. While this is only partially true, because we have to maintain backwards compatibility including older JVMs (Groovy still runs with JDK 1.5, for example), using invokedynamic to implement a dynamic language on the JVM is undoubtfully the way to go.

Groovy 2.0 came with initial support for invokedynamic and the recently released Groovy 2.1.0 pushes support one step further by completing the implementation (in Groovy 2.1, if you compile code with invokedynamic support, the call site caching technique that regular Groovy uses to make dynamic calls faster is removed). What invokedynamic brings to Groovy 2.1 is basically improved performance. One has to know that talking about performance is always polemical and invokedynamic performance is even more difficult, because from one JVM version to another, you can have significantly different results. The JVM is still not perfectly optimized for invokedynamic. However, you may know that Java 8 will introduce lambdas to the language and the implementation of such a feature heavily relies on invokedynamic, so the JVM implementors are doomed to improve performance of invokedynamic!

As a reminder, people wanting to test a Groovy runtime that uses invokedynamic have two things to do:

use the groovy-indy jar which includes support for invokedynamic
compile classes with the -indy flag

If you try to compile Groovy classes with -indy and the ``normal'' jar, you would have an error saying that indy is not supported, but if you compile a Groovy class using the indy jar without activating the -indy flag, then classes would be compiled with call site caching instead of invokedynamic. While this might be suprising, there are good reasons for that:

invokedynamic support classes are only available in JDK 1.7+. That alone wouldn’t be a big deal as we could have worked around using stubs
the Groovy core classes are, sometimes, written in Groovy, so we use a ``bootstrap'' compiler to compile Groovy classes. In the -indy jar, core Groovy classes written in Groovy are compiled with invokedynamic, not call site caching. Therefore, we produce JDK 1.7+ bytecode only. Since Groovy is compatible with JDK 1.5, if we activated indy for all, you would have had incompatible classes for older JVMs.

In this blog post, however, I will not talk about how Groovy 2 uses invokedynamic, but rather how you can use Groovy to test invokedynamic by yourself and start playing with the API without too much hassle.

Using Groovy as a sandbox for invokedynamic

The @Bytecode AST transformation

Two years ago already, I released a new AST transformation for Groovy called the @Bytecode transformation. Basically, the idea was that if you thought you were smarter than the compiler, you could write bytecode as a method body had have the compiler write it. Started as a joke, some people found a lot of interested in it (some others thought I opened the box of Pandora) but my idea has always been that it was an amazing tool for teaching.

What I’m going to show you, here, is an extension of the @Bytecode transformation that supports the invokedynamic instruction. The main reason to use this annotation is that there is no way of generating invokedynamic instructions using regular Java code. The only way to do this is to use a library like ASM. However, the ASM library is not easy to handle and it’s a pity that you have to spend time on bytecode generation tools just to be able to test invokedynamic. With the @Bytecode transformation, you now have a way of testing bytecode that makes use of invokedynamic very easily.

Setting up

First of all, support for invokedynamic in @Bytecode is still experimental. You’ll have to build the jar by yourself, but don’t worry, it’s very easy:

$ git clone https://github.com/melix/groovy-bytecode-ast.git
$ cd groovy-bytecode-ast
$ ./gradlew jar

If you already have Gradle 1.4 installed, the build should only take a few seconds, otherwise Gradle will be downloaded for you.

Now, we’re going to create a Groovy project with a simple script that we’re going to run with invokedynamic. Althought this is not mandatory, I strongly suggest you to use the Gradle wrapper (which avoids installing Gradle everywhere), so we’ll just take advantage of the fact that the bytecode transformation is built with the wrapper to create a new project without having to install Gradle 1.4!

$ cd ..
$ mkdir bytecodetest; cd bytecodetest
$ cp -Rp ../groovy-bytecode-ast/gradle* .

Here we’ve just copied the Gradle wrapper into a empty project, now we’re going to populate it. Open your favorite editor and paste the following code in a build.gradle file:

apply plugin: 'groovy'
apply plugin:'application'

repositories {
    mavenCentral()
}

dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.1.0:indy'
    compile fileTree(dir: 'lib', include: '*.jar')
}

sourceCompatibility = 1.7
targetCompatibility = 1.7

[compileGroovy.groovyOptions,compileTestGroovy.groovyOptions]*.with {
    fork = true
    useAnt = true
    optimizationOptions = [ indy: true, 'int': false]
    encoding = 'UTF-8'
}

mainClassName = 'Main'

What does this build file do? Basically, we’re using the Groovy plugin to compile Groovy files, including the indy'' version of Groovy 2.1.0 (note the `indy' classifier in the dependency and the fact we’re using the -all version of Groovy as it’s the only one compatible with @Bytecode) and configuring compilation so that it makes use of invokedynamic. The application'' plugin will allow us to run our script directly from Gradle once it’s compiled. For that, we’re just using a ``Main'' class.

Last step is to copy the @Bytecode jar into the ``lib'' directory of your project:

$ mkdir -p src/main/groovy
$ mkdir lib
$ cp ../groovy-bytecode-ast/build/libs/*.jar lib

Now we’re ready to create our first script! Create the src/main/groovy/Main.groovy file with the following content:

println 'Hello, indy Groovy!'

Run the build:

$ gradlew run
:compileJava UP-TO-DATE
:compileGroovy
:processResources UP-TO-DATE
:classes
:run
Hello, indy Groovy!

BUILD SUCCESSFUL

Total time: 5.291 secs

Congratulations! At this point, you have generated a Groovy script which does compile using the invokedynamic version of Groovy. You can verify that it’s the case dumping the generated bytecode:

$ javap -v build/classes/main/Main.class
public java.lang.Object run();
    flags: ACC_PUBLIC
    Code:
      stack=2, locals=1, args_size=1
         0: aload_0
         1: ldc           #55                 // String Hello, indy Groovy!
         3: invokedynamic #61,  0             // InvokeDynamic #1:invoke:(LMain;Ljava/lang/String;)Ljava/lang/Object;
         8: areturn

[...]

Excellent! Now, I said that our focus wasn’t testing Groovy with invokedynamic, but rather the opposite: testing invokedynamic using Groovy. For this, what we would like to do is generating a method which introduces an invokedynamic call and wire the target method by ourselves. For example, take this code:

int bar() { 666 }

int foo() { bar() }

10.times {
   println foo()
}

the foo() just delegates to ``bar''. Now, instead of letting Groovy generate the bytecode for us, we want to generate the invokedynamic instruction by ourselves, then write the bootstrap method that will hardwire the link between bar() and foo().

First, we need a bootstrap method. The role of this method is, when an invokedynamic instruction is found, to setup the callsite, that is to say create a link between a method call in bytecode and an actual target method being executed. That is to say that with invokedynamic, the relation between a call site and the method that will effectively be called is done at runtime, not compile time.

public static CallSite bootstrap(Lookup lookup, String callType, MethodType type) {
    new ConstantCallSite(lookup.findVirtual(Main, 'bar', MethodType.methodType(int)))
}

Here, we’re creating a constant call site, which means that once a target method is chosen, it will never change for this call site. We’re linking our call site to a virtual method on the Main class, called `bar' and returning an `int'.

Now, instead of letting Groovy do the job, let’s generate the method body by ourselves, so replace the foo() method with the following code:

@Bytecode
int foo() {
    aload 0
    invokedynamic 'experiment', '(LMain;)I', [H_INVOKESTATIC, 'Main', 'bootstrap', [CallSite, Lookup, String, MethodType]]
    ireturn
}

Uh! That’s bytecode! Yes, and actually, it also hides all the complexity of the ASM library thanks to a nice Groovy DSL. The first instruction, aload 0 just loads the receiver on stack, that is to say ``this''. The second instruction is our method call, the one we want to be linked to bar() and eventually, ireturn takes the result of the call and returns it as an int.

Let’s explain a bit what are the arguments of the invokedynamic method call. The first one is a label. You can put whatever you want in this, as long as it helps you. It’s often interesting if you have to debug your code, but it can also contain information that you want to have at hand when you will select the target method. The second one is the signature of the method you will call. Here, we say it’s a method on Main that returns an int. The last argument is a method handle to the bootstrap method. We’re saying that it’s a static method (INVOKESTATIC) found on the `Main' class, named `bootstrap' and accepting the arguments of types [CallSite, Lookup, String, MethodType]. Invoke dynamic supports more arguments for the bootstrap method, but it’s not discussed here. Putting it altogether:

import groovyx.ast.bytecode.Bytecode
import java.lang.invoke.*;
import java.lang.invoke.MethodHandles.Lookup;
import static groovyjarjarasm.asm.Opcodes.*
import static java.lang.invoke.MethodHandles.*

public static CallSite bootstrap(Lookup lookup, String callType, MethodType type) {
    new ConstantCallSite(lookup.findVirtual(Main, 'bar', MethodType.methodType(int)))
}

int bar() { 666 }

@Bytecode
int foo() {
    aload 0
    invokedynamic 'experiment', '(LMain;)I', [H_INVOKESTATIC, 'Main', 'bootstrap', [CallSite, Lookup, String, MethodType]]
    ireturn
}

10.times {
    println foo() // prints 666
}

Let’s run it!

$ ./gradlew run
:run
666
666
666
666
666
666
666
666
666
666

BUILD SUCCESSFUL

Total time: 5.762 secs

And we did it! The link between the invokedynamic call site and our bar() method is dynamic, handled by our bootstrap method. To convince yourself, let’s just add a new method called baz():

int baz() { 123 }

And now, replace, in the boostrap method, `bar' with `baz'. Run the build again, and see what happens:

$ ./gradlew run
:run
123
123
123
123
123
123
123
123
123
123

BUILD SUCCESSFUL

Total time: 5.693 secs

Easy? Now you’re ready to play with the invokedynamic API. For example, our next challenge was to call bar() and baz() alternatively. For that, your friend is going to be the MutableCallSite class, instead of ConstantCallSite. The difference between the two is that a MutableCallSite allows the target method handle to be changed over time. Here’s our code:

import groovy.transform.CompileStatic
import groovyx.ast.bytecode.Bytecode
import java.lang.invoke.*;
import java.lang.invoke.MethodHandles.Lookup;
import static groovyjarjarasm.asm.Opcodes.*
import static java.lang.invoke.MethodHandles.*

public static MethodHandle findMethod(Lookup lookup, String name) {
    lookup.findVirtual(Main, name, MethodType.methodType(int))
}

public static CallSite bootstrap(Lookup lookup, String callType, MethodType type) {
    def (bar,baz) = ['bar','baz'].collect { findMethod(lookup, it) }
    def callsite = new ConstantCallSite(
            insertArguments(
                    lookup.findVirtual(Main, 'selectAndCall',
                            MethodType.methodType(Object, MutableCallSite, MethodHandle, MethodHandle))
                    , 1, new MutableCallSite(type), bar, baz).asType(type)
    )
    callsite
}

@CompileStatic
public def selectAndCall(MutableCallSite callSite, MethodHandle bar, MethodHandle baz) {
    callSite.with {
        target = (target.is(bar))?baz:bar;
        dynamicInvoker().invokeWithArguments(this)
    }
}

int bar() { 666 }
int baz() { 123 }

@Bytecode
int foo() {
    aload 0
    invokedynamic 'experiment', '(LMain;)I', [H_INVOKESTATIC, 'Main', 'bootstrap', [CallSite, Lookup, String, MethodType]]
    ireturn
}

10.times {
    println foo() // prints 666 and 123 alternatively
}

Now run this:

./gradlew run
:run
666
123
666
123
666
123
666
123
666
123

BUILD SUCCESSFUL

Total time: 6.002 secs

Let’s explain a bit what we did… Calling a different method each time requires us to change the target handle. However, the bootstrap method is only called once. The trick, here, is to create a handle to a new method, here selectAndCall, that takes our MutableCallSite as an argument. Then, we create a ConstantCallSite for which the target method is this method with the first argument bound to our mutable call site. This means that instead of calling bar() or baz() directly, we’ll be calling selectAndCall which will change the target method then call the bar() or baz() method.

Conclusion

In this post, we’ve explained to you how you can leverage Groovy to test the invokedynamic API by yourself. I spent several months working on static type checking and static compilation for Groovy, without finding too much time to work with invoke dynamic. The next major version of Groovy should come with a new MOP (meta-object protocol) that heavily relies on the invokedynamic API, so this ``tool'' is also a good way for me to learn the API and play with it very easily. I hope this will be useful for you too!

Groovy&Grails eXchange 2012, type checking and upcoming conferences

Wed, 19 Dec 2012 00:00:00 +0100

Long time no see!

It’s a long time since I last blogged. I’ve been quite busy of course, but it doesn’t give me a good reason not to maintain this blog!

The Groovy&Grails eXchange 2012

Last week, I spoke at the Groovy&Grails eXchange 2012. It was a really pleasant experience and I wanted to share some feelings with you.

The ``static'' guy

There is something very funny when I meet people now. One or two years ago, people knowing my name were just a few, focused, guys on the mailing list. Now, with the work I’ve done in Groovy 2, that is to say static type checking and static compilation, more and more people recognize me and even those who knew me before start to think of me as the ``static guy''.

The truth is that while it’s correct that my main focus was to develop those features and that I really enjoy it, I spent a lot of time working with Groovy before, and advocating its dynamic nature to my coworkers. The dynamic nature of the language helped us a lot and it’s definitely the main focus of the language. This means that in the dynamic vs static language war, I’m in the middle camp: I like both!

This directly drives me to the most surprising talk given at the Groovy&Grails eXchange, I mean Why Groovy when Java 8 or Scala or…'' by Russel Winder. When I say surprising, I mean that it’s not often that you can hear War Pigs'' from Black Sabbath before the talk begins, and it’s not often either that in a Groovy conference, a speaker tries to demonstrate that Groovy is no longer necessary. At least, he tried to, because knowing him as one of the best Groovy advocates in town, I was just wondering all the short 45 minutes session along, when turnaround would occur. It did not. Maybe because Russel expected the session to last 1 hour, but even with 15 more minutes, I’m not convinced that he would, because if I understood him well, he wanted by this talk to conduce the Groovy community to be more active in the JVM community in general, by proving by the fact that Groovy is a major language, not just a poor substitute to Java 8.

I think I’m not too far from the truth when I read this very funny comment about me in his blog post: Cédric Champeau the person driving the static type checking and compilation was in the audience and I survived. Perhaps I wasn’t contentious enough.. I’m sorry if I disappointed you, Russel, but as I’m not in particular the ``static guy'' people think I am and that I have a very clear idea of what static compilation in Groovy is good for, I think it makes it difficult to make me unconfortable on the subject. In fact, I’m most worried about what people would think of this talk just reading the slides! Another thing, maybe, is that humour and cultural references in this talk were very british centered (that makes sense, we were in London after all) and that it was sometimes difficult to follow for me.

Static type checking for Groovy

Let me explain again why we introduced static type checking and static compilation in Groovy:

Java scripting: (which has absolutely no relation to Javascript). A lot of people use Groovy as a Java scripting language. They do not use dynamic features of Groovy, and they are often disappointed when the compiler doesn’t offer the same kind of feedback as the Java compiler.
Groovy as a better Java: Groovy syntax is more concise, Groovy has closures, learning curve is close to zero. Ah, but Groovy is a dynamic language. Period. @TypeChecked to the rescue!
Performance. You always have to be careful here, because as I always explain, Groovy is not slow (anymore). It even perfoms really good when you think of all the wonders it does before invoking methods. But in some situations, it is significantly slower than Java. For those cases, you can use @CompileStatic and have generated bytecode very close if not equal to what Java produces, so acheive the same level of performance.

What is interesting regarding performance is what Russel said in his talk: Java 8 will have lambdas, type checked lambdas (and using the new collections APIs the type checked syntax can even be more concise than the Groovy one, that’s a problem we must address) and for it, it makes use of the invoke dynamic feature that was introduced in Java 7. As you know, Groovy 2 introduced usage of invoke dynamic too, and Groovy 2.1 pushes it further enough that performance of dynamic Groovy is close to what Java offers (as always, be careful with figures, even more when talking about invokedynamic because from one VM to another you can have very different results). But that’s what we hoped from day 1 when adding invoke dynamic support to Groovy: we seriously hope that performance of dynamic languages, and here Groovy, will become close to the performance of static languages. So if performance is close to Java, why did we introduce static compilation in the same version as invoke dynamic? There are various reasons for that:

unfortunately, people will not use Java 8 in production before years. Java 7 in production is still rare, most people are stuck with Java 6. We had to offer performance for those people too.
type checking: as explained before, even if you use a dynamic language, some people just want compilation errors
enforced semantics: statically compiled code is immune to monkey patching. Any code compiled with invoke dynamic support will have the same behaviour as dynamic groovy, so will go though the meta-object protocol. This is not a problem, this is what we want and the good thing is that indy (invoke dynamic) makes it fast, but it can be a problem for framework creators, for example, that do not want their code to be subject to metaclass changes in a third party library.

Russel’s talk echoed to the last talk from day 1, which was an open discussion with the Groovy community. One of the covered subjects was why would people use static type checking or static compilation for Groovy. There were a lot of interesting comments but one of them shocked me: developers were saying that static type checking wasn’t necessary because you write unit tests. Ok, let me be very clear on the subject: I’m not against test-driven development (though I usually prefer writing unit tests as I implement things), but I seriously think that focusing on unit tests is a very big error:

static type checking is about catching errors early, without having to run code. This is very important to understand
people do not write tests, and if they do, it’s always not enough. If you do, you can only write tests for what you think of. Just as well as 100% code coverage doesn’t mean no bugs, thousands of unit tests doesn’t guarantee you’re covering all cases.
DSLs: Groovy is a platform of choice for implementing DSLs. The thing is that a lot of DSLs are not aimed at developers. They are often aimed at people that know business rules and can focus on some formalism to describe them. Those people are not developers and you cannot expect them to write unit tests. That doesn’t mean we can’t do anything for them, and that’s also what my talk was about

To be short, Groovy puts itself in the middle of the dynamic vs static war for very good reasons. One of them is choice. I like to have the choice to use the best strategy for my needs. If it’s dynamic code, let’s do it. If it’s static code, do it too. If you absolutely want type checking, you can, and if you want to use dynamic Groovy and still be able to throw errors at compile time for people using your DSL, now you can too… Because if you ever want to compare Groovy with Java 8, never ever forget that Groovy is more than statically compiled Groovy compared to Java 8. It offers both runtime and compile-time programming which makes it unavoidable if you want to implement a DSL on the JVM…

Static type checking for DSLs

GGX was the first conference where I talked about a new feature in Groovy 2.1, which is ``customizable'' type checking. The idea is to help the type checker where it fails, because you know more about your code, what it does, how it is implemented than it does. This allows very nice features, like the ability to throw errors at compile time where Groovy would normally fail at runtime, but using DSLs, independently of their implementation. This is in particular important for people that implement DSLs using dynamic features of the language, because the type checker is obviously totally lost in that situation. Having the ability to check a user script at compile time, report errors before it goes in production is a big win here. You can even have the compiler be smarter than Java (or another statically compiled language) is given your domain.

The talk was rather technical and dealing with such a subject in only 45 minutes was a challenge (that I only half acheived) but I had good feedback. For those who are interested in the subject and missed this talk, you can still attend my session at the Greach conference in Madrid in the end of January. I hope I’ll be able to do it at GR8Conf in an extended version in May too. Of course, you can still watch the recorded session online.

I think this is a very powerful feature that opens a lot of new capabilities just like adding AST transformations to the language drove Groovy to a superior level. I’m eager to see what people will be able to do with it, even if it requires, that’s true, some technical skills, in particular some knowledge about the Groovy AST (abstract syntax tree). For people who are used to writing AST transforms, that shouldn’t be much a problem.

Meeting the community

Talking at conferences is always the best way to meet the community and gather feedback and new ideas from actual users. We are not, as Groovy or Grails committers, super heroes. I’m always amazed to see what users manage to do with what we implement in the language and I tend to think you are the real super heroes. At GGX, once again, I was really impressed with what Spock offers, for example. The talk was given by Rob Fletcher who also gave another talk about Grails for Hipsters. Both of his talks were so attended that at this point, the conference could have been named ``The Rob Fletcher eXchange''. If you want someone to demonstrate the superiority of dynamic languages over static ones, he’s your man.

Oh wait, did I say dynamic? ;-)

Last word as a special thanks to Peter Ledbrook who left VMware a few weeks ago. He did a lot for this GGX conference but more than that, he was the best Groovy/Grails advocate and we’ll definitely miss him!

My Git tweaks for daily usage

Wed, 25 Jul 2012 00:00:00 +0200

A few tricks for daily Git usage

Since Groovy switched to Git for version control, and more recently to GitHub for hosting, I’ve had great pleasure working with this tool. In this post, I’ll just show you some basic tricks that I use on a daily basis.

Adding the git branch to the Bash prompt

My first tweak is to have my bash prompt show me the current branch I’m working on. This is very handy. I borrowed the setup from this pastebin but unfortunately, I can’t find the original author. Note that it also adds support for svn. Add the following to your .bashrc file:

#GIT
simple_git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1/'
}

parse_git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(git::\1)/'
}
parse_svn_branch() {
  parse_svn_url | sed -e 's#^'"$(parse_svn_repository_root)"'##g' | awk -F / '{print "(svn::"$1 "/" $2 ")"}'
}
parse_svn_url() {
  svn info 2>/dev/null | grep -e '^URL*' | sed -e 's#^URL: *\(.*\)#\1#g '
}
parse_svn_repository_root() {
  svn info 2>/dev/null | grep -e '^Repository Root:*' | sed -e 's#^Repository Root: *\(.*\)#\1\/#g '
}
export PS1="\[\033[00m\]\u@\h\[\033[01;34m\] \w \[\033[31m\]\$(parse_git_branch)\$(parse_svn_branch) \[\033[00m\]$\[\033[00m\] "

Some aliases

All those aliases must be added to your .gitconfig file into the [alias] section.

One thing I’m doing often with Groovy development is to fix some bug into a branch, then backport the fix to other branches. My preferred tool for this is cherry-pick. Cherry picking a commit is really easy, but to make it even easier, I’m using an alias that shows me the hash of the last commit on a branch:

[alias]
 last = log --oneline -n 1

Then calling git last will show me the hash of the commit to be cherry-picked. Another problem is that git cherry-pick is a bit long to type, especially if you do this often, and git cherry doesn’t get autocompleted when you press tab, so I’m using another alias for cherry-picking:

[alias]
 cp = cherry-pick

If you are used to svn, you also know that co is an alias for checkout so I’m adding this too:

[alias]
 co = checkout

Eventually, here is a nice history view of commits:

[alias]
 hist = log --oneline --graph --decorate --all

That’s all, those aliases are currently the only ones I’m using on a daily basis, but of course, I’m relatively new to git, so there are chances that new aliases will show up in the future!

Improved AST transformation testing in Groovy 2

Thu, 12 Jul 2012 00:00:00 +0200

Structural (AST) testing

Since Groovy 1.6, the language provides a very nice feature called AST transformations. AST transformations allows you to hook into the compilation process in order to modify the internal representation of the source code at compile time. This internal representation, known as the Abstract Syntax Tree(AST) is a very powerful way to enhance the language or reduce verbosity of the code.

The different releases of Groovy came with new AST transformations such as:

@Log/@Log4j/@Slf4j: transparently adding a logger to your class
@ToString: Generating a nice toString implementation for your class
@EqualsAndHashCode: Generating the famous equals and hashcode methods
@Canonical: Combining @ToString and @EqualsAndHashCode
@WithReadLock/@WithWriteLock: for guarding method bodies with read/write locks (reentrant locks)

and many more!

Developing such AST transformations is very interesting but requires a deeper knowledge of the internal APIs of the Groovy language (specifically, the AST part). However, people who have developped such AST transformations know that testing them is a bit difficult:

The most useful tool is the AST browser which can be found in the Groovy Console. It allows you to ``browse'' the AST of the code which is in the console. This is very useful because it allows you to write the code you’d like to generate and see what AST structure you need to produce. Or you can use it the opposite way, looking for patterns of AST to recognize in order to transform them.
Testing if the AST transform is successul can only be done externally, that is to say by testing that the effects of the transform are visible on a class. For example, testing the @Log AST transform consists in applying the transform on a test class, then add calls to log and see if they are successful.

External testing is easy to do, but the main problem is that you no longer manipulate an AST: you manipulate classes which have already been generated. This is problematic for several reasons:

You can only test when your AST transform code is valid, that is to say that it doesn’t throw an error in the compilation process.
The Groovy compilation process is separated into phases, and it is useful to add tests to check was exists in one phase, and what exists in another phase.
You can only check easily things that introduce/remove fields or methods. It’s impossible to test, for example, an AST transform which doesn’t touch the AST…
You cannot check for properties found in the AST

The two last points were my major concern when I developped the type checker for Groovy 2. As you may know, the type checker is implemented in the form of an AST transformation which annotates the AST with node metadata. This node metadata is not visible in the AST browser and is of free form. In the case of the type checker, node metadata consists of type inference information. So, in that case, testing my AST transformation consisted of scripts with variables defined with certain types, and just checking that the compiler didn’t throw any error (or did throw an error). The problem is that you cannot check if the inferred type of a precise AST node is correct.

Improved AST transformation testing

@ASTTest

For that reason, Groovy 2.0 introduces a new AST transformation which is precisely aimed at testing AST transformations! This AST transformation is named ASTTest and allows you to execute assertions on the AST tree. Let’s take a simple example:

    for (int i in 1..n) {
        @ASTTest(phase=INSTRUCTION_SELECTION, value= {
            assert node.getNodeMetaData(DECLARATION_INFERRED_TYPE) == int_TYPE
        })
        def k = i
    }

This test is extracted from the type checker test suite. You can see:

we’re annotating a declaration with ASTTest
we specify, as an annotation parameter, the compilation phase the test is expected to run
and we specify code which will be executed on the annotated AST node

In this case, node in the closure refers to the annotated AST node, that is to say, here, a DeclarationExpression. We’re calling getNodeMetaData which is a method defined on any AST node which allows to retrieve the node metadata that we talked about earlier. Here, we want to retrieve the type of the variable which is inferred by the type checker. Last and important, we add an assert statement to check that this inferred type corresponds to an int.

If the assertion fails, the unit test will fail. This is very important, because we now have a mean to test the AST itself. I said it was impossible before, but in fact, it was possible but it required a lot of trickery such custom classloaders and so on…

In Groovy 2.0.0, ASTTest does nothing more than this, but we have a problem: not every node in the AST can be annotated. We’re limited to what it is syntactically possible to annotate: classes, methods, types, packages, declarations, … If what you want to test is a node which doesn’t belong to that category, then you need to annotate a wrapping node (in most situations, a method) then ``browse'' manually to the node you want to test. For example, in the previous example, we could access the right hand side of the declaration expression using node.rightExpression and perform assertions on that.

However, if you start from a method and try to access a particular node into the method body, I admit it is quite painful, because you have to check what is the exact AST tree which is generated by the compiler, then find a ``path'' to your node. And it is very easy to break this path just by adding a statement in the code, which makes the unit tests fragile.

Improving @ASTTest

For that reason, I wrote an utility class which will allow you to perform ``smart'' lookups in the AST tree. This helper class is not available in Groovy 2.0.0 but it is easy to add and I will probably introduce it in the next release(edit: see GROOVY-5597): it is called LabelFinder. The idea is that even if you cannot annotate everything in the code, it’s still very easy to add labels to your code. Then, Groovy may be able to search for a particular label and return to you the list of AST nodes which are defined on such a label:

void foo() {
    def x = 1
    x++
    forLoop:
    for (int i=0; i
Here, we added a forLoop label in the code, and we will use it as a lookup point for @ASTTest:

@ASTTest(phase=SEMANTIC_ANALYSIS, value= {
     lookup(node, 'forLoop').each {
        assert it instanceof ForStatement
     }
})
void foo() {
    def x = 1
    x++
    forLoop:
    for (int i=0; i
The major advantage of this technique is that you don't suffer the structural code syndrom anymore: the test is not fragile with regards to AST changes anymore. If you introduce
statements before the forLoop label, this won't change the result of the call to lookup. If you don't use such an utility, you would have to change, for example, the index
of a statement in a block...
In this example, I made use of a static import to reduce the verbosity of the test code. You may be interested in seeing the actual code of LabelFinder. It's actually quite simple:

public class LabelFinder extends ClassCodeVisitorSupport {


    public static List lookup(MethodNode node, String label) {
        LabelFinder finder = new LabelFinder(label, null)
        node.code.visit(finder)

        finder.targets
    }

    public static List lookup(ClassNode node, String label) {
        LabelFinder finder = new LabelFinder(label, null)
        node.methods*.code*.visit(finder)
        node.declaredConstructors*.code*.visit(finder)

        finder.targets
    }

    private final String label
    private final SourceUnit unit

    private List targets = new LinkedList();

    LabelFinder(final String label, final SourceUnit unit) {
        this.label = label
        this.unit = unit;
    }

    @Override
    protected SourceUnit getSourceUnit() {
        unit
    }

    @Override
    protected void visitStatement(final Statement statement) {
        super.visitStatement(statement)
        if (statement.statementLabel==label) targets << statement
    }

    List getTargets() {
        return Collections.unmodifiableList(targets)
    }
}

Enjoy!

I hope you liked this little introduction to a simple yet powerful tool introduced in Groovy 2. In sincerely hope this will make the life of AST transformations developpers easier!

Gradle plugin for CloudFoundry now supports standalone applications

Tue, 3 Jul 2012 00:00:00 +0200

gradle-cf-plugin upgraded to 0.2

I have just published a new version of the Gradle plugin for CloudFoundry. The main change is that it supports standalone applications, meaning you will be able to push your applications to CloudFoundry all from Gradle.

To deploy a standalone application, you can do the following:

buildscript {
       repositories {
         mavenCentral()
       }
       dependencies {
           classpath group: 'org.gradle.api.plugins', name: 'gradle-cf-plugin', version: '0.2.0'
       }
   }

   apply plugin: 'cloudfoundry'

Then add to your build.gradle file a section describing your deployment settings (note the standalone framework):

cloudfoundry {
   target='https://api.vcap.me'
   username = 'user-AT-domain-DOT-com'
   password='foobar'
   application = 'myapp'
   framework = 'standalone'
   applicationFramework='standalone'
   runtime = 'java'
   command = "$name/bin/$name"
   file = distZip.archivePath
   uris = ['https://myurl.vcap.me']
   memory = 256
}

If you migrate your application from version 0.1.0 to 0.2.0 of the plugin, then you have to change the warFile property to file. Now, you are able to push your application to CloudFoundry using the cf-push task:

gradle cf-push

Note that in future versions of the plugin, we will likely change the name of the tasks so that they follow the Gradle conventions (using, for example, cfPush instead of cf-push).

Groovy 2.0 from an insider

Fri, 29 Jun 2012 00:00:00 +0200

Groovy 2.0 is has landed!

It’s with great pleasure that we released Groovy 2 yesterday. For me, this is a special release and some kind of achievement because this is the first one for which I had a large contribution (a bit selfish I admit, but well…). Before that release, I had contributed code, then became committer, but I didn’t have much time to give to the Groovy project. Now that I’m working full time on it, it’s a bit easier!

In this post, I will try to investigate some of the new features of Groovy 2 and provide you additional details that you won’t find in the announcement like some technical insight. There fore, it’s a good complement to the introduction that you can find on InfoQ.

Modularization

Modules

This is probably one of my favorite features of Groovy 2. In this version, we made a lot of effort to split Groovy into several modules. This effort, driven by Paul King and backed by a Groovy Enhancement Proposal has introduced several major changes in the codebase:

A build which is now fully based on Gradle
Core Groovy only contains the minimal set of classes required to run the language
Several optional modules in their own source subtree
an extension module mechanism has been put in place

There are dependencies between modules. Actually, if you look at the Gradle build, you would find a rather complicated dependency graph, but most of the complexity is just that we need to build a minimal core of Groovy (written in Java) to build the full Groovy project itself. As for the modules, here is the current dependency graph (click for larger image):

Each module is available as a Maven artifact, under the groupId org.codehaus.groovy and an artifactId corresponding to the name on the schema. The core Groovy module is named groovy (not groovy-core) for compatibility with previous releases (you just need to upgrade the version number). We also provide a groovy-all jar which bundles all those modules in a single jar, as well as packaging some dependencies into a separate package to avoid class loading issues (in particular with ASM which is used by a lot of frameworks, but in different versions). If you upgrade from a previous version of Groovy, there are only two possible paths: if you were using the groovy-all jar, then just upgrade to the new groovy-all version. If you were using groovy.jar, then it is possible that some of the classes you were using have now moved into a module. In that case, include the module jar as a dependency.

It’s worth noting that Groovy 2.0.0 is just a milestone in the modularization task: the following releases of Groovy will add new features, like the ability to add custom file extensions that actually do something different (like transparently applying AST transformations).

Extension modules

When I talked about modularization, I told you that this was one of my favorite features. Now, I’m going to explain why! Groovy is more than a language. It also provides a lot of helpful APIs to reduce code verbosity. For example, if you have the java.io.File class in Java, there is no getText(String encoding) method on it which allows you to retrieve the contents of the file, although this is a quite common operation. Groovy adds this method onto the File class. This is sometimes (incorrectly) called known category methods or to be more correct_extension methods_. In previous versions of Groovy, most of those methods were defined in a class named DefaultGroovyMethods. With modularization, we have exploded that class into multiple helper classes, but more importantly, we added the ability for you to add your custom extension methods!. In the past, if you wanted to add a method named foo on the String class, there weren’t many options:

write a Category class, then use it: while this is quite easy to do, it has major drawbacks. First, it is lexically scoped and thread bound. This means that it is a dynamic feature, and that opening a use block influences all Groovy code being executed until the block is closed, including method calls. In other words, category usage ``leaks'' into the current thread (which can have its own advantages but I don’t like side effects). Moreover, it cannot be statically checked (which is one of the major additions of Groovy 2). Additionally, categories are performance killers, as they imply disabling a lot of possible optimizations. I think you understand, now, that I really don’t like categories!
use a metaclass: it is very easy to do, but once again, it’s a pure dynamic feature, so it cannot be statically checked. It also has the disadvantage of not being easily reusable, and eventually, you may still be in conflict with another part of the code replacing the metaclass (which would remove your changes). Unlike categories, metaclass changes are visible to every thread and have immediate effect on every Groovy class. This is why this feature is the perfect candidate for monkey patching.
write an AST transformation: done at compile time, this would transform your foo call into another call. This has the advantage of beeing done at compile time, also meaning that they can be statically checked (and compiled!), but writing AST transformations requires knowledge of (part of) the Groovy compiler infrastructure (specifically, the AST part). In general, unless you already have written AST transformations, this is much slower to develop than playing with a metaclass. It is also much harder to debug. But in the end, this is a very powerful feature and performance-wise, it is in general better than metaclass hacks. One important thing with AST transforms, though, is that they are selectively applied: you may choose on what classes they run, meaning that it’s not necessarily a global feature.

With Groovy 2, we provide a new way to write such extension methods, in a very friendly way: basically, you just need to write a support class that ``looks like'' DefaultGroovyMethods or Groovy categories. I will not repeat here how you can write such extension methods, but documentation can be found here. This technique has many advantages:

easy to write
easy to share (modules are jar files which may be included as a dependency)
compatible with static type checking and static compilation

This is so easy to put in place that we already have, for example, an extension module from Tim Yates adding generators to Groovy. Another interesting thing with the way extension modules are written is that many third-party libraries already use that kind of extension mechanism. Think of Apache Commons StringUtils class for example. Every method in that class is a static method taking a String as the first parameter. With extension modules, this means that all you have to do is to write a module descriptor referencing that class to have those methods added directly on the String class!

For advanced programmers, the extension module framework that have been added to Groovy allows you to register MetaMethod_s to Groovy. This means that it not necessary for you to follow the static method way of doing extension methods: you could implement your own! Whatever technique you choose, you must be aware that just like _metaclass tricks, extension methods are global: you cannot choose on what lexical scope they apply. Once the hooks are installed, extension methods are there forever.

Static type checking and compilation

Why static type checking in a dynamic language?

There are various reasons why we wanted to add static type checking to the Groovy language. They are discussed in Guillaume Laforge’s article on InfoQ but the main reason is definitely that we want Groovy to be even easier to use by Java developers. There has been a lot of FUD about Groovy performance and the fact that it is dynamically typed, but in the end, Groovy is among the fastest dynamic languages (if not the fastest) and performance problems are more often coming from I/O operations than from the language itself! Also, we must think about all developers: not everyone builds a Twitter… But for people coming from Java, the lack of static type checking was often a barrier preventing them from using Groovy. There are many things to say about dynamic vs static languages, but for people who are interested in evaluating productivity or ease of debugging, there are some interesting results in this study from Stefan Hanenberg.

Speaking of that, we often see (and we are really irritated by this) the quote from James Strachan, founder of the Groovy language, promoted as a very good reason why not to choose Groovy (and its corrolate, prefer Scala over it): ``I can honestly say if someone had shown me the Programming Scala book by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I’d probably have never created Groovy.'' I don’t know James personaly and I really respect his work and opinion and I joined the project well after but:

James made this statement in 2009 after leaving the project in 2006 even before Groovy 1.0 was released. He is now also working on the Kotlin language. I don’t think that would mean that if Kotlin had existed back in 2003, Odersky wouldn’t have created Scala… or that James would drop Scala anyway. Different languages acheive different goals.
If JPA existed in 2002, do you think Hibernate would have been created? Technology is innovation and languages are not different from any other piece of software in that case!
a lot of improvements have been made on the language since then. I mean, you cannot honestly compare Groovy 1.0 with what Groovy 2.0 is. It’s now a mature, widely adopted language and reading such a quote should not prevent you from testing it by yourself.
this is a personal comment, but definitely, I find Groovy much more readable than Scala. It was easy to make Java developers use Groovy, it’s not so easy to write Scala, and it’s even more difficult to debug other people’s Scala code…
Groovy was never meant to replace Java. All we want is to make the life of developers easier and reintroduce fun in programming.

One interesting thing to know is where I come from. Before working on Groovy for VMware, I was employed by a small French company named Lingway where I introduced the usage of Groovy in several projects. Groovy was used in many contexts, but one thing that is worth noting is that the audience was pretty large: from developers to linguists. Developers used Groovy as a Java scripting tool, sometimes as an ETL processor and linguists used it in the form of a DSL for natural language processing. In the end, we wrote a high performance natural language processor which performed text extraction, named entity recognition, part-of-speech tagging, … just using DSL rules written in Groovy. We had performance issues with Groovy, but in the end, they were all fixed by rewriting classes in Java. Of course, none of the rules were written in Java, so the engine was mostly written in Java, but rules, which are the core of the extraction system, are all written in Groovy, and performance is very good (and it’s using Groovy 1.8 by the way). However, it’s true that it was quite frustrating to rewrite classes in Java when they would have been much shorter and readable if written in Groovy. Moreover, none of those classes were using any of the dynamic features of the language…

Developers were Java developers. And some of them (who may recognize themselves reading this paper) were defintely not fans of dynamic typing. While I spent a lot of time advocating the advantages of Groovy over Java for readability and conciseness, in the end, if you end up using Groovy as a scripting language for Java and that you don’t use the dynamic features of the language, it is very legitimate to ask for static type checking, so that errors are catched early in the development process.

It’s optional!

With Groovy 2, we want people to have choice. Groovy is a very elegant language that removes most of the boilerplate of Java while staying very close to it, grammar-wise, hence very easy to learn. But it was definitely a problem for us to see people choosing other languages because of the lack of a type checker, although we think Groovy is much more readable than a lot of its competitors. Now, you will have the choice. You will have the choice to type check your code, and explicitely choose not to use the dynamic features of the language, and you will even have the choice to statically compile your code (with its own semantics). This adresses the two main issues we’ve talked about here: failing early in the development process thanks to static type checking, and adressing episodic performance issues by statically compiling your code.

Insider

Developing the static type checker was a very challenging task. I spent much less time on static compilation, which is an order of magnitude simpler than type checking. I learnt a lot doing this, and despite all the time spent in implementing that feature, I am pretty sure there are still bugs (in fact, I’m already aware of some), especially when it comes to generics. Oh man. Generics. I mean, I really like using generics and the kind of type inference they offer, but algorithms are damn complex. Type inference with generics is the real challenge. There were two problems that I had to face: first, the internal representation of generics in Groovy is not well suited for type inference, which makes algorithms unnecessary complex (complexity over complexity). Second problem was testing: although you think about a lot of cases, you only develop what you think of. We need testing and we need tests made by others. This is also why there were so many bug fixes in the RC phase: people tend to test the RC versions, not the betas, but this is life!

If you’re interested in the internals, just be aware that the type checker is implemented as an AST transformation (yes!). So basically, it’s a (very) big transformation. The first step was to make it work, then make it right. Now, before making it fast (we didn’t talked about performance of the type checker itself, but it should be almost unnoticeable) I need to simplify code and, of course, fix the bugs!

Invoke Dynamic

A small word on invoke dynamic support in Groovy 2. We must congratulate Jochen Theodorou for that, because he spent a lot of time working on a feature which is hidden for most users. But his work is an important milestone for the future of Groovy, and while we will work on improving the performance of invoke dynamic in the 2.x versions of Groovy (as well as the JVM folks are going to improve performance of invokedynamic in the JVM itself because it will be at the core of the upcoming lambdas in JDK 8), Groovy 3 will probably be a language heavily relying on this feature.

Get it and test it!

Last but important, here are some links:

Static compilation for Groovy poll results

Sun, 29 Jan 2012 00:00:00 +0100

Static compilation or no static compilation?

In my previous blog post, I published a link to the slides from my talk at the Paris Groovy and Grails User Group, where I presented the state of static type checking and compilation for Groovy 2.0. This talk was a unique occasion to retrieve feeback from the user community before we make any final decision about what should (or should not) be included in Groovy 2.0. Especially, we discussed about static compilation (there’s a consensus about static type checking, so I won’t talk about that here).

I decided to add two polls to my presentation and especially, I wanted the poll about why people wanted static compilation to be answered before I talked about the feature. This was important to me because I didn’t want the result to be biased by my talk. After two weeks, 42 persons have answered the polls. Nor me, nor Guillaume Laforge nor Jochen Theodorou answered the poll, once again not to bias the results. Here are the results for question 1:

Again, the question was introduced before I talked about the feature and it’s a multiple choices question (you may choose several answers). What is interesting is that only 3 persons said they didn’t want static compilation, although on the development mailing list, people against static compilation were very vocal. If only 3 people do not need static compilation, this would mean that the others have a good opinion about why they need it, so the following figures are interesting too:

performance and type safety are both at the same level and the highest number of answers (17). While performance was really to be expected, I didn’t expect type safety so high, because basically, if you need type safety, the @TypeChecked annotation is enough (you don’t need static compilation). However, this is only true if you don’t consider the monkey patching issues, where the behaviour of Groovy can be changed at runtime, leading to runtime type errors while the program was statically checked.
I want a better Java comes next, with 12 answers. I expected this one, because when we talked about the static compilation feature on the mailing list, we had an interesting discussion about why people use Groovy, and I was thinking a lot of people already used Groovy as a better Java, although the language wasn’t designed as a replacement but more as a companion to Java.
the next reason coming is that people do not need the dynamic features of Groovy. This is interesting too, because it has to be linked to two different things. First, type safety, because as we already said, you can type check the program to ensure type safety (not needing static compilation) only if you can guarantee that nothing beyond the scope of the compiler (already compiled classes, third party libraries or frameworks) do no modify the behaviour of a program at runtime. Statically compiling the program disables dynamic features, so using static compilation you can ensure type safety. The second point is that if you do not need dynamic features, then you are probably using Groovy as a better Java.
the last option was to comment to tell about why you need static compilation. 4 people chose that option, but didn’t comment, so unfortunately I won’t be able to tell much about that :-). But it’s already interesting to notice that this score is still higher (but not far) than people who don’t need static compilation.

In my talk, after this poll, I started explaining the semantic differences that may appear when you statically compile a program, as compared to dynamic behaviour. I did that because I expected my explanation to make some people change their mind about static compilation. Especially, the most visible difference comes from method dispatch (runtime based in dynamic Groovy or ahead of time in static Groovy). I wanted to tell people that this difference is probably the one which will ultimately make the decision about integrating static compilation in Groovy Core or not. I explained 3 options, which were Java-like method dispatch, dynamic groovy like method dispatch and inference based method dispatch. In the end, I asked people to tell which of the options they preferred:

To summarize once more the pros and cons of each method, I would say that:

Java-like method dispatch has the advantage of well known semantics (though many people are uncapable of telling what precise corner cases would do in Java), but adds a lot of verbosity and disables de facto some of the features we already implemented in the type checker (like flow typing).
Dynamic Groovy like behaviour has the major advantage of keeping the same semantics as dynamic Groovy, but has also major problems. First, performance, because runtime time dispatch is much slower than compile time dispatch. Second, statically checking a dynamic program is always error prone: the compiler may think that method 1 would be called, though at runtime, method 2 would be (because Groovy chooses the best method at runtime according to the actual type arguments), so you would have a difference between what the compiler thinks and what is really done. This means that type safety cannot be ensured anymore…
Inference based dispatch, which is the current implementation, which has the advantage of being close to dynamic groovy while preserving a high level of performance. The major problem with that solution being that we introduce a third semantics which is nor the one from Java, nor the one from dynamic Groovy. The question, in the end, is always: ``is it a problem?''

Looking at the poll results, there are already two very interesting things to notice: first, very few people answered that question (14 persons, barely 4% of all viewers of the presentation). My interpretation is that the question is complex, and that not many people actually understand the problem (which also makes me think some talks about the subject are important ;-)). But this also confirms to me that even in Java, very few people are actually capable of interpreting how method dispatch works. Basically, I think the reasoning is much simpler. If a program doesn’t do what you expect, then you debug it and solve the problem, be it a dispatch problem or not. The second interesting figure, which I did not expect at all, is that nobody chose the Java option. Once more, I interpret this as people are not afraid of having different semantics as long as debugging is possible.

Ultimately, let’s compare the results of the two chosen options. Dynamic based dispatch has 6 answers, while inference based dispatch has 8. Being aware that most of the people who viewed the presentation weren’t at my talk and didn’t get the explanations about why I preferred inference based dispatch, I think the results are very interesting. If you consider that most people want static compilation for performance and type safety which, as I explained, cannot be ensured with dynamic based dispatch, the figures suddenly highlight a strong contradiction. Many people are interested in performance and type safety but still want to keep the dynamic behaviour. My traditional answer for that kind of people is that they should choose invoke dynamic support which will appear in Groovy 2.0 too, as it will guarantee the runtime semantics of a dynamic groovy program while improving performance (though, be warned, type safety cannot be ensured). Of course, static compilation is primarily aimed at people who cannot use invoke dynamic. For that, it seems that the results are in the good direction, because as long as you are aware that you choose a different semantics, then you are not lost. Hopefully, as I said, static compilation is totally optional, so if you choose to use static compilation, you are warned that you will have a slightly different runtime semantics. People who make intensive use of unit tests will probably be very happy with that :-)

Last but not least, the polls are still open. Feel free to answer and comment, as feedback is really important. The most important thing, in the end, is the user community.

Update: below you’ll find the updated figures, as of February 9th, 2012:

Static type checking talk from Paris Groovy/Grails User Group

Sun, 22 Jan 2012 00:00:00 +0100

Static type checking and compilation in Groovy 2.0

This week, I had an interesting 2 days work session at Paris (La Defense) with Guillaume Laforge, Jochen Theodorou and Rémi Forax as a guest star. On Thursday night, I talked about my work on static type checking and experimental static compilation at the Paris Groovy and Grails User Group. Here are the slides:

Some comments

First of all, a large part of the session was spent in explaining what static type checking implies in terms of semantics for a language like Groovy. I tried to explain that by definition, type checking a dynamic language is impossible, so you have to make decisions and drop support for dynamic features in that case. However, this is in most situations not a problem for people looking for static type checking, because what they need is to guarantee the behaviour of a program at runtime and compile time.

The last part of the talk was spent in explaining the most important difference in behaviour between dynamic method dispatch and static method dispatch. This was necessary to introduce the experimental static compiler which should make its way in the next beta of Groovy 2.0. I tried to explain the three different options we have regarding static compilation and method dispatch:

Java-like method dispatch, which has the main interest of being known of most people (at least, people who faced the problem once). This solution however has major drawbacks. In particular, it removes most of the interest of the flow typing mode, and requires extra verbosity which is the opposite of the Groovy philosophy.
Dynamic-Groovy like method dispatch. Typically, this is wanted to avoid a third language'' (not Java, nor Groovy), and this is were you want a statically compiled Groovy program to behave at runtime like dynamic Groovy does. While this sounds interesting, it also has major problems. The first one is performance, which would be awful because we would have to introduce tons of instanceof'' checks at compile time to correctly dispatch methods. More complicated, this ``instanceof cascade'' would generate unreadable bytecode (think of parameter combinations) which would also be a major performance issue for CPUs (which, as Rémi said during the session, really do not like branching). Even worse, this wouldn’t work anymore if you subclass the statically compiled class because the new overloaded methods wouldn’t be known from the compiler. This is why I consider this solution the worse of all, though I understand people who want it. Last but not least, Groovy 2.0 will introduce InvokeDynamic support which should greatly improve performance of dynamic Groovy without loosing its semantics.
The third solution I presented is the one currently implemented in the experimental compiler: inference based method dispatch. This definitely creates a third language in terms of dispatch semantics, but as the developer explicitly adds an annotation, he should be aware of the differences (which only occur when you have overloaded methods). Inference based dispatch also has the advantage of being ``flow typing'' compatible, and removes a lot of verbosity. It has my preference, which explains why this is what is implemented right now.

Our guest star Rémi Forax also suggested an alternative solution, which would in theory maintain the semantics of dynamic Groovy, though we are in a statically compiled world. The idea is that when the compiler detects an overloaded method, it should generate the appropriate instanceof checks and seal the class (to avoid it to be subclassed). But as we discussed that solution the next day, we found some cases where even in that situation, we wouldn’t be able to guarantee dynamic Groovy like behaviour without adding explicit guards on every static method call (in particular, changing the metaclass or detection of category usage). This would also lead in a major performance penalty (Rémi couldn’t avoid telling us that all answers to our problem are in InvokeDynamic, but still, we want to offer something for pre-Java 7 people).

The future

I ended the talk with a small view of what’s still to be done. There are still bugs in the static type checker (especially in loops where an overloaded method result is assigned to one of its parameters) and probably a lot more in the compiler. Eventually, I insisted on the fact that every point of both static type checking and static compilation are to be discussed with the community. While static type checking will most probably be part of Groovy 2.0, it’s still unsure about static compilation, and that’s why we would like you to answer the surveys that are in the presentation.

Toward a new MOP

As this session was a gift for static types lovers, we also wanted to offer something for our dynamic language users. This came with a long discussion about a new MOP (also known as MOP 2.0). The result of the discussion will be published by Guillaume Laforge as a new GEP (Groovy Enhancement Proposal). Once again, community will be involved.

Thanks a lot!

French speaking users might also be interested in my talk at the Nantes JUG which introduced DSLs in Groovy.

CodeNarc plugin for IntelliJ IDEA updated

Wed, 18 Jan 2012 00:00:00 +0100

CodeNarc for IntelliJ IDEA 0.16.1

Eventually, I managed to upgrade the CodeNarc for IntelliJ IDEA plugin to use CodeNarc 0.16.1. It took a while (sorry Hamlet ;)), but I faced an awful jar hell issue.

java.lang.LinkageError

It’s quite interesting indeed, though I am not fully happy with the ``fix''. Let’s see what’s the problem. If any IntelliJ expert is here, I’d be happy to have some help. Basically, for compatibility with newer versions of Groovy, both CodeNarc and GMetrics (a dependency of CodeNarc) embed in their jar three classes which have disappeared from the newest Groovy version (in fact, they were moved in other packages):

org.codehaus.groovy.ast.expr.RegexExpression
org.codehaus.groovy.transform.powerassert.Value
org.codehaus.groovy.transform.powerassert.ValueRecorder

Normally, this shouldn’t be a problem, but when I ran the plugin, it failed loading with this cryptic error, well known from OSGI users:

java.lang.LinkageError: loader constraint violation: loader (instance of com/intellij/ide/plugins/cl/PluginClassLoader) previously initiated loading for a different type with name "org/codehaus/groovy/ast/expr/RegexExpression"
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:465)
 at com.intellij.util.lang.UrlClassLoader._defineClass(UrlClassLoader.java:124)
 at com.intellij.util.lang.UrlClassLoader.defineClass(UrlClassLoader.java:120)
 at com.intellij.util.lang.UrlClassLoader._findClass(UrlClassLoader.java:96)
 at com.intellij.ide.plugins.cl.PluginClassLoader.d(PluginClassLoader.java:102)
 at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:63)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Oh my, a LinkageError. Worse than a classical classpath issue. The problem comes from the fact that IntelliJ internally uses Groovy (where ?) and bundles groovy-all-1.7.3.jar. I had to remove the compatibility classes from the CodeNarc jars in order to have the plugin loaded correctly. I think a better solution would be to be able to use an embedded version of Groovy, but the plugin loaded would always include those classes. By the way, it looks to me that CodeNarc would use the internal Groovy version to run (1.7.3) whereas it would probably be better if we could use the version used by the project…

Anyway, the plugin is upgraded, have fun!

Groovy static type checker: status update

Thu, 3 Nov 2011 00:00:00 +0100

About Groovy

As some of you may already know, I joined SpringSource/VMware a month ago and I am working on the Groovy language.One of my tasks, which will make me busy a little while, consists of developing a static type checker. This project, internally, is known as Grumpy. Why this name? Is it supposed to be public? Indeed; no. It’s an internal name only, because the main goal of this project is to make the compiler grumpy, meaning it will complain where regular Groovy does not. In this post, I will explain what static type checking (STC) consists of, what is already implemented, what’s in progress and eventually, I will need your help. The Groovy language is community driven. While developers like me may propose features, we want to make sure that they are useful to the community and that they are implemented the way YOU expect it to be. This is why, at some points, we will ask you for a decision to be made. Most of times, decisions to be made are not related to technical issues, but rather behaviour issues. I will try to hint you where decisions are to be made, or when they have already been taken. Be sure to give your opinion, either commenting on this post, either sending an e-mail to the user mailing-list.

Groovy Static Type Checking

Update: Now Groovy 2.0.0-beta-1 has been released with the static type checker integrated! You can give it a try, and give us feedback of course.

First of all, you’ll want to take a look at the Groovy Enhancement Proposal page which describes what are the goals of this feature. Right now, it’s still in heavy development, so it’s being developed on its own branch (called grumpy). Basically, the idea is to make Groovy complain at compile time about types, instead of runtime. In fact, as Jochen Theodorou explains on his blog post, the Groovy compiler in ``regular'' mode, cannot make many checks at compile time regarding types, missing properties, missing methods and so on, because the language allows runtime meta-programming. Runtime meta-programming allows you, for example, to dynamically add members to a class, change the methods which are invoked, change the return types, bind external properties, … This means that if a program was to be statically checked, then it would fail, although it would work perfectly at runtime. An example is better than words, let’s take this code:

String.metaClass.foo = { -> 1 }
assert 'Hello, World!'.foo() == 1

If you run it, then you will see that it’s perfectly fine: at runtime, we modify the metaClass of the String class so that it adds a new method on the String class which is known as foo() and returns an int. If this program was to be statically checked, then it would fail at compile time, because the compiler would not find the ``foo'' method on the String class.

So, why should we add STC if it will make the compiler fail where a program is perfectly valid ? In fact, there are many reasons for that. One of them is that Java programmers who discover Groovy are often amazed about the conciseness of the language as compared to Java, and start programming in Groovy like they would in Java, that is to say with types and leveraging the syntax of Groovy. The key here is that many programmers never use the dynamic features of Groovy, but rather use the language as a ``better Java syntax''. If you don’t do runtime metaprogramming, then you can do static type checking. This drives me to another reason for using STC: fail early. It’s always better to discover errors at compile time that at runtime. No need for long talks here: you can easily imagine a program running perfectly fine for long, until it reaches a poorly tested branch where you made a typo:

void method(String message) {
   if (rareCondition) {
 println "Did you spot the error in this ${message.toUppercase()}?"
   }
}

I tend to think that even if you use the dynamic features of Groovy (say, for example, builders), it’s good to be able type check parts of code you know should be statically checked.

The behaviour of the static type checker

Implementation details

Now it’s time to give you more information about the static type checker, that is to say the current implementation. Basically, you activate the static type checker either by annotating a class or a method with @TypeChecked:

import groovy.transform.TypeChecked

@TypeChecked
void method(String message) {
   if (rareCondition) {
 println "Did you spot the error in this ${message.toUppercase()}?"
   }
}

I like code, so let’s see what the compiler has to say about this example, and let’s run it :

2 compilation errors:

[Static type checking] - The variable [rareCondition] is undeclared.
 at line: 5, column: 8

[Static type checking] - Cannot find matching method java.lang.String#toUppercase()
 at line: 6, column: 46

First, it complains about an unbound variable, rareCondition. The compiler cannot statically determine its type, which is not allowed. The second error demonstrates the fact that missing methods are now disallowed and replaced with a nice error message that helps you spot the typo. Those explain the primary goals of the static type checker:

check for missing methods or properties and report error
check for type safety
type inferrence
support for ``GDK methods'', that is to say methods added by Groovy to common classes, aka extension methods
support for Groovy idiomatic constructs like with, each, …

There are also secondary objectives, which are under discussion:

flow typing (see Jochen’s post)
unification types
static compilation

What’s implemented, what’s not implemented

The best way to find out what’s implemented or not is to checkout the grumpy branch from Codehaus Git repository (also available on GitHub and have a look at the unit tests located in src/test/groovy/transform/stc. Another way is to read the GEP-8 page on the wiki which gives you examples plus textual explanations. However, this page is not always up-to-date. Let’s show some nice examples of already implemented stuff:

Groovy-style constructors checks

Checks that arguments provided when using the Groovy-style constructors are valid. Especially, this will throw errors:

Dimension d = [100] // wrong number of arguments
Dimension d = ['100','200'] // wrong argument types

class A {
 int x
 int y
 }
A a = [x:100, y:200, z: 300] // missing property

Type inferrence

def myname = 'Cedric'
"My upper case name is ${myname.toUpperCase()}"
println "My upper case name is ${myname}".toUpperCase()

Type inference with spread operator

def keys = [x:1,y:2,z:3]*.key
def values = [x:'1',y:'2',z:'3']*.value
keys*.toUpperCase()
values*.toUpperCase()

instanceof

If we detect that we are in an ``instanceof'' block, we can avoid explicit casting:

Object o
if (o instanceof String) o.toUpperCase()

Flow typing

As we already said, this is still under discussion, but the prototype static type checker already implements some kind of flow typing. It’s still buggy (I will let you discover how to mislead it ;-)), but already allows some nice things like this:

class A {
 void foo() {}
}
def name = new A()
name.foo() // no need to cast
name = 1
name = '123'
name.toInteger() // no need to cast, and toInteger() is defined by DGM

To give you an idea, if we choose not to do flow typing (give your opinion about that!), then you would have to write this :

class A {
 void foo() {}
}
def name = new A()
((Foo)name).foo() // no need to cast
name = 1
name = '123'
((String)name).toInteger() // no need to cast, and toInteger() is defined by DGM

Casts are necessary because the name variable is assigned multiple times with different incompatible types. Personnaly, I think flow typing is more Groovy style than ``regular'' typing.

Not implemented

Currently, there are two major features that are not implemented:

Support for generics
Support for Groovy idiomatic constructs like ``each''

The first one (generics) is already under heavy development (and is, I must admit, giving me headaches), and there’s not much to say about it (apart that it is really complicated to implement!).

The second point is trickier. We already support the with idiomatic construct. However, we must pay attention on the constructs with uses closures as arguments, and that is the case with lots of Groovy extension methods like each, collect, … I think it is important that the static type checker deals with those before the first official beta including the STC goes out, but it requires decisions from the community.

We need you!

Don’t worry, I won’t ask you to implement those features (though, if you want to, I will never prevent you from doing this :-D), but I do ask for your opinion. Before all, I will illustrate the problem with a simple example:

['I','feel','grumpy'].each {
   println it.toUpperCase()
}

This is currently unsupported, and will throw an error, saying that you cannot call the toUpperCase method on a String. We all agree that it should not complain, and I am in favor of fixing this as soon as possible. However, we must decide how to fix this, in a manner that allows, for example, library writers to be compatible with the type checker.

What’s the problem here ? Basically, the signature of the each method is the following:

private static  Iterator each(Iterator iter, Closure closure)

It is defined in DefaultGroovyMethods, so it’s an extension method, and you could think of it as a method available on the Iterator interface, thus, let’s modify the signature to make it clearer:

public static  Iterator each(Closure closure)

Now, the only thing we know at this point is that the each method takes a closure as an argument, but:

we don’t know what the implementor does with the closure
we don’t know what arguments will be used when calling the closure
Java doesn’t provide enough type information for us to infer the type of the arguments

This means that the block of code representing the closure, in our example, cannot be type checked correctly, because we cannot infer the type of the arguments (here, the implicit it) that will be used when the closure gets called.

For those who know Groovy++, it states that the problem has been solved by adding generics information to the DGM signatures. However, I don’t think this solution is perfect, and I’d rather like to propose various solutions.

The first, obvious solution is to let the developer explicitely specify types. This is a solution, but not really a Groovy one:

['I','feel','grumpy'].each { String it ->
   println it.toUpperCase()
}

The main idea is for the implementor to provide type information through an annotation. Let’s call this annotation @ClosureTypeInfo (if you have better names…). Then I could write this:

void myMethod(@ClosureTypeInfo(argTypes=[String,Set]) Closure strategy) { ... }

This allows the type checker:

to determine that the closure code block requires two arguments of types String and Set
to type check the closure code block using this type information

The problem is that if your class is parameterized, then you can’t use generic types as the arguments of the closure:

class MyClass {
   void myMethod(@ClosureTypeInfo(argTypes=[T]) Closure strategy) { ... } // this is not allowed because of type erasure
}

So, another idea (which is probably the one used by Groovy++, but I could not determine if it is actually how it works) is to use the parameterized types from the declaring class. For example, we would change the each signature to:

public static

When used like this, the type checker will infer that the arguments used for the closure will be the parameterized types of the declaring class. Therefore, if you use each on a List, then it is an each on an Iterator which is parameterized with , so we determine that the arguments which will be used by the each method will be a single String argument.

If you are aware of how Java generics work, you may wonder why we use the generic types from the declaring class, and not those from the method. Indeed, we could have the following method signature:

public static  Iterator each(@ClosureTypeInfo(useGenerics=true) Closure closure) { ... }

Then, the type information from which the argument types would be taken would be those from the method call:

['I','feel','grumpy'].each {
   println it.toUpperCase()
}

Now you have the explanation about why I don’t like this idea:

awkward, not groovy at all, syntax
requires you to add type information although we want it to be inferred
why, in that case, not explicitely specify type information in the closure ?

Indeed, this is more understandable:

['I','feel','grumpy'].each { String it ->
   println it.toUpperCase()
}

So, we must stick with the parameterized types from the class. Now, we must also take care of subclasses and extract parameterized types from superclasses. This is why I started dealing with generics before dealing with closure argument types. For example, this should work:

class StringList extends LinkedList {}
def list = new StringList()
...
list.each {
   println it.toUpperCase()
}

So we have an annotation that allows us to either provide explicit type information (classes used as arguments) or infer types from parameterized types. Note that in the latter case, this works great for methods like each, collect, … but this may not make any sense. For example, imagine that we didn’t use an annotation, and always inferred type from generics. Then imagine the following code:

class Foo {
    T foo(Closure cl) {
        cl.call(1)
    }
    U bar(Closure cl) {
        cl.call([] as Set)
    }
}

def foo = new Foo()
assert foo.foo {
    2*it
} == 2
assert foo.bar {
    it
}.class == HashSet

In this case, the foo() and bar() methods both take a different closure arguments, but the annotation would have expected that both take two arguments (a String and a Set). So, in that case, we would definitely want the generics to be inferred from the method generics. If we do so, we’re back to the situation we described previously, where types are more likely to be written explicitely in the closure block rather than in a generic declaration. I am not sure how we should solve this, so any idea is welcome.

Additionaly, this annotation can also be used to solve another problem: how will the type checker know what kind of delegate is used? The delegation strategy changes the order to which the missing methods or properties are looked up in a closure. Depending on the context, the developer may want to say that the closure passed as an argument of a method must have a specified delegation strategy.

Last thing in this long post, is things we won’t be able to solve (at least, I can’t think of any proper way to do this). For example, regular Groovy allows you to do this:

def strategy = { println it.toUpperCase() }
list.each(strategy)

In static mode, you’ll have, at least, to do this :

def strategy = { String it -> println it.toUpperCase() }
list.each(strategy)

But you’ll have to keep in mind that the compiler cannot, in that case, check that the closure passed as an argument to the each method uses the correct argument types.

Last word

I hope this post has given you the opportunity to better understand the goals of the static type checker in upcoming Groovy. It is also there for you to give your opinion, as we want the behaviour of the type checker to be ``community driven'', that is to say we want to find a behaviour that matches your expectations. Eventually, feel free to contribute, and suggest ways to solve problems like the ones I have highlighted here. Thanks for reading!

JLangDetect 0.3 released with bundled language profiles

Mon, 27 Jun 2011 00:00:00 +0200

It’s been a while since I did not release a new version of JLangDetect, a simple Java library for language identification. I have made several changes to this version, which should make it simpler to integrate and test.

package name changed from com.lingway.jlangdetect to me.champeau.jlangdetect : there have been several questions regarding licensing, and whether JLangDetect was related to Lingway or not. The answer is no : it’s a pet project I’m leading, so to be clearer about that, I decided to rename the main package.
3 modules : jlangdetect, jlangdetect-europarl and jlangdetect-extra :
- jlangdetect provides the basic language identification tooling : learning algorithms and language detection support, but it does not integrate any language profile. This is basically the same level of support as the previous releases.
- jlangdetect-europarl provides an Europarl based language detector which already includes resources for detecting languages for 21 european languages.
- jlangdetect-extra extends the Europarl detector with 4 languages : Russian, Chinese, Japanese and Korean. Those detectors are less robust than the Europarl ones due to the lack of royalty-free resources available, but should be sufficient for most needs.
less memory usage: JLangDetect 0.3 introduces a simple algorithm to reduce the size of each language profile without loosing too much accuracy. In version 0.2, an Europarl based language detector for 11 languages took about 25MB of RAM. Now, you can detect 21 languages with only 4MB of RAM for language profiles.

How to use it ?

The simplest way to use JLangDetect is to use the UberLanguageDetector singleton, available in the jlangdetect-extra module :

import  me.champeau.ld.UberLanguageDetector;
UberLanguageDetector detector = UberLanguageDetector.getInstance();

// ..

String language = detector.detectLang("ceci est un petit texte en français");

Alternatively, if you don’t need to detect russian, chinese, japanese or korean languages, you can use the EuroparlDetector available in the jlangdetect-europarl module. Note that you can still create your own language detector and register custom languages using the core module.

Maven integration

JLangDetect is now available through Maven. To use it, you can add the following repository into your pom.xml file :

  jlangdetect-googlecode
  JLangDetect Maven repository
  https://jlangdetect.googlecode.com/svn/repo

Then use the following dependency :

  me.champeau.jlangdetect
  jlangdetect-extra
  0.3

Use from Groovy

As a last integration example, here is how to use it from Groovy, through a simple script :

@GrabResolver('https://jlangdetect.googlecode.com/svn/repo')
@Grab('me.champeau.jlangdetect:jlangdetect-extra:0.3')
import me.champeau.ld.UberLanguageDetector as ULD

ULD.instance.with {
  assert detectLang('ceci est un petit texte en français') == 'fr'
  assert detectLang('this is a text in english') == 'en'
}

Direct downloads

Customizing Groovy compilation process

Thu, 12 May 2011 00:00:00 +0200

Groovy compilation customizers

Groovy 1.8 introduced the notion of compilation customizers. They are used to tweak the compilation process, in order, for example, to add imports to scripts transparently, apply AST transformations without annotating classes or limiting the language for security. In this post, I will show you how to use them, and how you can write your own customizer.

Prior to Groovy 1.8, you would have to override the GroovyClassLoader and write several utility classes to add compilation units to the compilation process. While this worked, this was not really easy to do. Our first example will show you how easy it is now :

Adding imports transparently

One of the recurring questions on the Groovy mailing lists was how one could add default imports to scripts. Groovy, for example, imports the java.util classes by default, which improves readability, but there was no simple way to add your own. Now, you can use the ImportCustomizer class :

import org.codehaus.groovy.control.customizers.ImportCustomizer

...

def importCustomizer = new ImportCustomizer()
// regular imports
importCustomizer.addImports('java.util.concurrent.atomic.AtomicInteger', 'java.util.concurrent.atomic.AtomicBoolean')
// star imports
importCustomizer.addStarImports('java.util.concurrent')
// static star imports
importCustomizer.addStaticStar('java.lang.Math')

Then all you need to do is to create a compiler configuration where you will register the customizer :

def configuration = new CompilerConfiguration()
configuration.addCompilationCustomizers(importCustomizer)
def shell = new GroovyShell(configuration)
shell.evaluate """
  new AtomicInteger(0) // won't throw ClassNotFoundException
"""

The import customizer provides additional methods to specify, for example, import aliases. Take a look at the javadoc for more details. Now we will take a look at another customizer which is provided by default in Groovy 1.8 and allows you to secure the execution of your scripts.

Securing user scripts thanks to the SecureASTCustomizer

The goal of this customizer is to filter the script to allow or disallow some constructs. For example, one building an arithmetic shell with Groovy would want to disallow the user to import or create classes, and limit the syntax to the basic operators. This can be achieved thanks to the SecureASTCustomizer, which filters out AST nodes and throws a SecurityException whenever a disallowed construct is used.

Let’s take a look at the example from the Javadoc, which defines an arithmetic shell :

final ImportCustomizer imports = new ImportCustomizer().addStaticStars('java.lang.Math') // add static import of java.lang.Math
final SecureASTCustomizer secure = new SecureASTCustomizer()
secure.with {
    closuresAllowed = false
    methodDefinitionAllowed = false

    importsWhitelist = []
    staticImportsWhitelist = []
    staticStarImportsWhitelist = ['java.lang.Math'] // only java.lang.Math is allowed

    tokensWhitelist = [
            PLUS,
            MINUS,
            MULTIPLY,
            DIVIDE,
            MOD,
            POWER,
            PLUS_PLUS,
            MINUS_MINUS,
            COMPARE_EQUAL,
            COMPARE_NOT_EQUAL,
            COMPARE_LESS_THAN,
            COMPARE_LESS_THAN_EQUAL,
            COMPARE_GREATER_THAN,
            COMPARE_GREATER_THAN_EQUAL,
    ].asImmutable()

    constantTypesClassesWhiteList = [
            Integer,
            Float,
            Long,
            Double,
            BigDecimal,
            Integer.TYPE,
            Long.TYPE,
            Float.TYPE,
            Double.TYPE
    ].asImmutable()

    receiversClassesWhiteList = [
            Math,
            Integer,
            Float,
            Double,
            Long,
            BigDecimal
    ].asImmutable()
}

// configuration ends here

CompilerConfiguration config = new CompilerConfiguration()
config.addCompilationCustomizers(imports, secure)
GroovyClassLoader loader = new GroovyClassLoader(this.class.classLoader, config)

First, we allow static usage of java.lang.Math constants thanks to a static import added automatically through the ImportCustomizer we’ve seen previously. Next comes the secure AST customizer configuration itself, which allows us to :

disallow usage of closures and methods
disallow imports by setting an empty import whitelist
disallow static imports by setting an empty static import whitelist
only allow static import of java.lang.Math methods and constants
filter out the allowed tokens of the language by allowing mathematical symbols only (tokensWhitelist)
filter out the types allowed to be used in the shell by explicitely specifying a list of allowed classes (note the usage of Float.TYPE, … for primitive types)
eventually, limit the classes of objects for which method calls are allowed

Doing this, a script like this one :

1+1
---

will run perfectly, while this one :

code,prettyprint

[source]

import org.codehaus.groovy.control.customizers.SecureASTCustomizer
import org.codehaus.groovy.ast.expr.*
import org.codehaus.groovy.control.CompilerConfiguration

class DeclarationExpressionChecker implements SecureASTCustomizer.ExpressionChecker {
    boolean isAuthorized(Expression expression) {
        if (expression instanceof DeclarationExpression) {
            if (expression.leftExpression instanceof VariableExpression) {
                def name = expression.leftExpression.name
                if (name[0]==name[0].toUpperCase()) { return false }
            }
        }

        true
    }
}

def secure = new SecureASTCustomizer()
secure.addExpressionCheckers(new DeclarationExpressionChecker())

def config = new CompilerConfiguration()
config.addCompilationCustomizers(secure)

def shell = new GroovyShell(config)
shell.evaluate """
   def allowed = 1+1
   def Disallowed = 1+1
"""

In this alternative example (suggested by Sven Haiges), we will prevent the user from calling System.exit() :

import org.codehaus.groovy.control.customizers.SecureASTCustomizer
import org.codehaus.groovy.ast.expr.*
import org.codehaus.groovy.control.CompilerConfiguration

class MethodCallExpressionChecker implements SecureASTCustomizer.ExpressionChecker {
    boolean isAuthorized(Expression expression) {
        if (expression instanceof MethodCallExpression) {
            if (expression.objectExpression instanceof ClassExpression) {
                if (expression.objectExpression.type.name==System.name) {
                    if (expression.methodAsString=='exit') return false
                }
            }
        }

        true
    }
}

def secure = new SecureASTCustomizer()
secure.addExpressionCheckers(new MethodCallExpressionChecker())

def config = new CompilerConfiguration()
config.addCompilationCustomizers(secure)

def shell = new GroovyShell(config)
shell.evaluate """
   System.exit(0)
"""

Applying AST transformations transparently with the ASTTransformationCustomizer

The last customizer provided in Groovy 1.8 allows you to transparently add AST transformations to your scripts. This is particularly useful if you think about the new ThreadInterrupt AST transformation, for example, which will not likely be added to scripts by hand. Another advantage of this customizer is to have an alternative to the global AST transformation mechanism, and can also be used to help the development of such global transforms. Usage is fairly easy :

def configuration = new CompilerConfiguration()
configuration.addCompilationCustomizers(new ASTTransformationCustomizer(Log))
def shell = new GroovyShell(configuration)

Here, we define an ASTTransformationCustomizer which will automatically apply the @Log AST Transformations to all classes in the compilation unit. Note that it is interesting in two different ways :

it prevents the user from having to add the @Log transformation to each class he writes
it allows a local AST transformation (@Log) to be applied just as if it was a global one

The second point is important to understand, because it also shows a limit of this customizer : every class in the compilation unit will have the transformation applied, but there is no way to pass AST transformation arguments. For example, here, you cannot change the name of the generated field, while if you used the @Log annotation directly, you could have passed an argument for it. If you need to pass arguments, I encourage you to take a look at the source code of the ASTTransformationCustomizer and write your own customizer.

Transparently adding global AST transformations is not more complicated. Unlike local AST transformations, global AST transformations do not have a corresponding annotation, and do not have arguments, so you must refer to the ASTTransformation class itself :

def configuration = new CompilerConfiguration()
configuration.addCompilationCustomizers(new ASTTransformationCustomizer(MyASTTransformationClass))
def shell = new GroovyShell(configuration)

The difference from the local case is the type of the class passed as the argument of the customizer constructor. In the case of a local AST transformation, you must pass the annotation class, while in the global case, you pass the AST transformation class directly. Using this customizer, you have an alternative option to the META-INF/services/org.codehaus.groovy.transform.ASTTransformation registration process.

Writing your own customizer

In addition to those 3 bundled customizers, you may want to write your own. To to this, you just have to extend the org.codehaus.groovy.control.customizers.CompilationCustomizer abstract class, which is basically a PrimaryClassNodeOperation for which the compiler phase is fixed. If you need to write a customizer, I strongly encourage you to read the ImportCustomizer or ASTTransformation customizer sources, which are quite simple to understand. Eventually, if you have particular needs or see potential improvements to the current implementations, do not hesitate to join the mailing list, or raise JIRA issue.

Inline assembly with Groovy… Evil or awesome?

Mon, 31 Jan 2011 00:00:00 +0100

Inline assembly with Groovy… Evil or awesome ?

I’m borrowing the title of this post from a Tweet by Cédric Beust, the creator of TestNG that followed my post about inlining JVM bytecode instructions in Groovy. I have read many different reactions from this, going from stupid to awesome. Though I started this as an ironic response to the languages micro-benchmarking frenzy, it seemed at the end that there was real interest for this. The question, now, is to determine whether this idea is, indeed, evil, or awesome.

One of the first things to consider is that inlining JVM bytecode instructions is not inlining assembly. The code that we’re inlining is indeed virtual machine instruction sequences. That it really important to understand, because a JVM won’t let you do nasty things. It’s really, when you’re inlining assembly code in a C program, to crash your program (or, worse, the operating system itself). In a JVM, every bytecode sequence gets verified by the JVM before getting executed, and checked at runtime just like regular code.

What does it mean ? It means that you won’t be able to execute bytecode which does things regular Java code wouldn’t be authorized to do. Let’s take an example. One could think that you could, through bytecode instructions, bypass the fact that you have to use reflection to access private fields of a class from another one :

class Private {
    private int value;
}

@groovyx.ast.bytecode.Bytecode
void set(Private b, int x) {
    aload 1
    iload 2
    putfield Private.value >> int
    return
}

def bidon = new Private()
set(bidon, 5)

The previous code, when executed, throws the regular java.lang.IllegalAccessError. It’s not different from what you would expect in pure Java. Now, what would happen if you wrote invalid bytecode sequences ? Do you think you would crash the JVM ? Let’s try :

@groovyx.ast.bytecode.Bytecode
int sum(int x, int y) {
    aload 1
    iload 2
    iadd
    ireturn
}

println sum(3,6)

Can you spot the error ? It’s the first line of the bytecode sequence : aload 1 means push the reference of the first parameter object onto the stack. Here, the first parameter is not an object, but a primitive type (int), so we should have used iload 1. Here’s the output of this program :

Caught: java.lang.VerifyError: (class: Helper, method: sum signature: (II)I) Register 1 contains wrong type

No crash at all, your program is just invalid and buggy, but nothing harmful. From this point of view, the @Bytecode annotation just provides another DSL for Groovy. It’s not different from any other DSL, apart from the fact that it’s translated directly into real JVM bytecode. Are DSLs inherently evil ? I don’t think so.

So where’s the evil ?

Well, I think the problem is that every feature you add to a language will eventually be used. So, what would prevent someone from writing his whole code with bytecode instructions ? How can this be maintainable ? Not everyone speaks bytecode (to be honest, I find this more readable than Clojure, but that’s another debate ;)).

That’s true, no one will prevent anyone from writing bytecode everywhere. But that’s exactly where code reviews and code analysis tools are useful : it’s easy to write crap with any language. It’s easy not to follow conventions, and it’s easy to take wrong decisions : why would you want to write inline bytecode right into Groovy ? The first bad reason would be to improve performance. Most likely, it’s easier to write a Java class that you’ll use into your groovy code. The second bad reason would be because that’s cool. It is, but don’t try this at work. The third one would be to think you are smarter than the Java compiler. Here are, in my opinion, a few reasons why you would want to inline bytecode into Groovy code :

you are a student having programming language theory and compilation courses
you already use bytecode generation in a project, and using this annotation it’s incredibly easy to test bytecode sequences. It makes Groovy the perfect partner for developers who use dynamic bytecode generation (try the IntelliJ IDEA plugin to have a real good combo)
you are a genius Groovy hacker and understand that inlining bytecode could help you save a bunch of metaclass resolutions, and smart enough to understand why writing a Java class wouldn’t help you much about that
you have written a dynamic bytecode generator which uses Groovy instead of direct ASM code to produce the class file
you eventually want to write your own JVM language
playing around with bytecode instructions helps you becoming a better Java programmer (totally subjective)

So, what do you think ? Evil ? Awesome ? Any good reason why you would allow or disallow inlined bytecode ?

Groovy CodeNarc plugin for IntelliJ IDEA is out

Sun, 23 Jan 2011 00:00:00 +0100

I’ve just released a CodeNarc plugin for IntelliJ IDEA. It adds the powerful static Groovy code analysis rules of CodeNarc right into IntelliJ. The inspections are available on the fly, and are configurable thanks to the regular IntelliJ inspections configuration window.

Here are a couple of screenshots :

The plugin is downloadable from the IntelliJ IDEA plugin repository or GitHub.

ASM plugin for IntelliJ and Groovy @Bytecode AST Transform updated

Thu, 20 Jan 2011 00:00:00 +0100

I’m releasing a new version of my ASM plugin for IntelliJ IDEA tonight, as well as a new version of the Groovy @Bytecode AST Transformation. Both are worth an upgrade as they greatly improve your experience :

ASM plugin for IntelliJ IDEA

The new version of the plugin provides several improvements :

Ability to show the differences between two versions of the bytecode
Configuration of output verbosity : skip frames, skip code, skip debug and expand frames
Support for showing bytecode of test classes
A new Groovy tab which displays Groovified code for the @Bytecode AST Transformation
Support for two types of Groovy code : legacy and Groovified

The last two items are directly related to the upgrade of the Groovy AST bytecode transformation, and will make integration of ASM bytecode into Groovy even easier !

Improved syntax for the @Bytecode AST Transformation

The new release of the transformation adds support for a Groovier syntax, mostly contributed by Guillaume Laforge. Basically, the AST Transformation now supports different syntaxes that you can mix :

a legacy syntax which is as close as possible of the output of tools like javap or the ASM plugin for Eclipse/IntelliJ
a Groovified syntax which makes it really easy to embed bytecode into Groovy : just copy and paste !

if you use the IntelliJ IDEA ASM plugin, you wont need to modify the generated bytecode to make it work with the @Bytecode transformation :

First, on the class you want to show the bytecode for, right click on the editor and choose ``Show bytecode outline'' :

The new plugin includes a third tab which shows generated Groovy code. Select the 3rd tab :

You’ll notice that the syntax here is the legacy syntax, as close as possible to the javap output. Now, click on the settings tab and choose Groovified code style :

Right click on the editor to show bytecode again. The Groovified tab will now use the brand new syntax. You can click on the ``Show differences'' button to highlight the differences :

I definitely wanted to keep the two syntaxes as both have great interest. Here are the differences :

Labels
- Legacy syntax : l0
- Groovified syntax : anyLabel:
- The Groovified syntax uses regular Groovy labels, allowing you to use any label, not only the l[0-9]+ syntax
Goto jumps
- Legacy syntax : _goto l15
- Groovified syntax : go to: myLabel
- The underscore is replaced with a Groovier go to: syntax
instanceof checks
- Legacy syntax : _instanceof 'java/lang/String'
- Groovified syntax : instance of: String
- Just like the goto variation, but allows class literals to be specified directly, not only string literals
Fields access
- Legacy syntax : putfield 'com/lingway/test/MaClasse.valeur','I'
- Groovified syntax : putfield MaClasse.valeur >> int
- get/put (static) field instructions support direct class literals. Field type is specified by the right hand side of the expression, after the right shift operator
Methods
- Legacy syntax : invokevirtual 'java/lang/StringBuilder.append','(Ljava/lang/String;)Ljava/lang/StringBuilder;'
- Groovified syntax : invokevirtual StringBuilder.append(String) >> StringBuilder
- method instructions support direct class literals. Field type is specified by the right hand side of the expression, after the right shift operator
`New' operator
- Legacy syntax : _new 'java/lang/String'
- Groovified syntax : newobject String
- A newobject keyword, supporting the class directly, not as a string literal
Arrays
- Legacy syntax : newarray T_INT
- Groovified syntax : newarray int
- Supports class literals as well as primitive types
Multidimensional arrays
- Legacy syntax : multianewarray '[[Ljava/lang/Object;',2
- Groovified syntax : multianewarray Object[][],2
- Supports classes directly, as well as primitive types
Type casting
- Legacy syntax : checkcast 'java/lang/String'
- Groovified syntax : checkcast String
- Supports class literals
Exceptions
- Legacy syntax : trycatchblock l0,l1,l2,'java/lang/Throwable'
- Groovified syntax : trycatchblock l0,l1,l2,Throwable
- Supports class literals

Downloads

Plugin for IDEA

The plugin for IntelliJ IDEA can be downloaded directly from the plugin manager or from this link.

Groovy @Bytecode AST Transformation

Download the new release from GitHub : groovy-bytecode-ast/groovy-bytecode-ast-0.3.jar

Groovy @Bytecode AST transformation released

Sun, 16 Jan 2011 00:00:00 +0100

Following my post about Fibonacci performance in Groovy, Guillaume Laforge gently asked me to continue my work, so I’m releasing the first version of a Groovy AST transformation which allows you to write a method body directly as bytecode. There are several usages for this, but I think the most important is educational : it’s one of the simplest (if not the simplest) way of testing JVM bytecode.

Here’s an example of what you may write :

@groovyx.ast.bytecode.Bytecode
int fib(int i) {
    iload 1
    iconst_2
    if_icmpge l1
    iload 1
    _goto l2
   l1
    aload 0
    iload 1
    iconst_2
    isub
    invokevirtual '.fib', '(I)I'
    aload 0
    iload 1
    iconst_1
    isub
    invokevirtual '.fib' ,'(I)I'
    iadd
   l2
    ireturn
}
println fib(40)

The documentation about how to write bytecode and the supported features are on the project homepage at GitHub. The source code is licensed under the Apache License version 2. If you find any bug or have feature requests, please fill in the bug tracker.

Downloads and sources

You must use the groovy-all version of Groovy for this AST transform to work. Then, download the following jar :

groovy-bytecode-ast-0.1.0.jar

Sources are built with Gradle and tests made with the Spock framework. Unit tests are also a great source of documentation, do hot hesitate to check them out.

Yes, Fibonacci in Groovy can be as fast as Java !

Tue, 11 Jan 2011 00:00:00 +0100

After reading this, take a look at the complete @Bytecode AST transformation implementation here

Yes, the famous Fibonacci test can be as fast in Groovy that it is in Java. In fact, Vaclav Pech showed us that it could even be faster. Here, I’ll show you how you can acheive the same level of performance as Java without changing the algorithm. I have buzzed on Twitter saying this was a pure Groovy solution. In fact, I’m cheating a little, as one would consider this is not really pure Groovy, but indeed, you can do this only with .groovy files.

It all started with one quote

Around one discussion about the integration of Groovy++ into Groovy, which always end up with benchmarks comparisons, Jochen Theodorou said :

Well, with the AST transforms you could also generate java bytecode directly. Many things are possible with Groovy.

When Jochen speaks, everyone listens. I did so. Was he really saying that I could generate bytecode through AST transformations ? I’ve read this phrase carefully multiple times, then digged into the Groovy source code, and indeed, there was a BytecodeSequence AST Node. Wow. Groovy never stops surprising me. Ok, so let’s start a proof of concept, and let’s do it with the king of the benchmarks (my appreciation), the Fibonacci benchmark.

Java vs Groovy

This benchmark is really interesting, because it shows us where Groovy is really bad against Java. That’s not important to me as I reckon this is not what Groovy is made for, but well, you know, benchmarks are all around, and people get nasty when they start telling your-favorite-language-is-crap-because-it-s-too-slow. I’m going to show you what I can do with Groovy, guys ! So here’s a Java implementation :

public int fib(int i) {
   return i < 2 ? 1 : fib(i - 2) + fib(i - 1);
}

And now, it’s Groovy counterpart :

int fib(int i) {
   i < 2 ? 1 : fib(i - 2) + fib(i - 1);
}

Definitely similar, but Groovy 1.7.6 runs this 30x slower than Java…

Groovy’s revenge, part 1 : ASM

The first episode for Groovy’s revenge consists of writing an ASM plugin for IntelliJ IDEA, because, you know, it was such a pain to work with Eclipse. That’s done.

Groovy’s revenge, part 2 : AST Transformations

The second episode, and most interesting, consists of writing an AST Transformation that will allow you to write bytecode right into Groovy. I did that, and here’s what my final script looks like :

@ast.Bytecode
int fib(int i) {
 l0
    iload 1
    iconst_2
    if_icmpge l1
    iconst_1
    _goto l2
 l1
    frame SAME
    aload 0
    iload 1
    iconst_2
    isub
    invokevirtual '.fib','(I)I'
    aload 0
    iload 1
    iconst_1
    isub
    invokevirtual '.fib', '(I)I'
    iadd
 l2
   frame same1,'I'
    ireturn
}

int groovyFib(int i) { i<2?1:groovyFib(i-2)+groovyFib(i-1)}
println "Pure Groovy"
long sd = System.currentTimeMillis()
println groovyFib(40)
println "Computed in ${(System.currentTimeMillis()-sd)}ms"

println "Bytecode Groovy"
sd = System.currentTimeMillis()
println fib(40)
println "Computed in ${(System.currentTimeMillis()-sd)}ms"

And here’s the output :

Pure Groovy
165580141
Computed in 18465ms
Bytecode Groovy
165580141
Computed in 576ms

Annotating a method with @Bytecode allows you to write it’s body in JVM pseudo-bytecode (you’ll notice some slight differences), but it’s almost exactly the same as what the ASM plugin will show when you display the bytecode of a method.

Interested in the AST Transformation code ? Here it is. Note that it is in very early stages, and I wrote this as a proof-of-concept. I’m unsure that there’s really a need for such a powerful tool in Groovy, that’s mostly fun for me ! However, it could be useful for self-generating code, you know, all the stuff about robots that write their own code and eventually generate giant networks of machines which destroys humanity… Maybe that could be useful for dynamic recompilation too, like what’s done in many emulators. However, if you think that could be interesting to have a complete implementation, let me know and I could push the code so that it comes as a Groovy module.

The code. It’s surprisingly easy. However, you’ll have to use the groovy-all jar to get this work, because the bytecode AST transformations uses the embedded ASM library which is relocated at build time. The first and easy step is the AST annotation itself :

package ast

import java.lang.annotation.ElementType
import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target
import org.codehaus.groovy.transform.GroovyASTTransformationClass

@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.METHOD])
@GroovyASTTransformationClass(["ast.BytecodeASTTransformation"])
public @interface Bytecode {

}

Then the implementation of the transformation :

package ast

import groovyjarjarasm.asm.Label
import groovyjarjarasm.asm.MethodVisitor
import groovyjarjarasm.asm.Opcodes
import org.codehaus.groovy.ast.ASTNode
import org.codehaus.groovy.ast.expr.ArgumentListExpression
import org.codehaus.groovy.ast.expr.MethodCallExpression
import org.codehaus.groovy.ast.expr.VariableExpression
import org.codehaus.groovy.ast.stmt.ExpressionStatement
import org.codehaus.groovy.classgen.BytecodeInstruction
import org.codehaus.groovy.classgen.BytecodeSequence
import org.codehaus.groovy.control.CompilePhase
import org.codehaus.groovy.control.SourceUnit
import org.codehaus.groovy.transform.ASTTransformation
import org.codehaus.groovy.transform.GroovyASTTransformation

@GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS)
class BytecodeASTTransformation implements ASTTransformation, Opcodes {
 void visit(ASTNode[] nodes, SourceUnit source) {
  def meth = nodes[1]
  def instructions = meth.code.statements
  meth.code = new BytecodeSequence(new BytecodeInstruction() {
   @Override
   void visit(MethodVisitor mv) {
    def labels = [:]
    // perform first visit to collect labels
    instructions.each { ExpressionStatement stmt ->
     def expression = stmt.expression
     if (expression instanceof VariableExpression) {
      def text = expression.text
      if (text ==~ /l[0-9]+/) {
       labels.put(text, new Label())
      }
     }
    }
    instructions.each { ExpressionStatement stmt ->
     def expression = stmt.expression
     if (expression instanceof VariableExpression) {
      def text = expression.text
      if (text ==~ /l[0-9]+/) {
       mv.visitLabel(labels[text])
      } else if (text =~ /[aild]const|[aild]sub|[aild]add|[aild]return/) {
       mv.visitInsn(Opcodes."${text.toUpperCase()}")
      } else {
       throw new IllegalArgumentException("Bytecode operation unsupported : "+text);
      }
     } else if (expression instanceof MethodCallExpression) {
      if (expression.objectExpression instanceof VariableExpression && expression.arguments instanceof ArgumentListExpression) {
       if (expression.objectExpression.text=="this") {
        def opcode = expression.methodAsString.toUpperCase()
        ArgumentListExpression args = expression.arguments
        switch (opcode) {
         case '_GOTO':
          mv.visitJumpInsn(GOTO, labels[args.expressions[0].text])
          break;
         case 'IF_ICMPGE':
         case 'IF_ICMPLE':
         case 'IF_ICMPNE':
         case 'IF_ICMPLT':
         case 'IF_ICMPGT':
          mv.visitJumpInsn(Opcodes."${opcode}", labels[args.expressions[0].text])
          break;
         case 'ALOAD':
         case 'ILOAD':
          mv.visitVarInsn(Opcodes."${opcode}", args.expressions[0].text as int)
          break;
         case 'INVOKEVIRTUAL':
          def (clazz,call) = args.expressions[0].text.split(/\./)
          def signature = args.expressions[1].text
          if (!clazz) clazz = meth.declaringClass.name
          mv.visitMethodInsn(INVOKEVIRTUAL, clazz, call, signature)
          break;
         case 'FRAME':
          def frameId = args.expressions[0].text.toUpperCase()
          if ('SAME'==frameId) {
           mv.visitFrame(Opcodes.F_SAME, 0, null, 0, null);
           break;
          } else if ('SAME1'==frameId) {
           if (args.expressions[1].text=='I') {
            mv.visitFrame(Opcodes.F_SAME1, 0, null, 1, [Opcodes.INTEGER] as Object[]);
            break;
           }
          }
         default:
          throw new IllegalArgumentException("Bytecode operation unsupported : "+expression);
        }
       } else {
        throw new IllegalArgumentException("Bytecode operation unsupported : "+expression);
       }
      } else {
       throw new IllegalArgumentException("Bytecode operation unsupported : "+expression);
      }
     } else {
      throw new IllegalArgumentException("Bytecode operation unsupported : "+expression);
     }
    }
   }
  })
 }

}

The code shows that there are many bytecode instructions that I did not manage. This is because I just wanted a proof-of-concept, so I mostly dealt with the instructions required to make the Fibonacci test run. However, thanks to Groovy dynamic nature, the code is rather compact, and consists of three steps :

Visit the method AST node to replace bytecode pseudo-instructions with a BytecodeSequence AST Node
which requires visiting the code block itself to collect the labels
then generate the visit instructions

Conclusion

My conclusion is rather simple : Groovy is amazing. Even when you think it’s beaten by another JVM language, you’ll always find room for improvement. This one looks like the most absolute solution : the DIY way, which reminds me when I was young and that I wrote assembler code on an Amstrad CPC 6128 for demos. This was fun, and I’m having fun again ! For you, if you find this code useful, then it’s a world of verify errors that opens to you. Good luck !

Footnote : for those who don’t get the irony, this is obvioulsy not a solution for making Groovy faster. It’s a nonsense to take a benchmark like this and say ``Groovy is slow''. I’ve blogged many times about Groovy performance and showed this is rarely an issue. No, this point is about AST transformations, and how you could use bytecode to extend the language and implement missing features at the lower level.

ASM Bytecode Outline for IntelliJ IDEA

Sun, 9 Jan 2011 00:00:00 +0100

An ASM bytecode outline plugin for IntelliJ IDEA

Today, I’ve just submitted a new plugin which integrates ASM bytecode views into IntelliJ IDEA : it’s a bytecode outline plugin similar to the one provided for Eclipse. It was really a pain for me to work with Eclipse, so I eventually decided to write my own plugin for my favorite IDE. It’s in early stages, but it just suits my needs right now.

I have released it under an Apache 2 license, so there’s no problem for you to contribute if you want to improve it. By the way, this is also a test for me, since source code is available at GitHut and that I’m not really familiar with Git concepts.

Until the release is available from Jetbrain’s plugin manager, you may download the first release here.

Update

I’ve uploaded a new version (0.2) which solves a problem under Windows, and doesn’t force compile when not necessary.

Here are a couple of screenshots :

Usage

It’s fairly easy : just right click on the editor, and choose ``Bytecode Outline''. Your class will be compiled, and the bytecode will be shown in the tool window on the right. Note that I plan not to compile the class if the binary is up to date, but I’m facing a strange bug in IntelliJ right now when I do so.

Getting TestNG, Spock and Cobertura to work together with Maven

Tue, 28 Dec 2010 00:00:00 +0100

Cobertura, TestNG and Maven play well together. However, I had problems when I started migrating some tests to the Spock framework. Long story short, Spock uses JUnit internally, not TestNG, and the surefire plugin only triggers tests for one of those frameworks at a time. Therefore, if you mix TestNG with Spock, you’ll run with problems where your Spock tests won’t be started automatically by Maven, and eventually, Cobertura will never generate test coverage for the corresponding classes.

I have found different proposals for fixing this problem, but only one worked for me, so here’s my working pom.xml :

...

                1.7.6
  0.5-groovy-1.7
  1.2

...



   org.testng
   testng
   5.8
   test
   jdk15


   org.codehaus.groovy
   groovy-all
   ${groovy.version}


     junit
     junit


     org.apache.ant
     ant


     org.apache.ant
     ant-launcher


     jline
     jline





   org.spockframework
   spock-core
   ${spock.version}
   test



   org.codehaus.gmaven.runtime
   gmaven-runtime-1.7
   ${gmaven.version}


     org.codehaus.groovy
     groovy-all



...





    org.codehaus.gmaven
    gmaven-plugin
    ${gmaven.version}



       1.7


       generateStubs
       compile
       generateTestStubs
       testCompile






    org.apache.maven.plugins
    maven-surefire-plugin
    2.7.1


      test

       test


       none:none



      test-testng
      test

       test


       none:none





    org.spockframework
    spock-maven
    ${spock.version}



       find-specs





    org.codehaus.mojo
    cobertura-maven-plugin
    2.4


      cobertura-test
      package

       cobertura



        false




      cobertura-clean
      clean

       clean

The trick is to add two executions of the surefire plugin, as indicated here, but forcing the plugin version to 2.7.1. Also, adding two dependencies to the surefire plugin, as suggested by Kristian Rosenvold didn’t work for me. My pom.xml file also includes the necessary configuration for GMaven to work with Groovy 1.7 instead of 1.6 by default.

Groovy Xmas gift : converting a number to its french writing

Thu, 23 Dec 2010 00:00:00 +0100

If you ever studied French, you must have struggled with its writing of numbers. There are various exceptions which even for natives are not easy to deal with. Fortunately, here’s a Groovy script that will help you. Okay, I did this because I remembered about a similar exercise I had several years ago in Caml I solved using pattern matching. I was thinking about a solution in Groovy. Here’s the result :

/**
 * Convertit un nombre en lettres.
 * Les règles d'écriture sont celles décrites sur https://www.miakinen.net/vrac/nombres
 *
 * @author Cédric Champeau (https://twitter.com/CedricChampeau)
 *
 */
def digits = ['zéro','un','deux','trois','quatre','cinq','six','sept','huit','neuf']
def specials = ['dix','onze','douze','treize', 'quatorze', 'quinze','seize']
def tens=[20:'vingt',30:'trente',40:'quarante',50:'cinquante',60:'soixante',80:'quatre-vingt',81:'quatre-vingt-un']
def asLetters
asLetters = { number ->
   number<0?"moins ${asLetters(-number)}":
   number<10?digits[number]:
   number<17?specials[number-10]:
   number<20?"dix-${asLetters(number-10)}":
   number==80?'quatre-vingts':
   number<100?tens[number]?:(
       number>69 && number<80?"soixante${number==71?' et ':'-'}${asLetters(number-60)}":
       number>89?"quatre-vingt-${asLetters(number-80)}":
       (number%10==1)?"${asLetters(number-1)}${number<70?' et un':asLetters(number-70)}":"${(number-number%10)==80?'quatre-vingt':asLetters(number-number%10)}-${asLetters(number%10)}"):
   number==100?'cent':
   number<1000?"${((int)(number/100))>1?(asLetters((int)number/100)+' '):''}cent${number%100==0?'s':' '+asLetters(number%100)}":
   number==1000?'mille':
   number<1000000 && ((number/1000)>1 && ((int)number/1000)%100==0)?"${asLetters((int)number/100000)} cent mille${number%1000==0?'':' '+asLetters(number%1000)}":
   number<1000000?"${((int)(number/1000))>1?(asLetters((int)number/1000)+' '):''}mille${number%1000==0?'':' '+asLetters(number%1000)}":
   number<1000000000?"${number/1000000 as int==80?('quatre-vingt'+(number%1000000>0?'':'s')):asLetters((int)(number/1000000))} million${((int)(number/1000000))>1?'s':''}${number%1000000>0?' '+asLetters((int)number%1000000):''}":
   "${number/1000000000 as int==80?('quatre-vingt'+(number%1000000000>0?'':'s')):asLetters((int)(number/1000000000))} milliard${((int)(number/1000000000))>1?'s':''}${number%1000000000>0?' '+asLetters((int)number%1000000000):''}"
}
(1..100).each { println "$it : ${asLetters(it)}" }
[200,210,300,1100,1200,1999,1342,
1001,1011,2000,2011,-2100,2101,2222,2999,12000,12101,54122,
80000,122000,999999,154322,1154322,13154399,177412666451
,100000000,10000000001,200001,80000000,80000001,80000000000,80000000001,300000,
82015, 200000,200000000].each { println "$it : ${asLetters(it)}"}

Edit: Fixed 80'' written quatre-vingt''. Sorry about the inline indenter, but copy/paste it and you’ll have a better rendering or check this

The next step would be to make it even more groovy. But it showed me Groovy misses some interesting features of Caml : pattern matching is interesting as it deals with multiple left hand side patterns, where closures only allow one signature with a single parameter. Here, I used indenting and over-used the ternary operator to emulate this behaviour.

Imagine you could write something like this :

def asLetters = { (..<0) -> "moins ${asLetters(-it}"
 | (0..10) -> digits[it]
 | it<17 -> specials[number-10]
 | ...

What do you think ? Would it be more readable ?

Upcoming Groovy goodness : automatic thread interruption

Tue, 30 Nov 2010 00:00:00 +0100

Introduction

Using Groovy can really be addictive. It deals with so many different problems that you’ll eventually end up using it everywhere in your application. One of the situations that I think is widely used is embedding Groovy for your business logic. For example, embedding a Groovy DSL which allows users to customize their application or even run scripts on your platform. However, when doing this, you expose your infrastructure dangerously : what if a user accidently creates an infinite loop ? What if another creates a memory leak ? Would it be reasonable if your whole application goes down because a customer just ran one poor script ?

There are common solutions for that, but most are not suitable. The real solution would be to isolate the script from the rest of the application by spawning it in its own process. However, this may be complicated to do if you need to share objects from the original VM.

As I had this problem multiple times in the past, I proposed a patch to Groovy 1.8 introducing a new AST Transformation which was just, at the beginning, an elegant solution for interrupting loops and closures when the running thread has been interrupted. Eventually, thanks to the Groovy folks (and a special mention to Hamlet D’Arcy), we ended up with three separate annotations that I’ll describe here.

A basic example

Here’s the most basic example : how would you manage this ?

while (true) {
 // eat cpu
}

You can test it in the Groovy console (groovyConsole command), and try to click on the script interruption button. Most likely you won’t be able to click on the script interruption button because the script doesn’t even make a call to Thread.yield() which does not let a chance to the UI thread to catch up. If you could, Groovy would have sent an interruption request to the script thread. However, even in that case, the script thread would never stop because no one ever checks for interruption. Here’s where the first annotation comes up.

New AST Transformations

Note that the following is based on an early implementation and subject to changes. Those annotations are available since the Groovy 1.8.0-beta-3 release.

@ThreadInterrupt

This is the most basic transformation we can apply. The script above becomes :

@ThreadInterrupt
import groovy.transform.ThreadInterrupt

while (true) {
// eat cpu
}

If you run it in the Groovy 1.8.0-beta3 console, you will not notice any change, apart from the fact that you can click on the interrupt button now. If you do so, you’ll see an exception thrown :

java.lang.InterruptedException: Execution Interrupted
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

The magic here is that our transformation automatically added a thread interruption check in the loop block. Thus, the annotated code is equivalent to the following :

while (true) {
  if (Thread.currentThread().isInterrupted()) throw new InterruptedException("Execution Interrupted");
// eat cpu
}

That’s all ! Adding this annotation ensures that for, while,do loops and closures will have an interruption check added. It will also add a check on method start. This annotation also applies on classes. The behaviour of the transformation can also be tuned for your needs :

checkOnMethodStart annotation parameter (defaults to true) adds an interrupt check as the first statement of a method body
applyToAllClasses annotation parameter (default to true) will apply the transformation to every class in your script (or compilation unit if not in a script), therefore allowing to interrupt more complex method call pathes.

You may wonder what use is this transform if the user has to explicitely add it to scripts (s)he writes. However, the good thing is that you can do it automatically, because Groovy scripts accept import statements anywhere. So basically, you could take the user code, append the annotated import, and you’d see the expected behaviour.

@TimedInterrupt

The second AST Transformation to be added allows interrupting a script after a maximum execution time. It is interesting when you want to automatically timeout scripts. For example :

@TimedInterrupt(10)
import groovy.transform.TimedInterrupt

while (true) {
// eat cpu
}

In this example, the script will automatically timeout after 10 seconds, throwning a TimeoutException. Apart from the two previously described annotation parameters, this annotation accepts two additional ones :

value (mandatory, defaults to 1) : the amount of time unit the script is allowed to consume
unit (defaults to TimeUnit.SECONDS) : the unit used to describe the timeout

@ConditionalInterrupt

This one is another powerful variant of automatic thread interruption annotation. It allows the user to provide a custom condition for interruption. For example, one would like to check that user credentials are valid, while other would like to ensure that there are enough file descriptors left to allow script execution. There’s a wide variety of problems that can be solved thanks to this one. In my examples, I’ll focus on simple test cases to show off the advantages of this annotation.

@ConditionalInterrupt({counter++>1})
import groovy.transform.ConditionalInterrupt

counter=0
def scriptMethod() {
     4.times {
         println 'executing script method...'
     }
}

scriptMethod()

In this example, the @ConditionalInterrupt annotation accepts a closure as annotation value. This amazing features which is already used by GContracts allows us to annotate a class (or import statement) with a custom condition. Here, we implement a custom interruption policy which decides that an InterruptedException should be thrown whenever counter1. The counter variable references the one defined in the script. This is the main difficulty you’ll have to face with this annotation : variable scoping. The semantics is not different from the one in regular Groovy scripts, but may be hard to understand. Indeed, replacing the line :

counter=0

with

def counter=0

would cause the compilation to fail because the counter variable would not be in the scope. Now the hard question is given the annotated script upper, how many times would you expect the `executing script method…' message to be displayed ? If your answer is 2, not bad, but you missed one point. If your answer is 1, either you are lucky, or you are brilliant. If your answer is anything else, you’ll be interested in the explanation.

How is this script updated ? To check it out, you can use the AST Browser found in the Groovy console. It will show you how your script gets transformed, and help you tune your custom conditions. Here, the generated script looks like this (stripped for clarity) :

public class script1291150384631 extends groovy.lang.Script {

    public java.lang.Object run() {
        counter = 0
        this.scriptMethod()
    }

    public java.lang.Object scriptMethod() {
        if (this.conditionalTransform$condition()) {
            throw new java.lang.InterruptedException('Execution Interrupted')
        }
        4.times({
            if (this.conditionalTransform$condition()) {
                throw new java.lang.InterruptedException('Execution Interrupted')
            }
            this.println('executing script method...')
        })
    }

    protected java.lang.Boolean conditionalTransform$condition() {
        ( counter )++ > 1
    }

}

Got it ? If your answer was two, you probably missed the fact that scriptMethod first statement will be, by default (understand unless you set the checkOnMethodStart flag to false), an interruption check…

Can we do anything more complex with it ? Indeed, you can. For example, I can imagine, in a web application, a custom condition that would use a helper class which looks up a thread local to extract the current request context and check user credentials. Here’s another example which demonstrates a very simple case where the condition is shared by multiple classes :

import groovy.transform.ConditionalInterrupt

class Helper {
  static int i=0
  static def shouldInterrupt() { i++>0 }
}

@ConditionalInterrupt({ Helper.shouldInterrupt() })
class MyClass {
   def myMethod() { }
}

class MyOtherClass {
   def myOtherMethod() { new MyClass().myMethod() }
}

new MyOtherClass().myOtherMethod()

Here, a Helper class stores a static counter. Every method call will increase the counter, independently of the class from which a method is called. A few tricks to understand though :

we are using static members for the helper class (both counter and method) because this annotation does not add interrupt checks on static members. If it had, then the Helper class itself would have been modified, and you would have ended with a stack overflow error.
it is not necessary to add the annotation to the second class as by default, the applyToAllClasses flag is on

A few gotchas

There is something more you need to be aware when using this annotation : first, it’s obvious (yet we talk about it) that this won’t deal with pure Java classes that the script may use. If an infinite loop occurs in a Java library used by the script, it won’t get interrupted. Second, you must take care of open resources, especially files, sockets, result sets, … that would not be closed if the script didn’t check for exceptions. Those magic annotations won’t do it for you.

Conclusion

I really hope you’ll enjoy those new features, as I think those are quite mandatory in production environments where Groovy scripts are not totally under control (this is not limited to user errors, but resources exhausted is a common case too).

Using custom Groovy metaclasses to boost performance

Wed, 10 Nov 2010 00:00:00 +0100

Groovy is a dynamic language by nature. This means, in Groovy, that method dispatches are controlled by metaclasses. This is really powerful, and allows nice tricks like the method missing'' or property missing'' calls that are used in many DSLs. The metaclass dictates what behaviour is expected whenever a call to a property or a method is made.

The ``problem'' with this is that method execution paths may be really long. Though Groovy uses call site caching to improve performance, the chain of invocation to call a method in Groovy is much, much longer than in plain Java. The only reason why this chain is long is to allow that dynamic behaviour.

Now, when performance is critical and that you still want to use Groovy as a DSL, there are not so many things you can do to improve performance :

coding critical parts in Java and using mixed compilation to call Java classes
use profilers to hunt down slow parts

Today, I’ll add a nice one that I’ve just experienced : using custom metaclasses.

The explanation

To simplify, almost every method call you make in Groovy will end up beeing dispatched by the class called MetaClassImpl. This class manages a wide variety of method calls or property access. It deals with closures as well as ``regular objects''. We’ll take, as an example, the body of the MetaClassImpl#invokeMethod(Class sender, Object object, String methodName, Object[] originalArguments, boolean isCallToSuper, boolean fromInsideClass) method :

checkInitalised();
        if (object == null) {
            throw new NullPointerException("Cannot invoke method: " + methodName + " on null object");
        }

        final Object[] arguments = originalArguments == null ? EMPTY_ARGUMENTS : originalArguments;
//        final Class[] argClasses = MetaClassHelper.convertToTypeArray(arguments);
//
//        unwrap(arguments);

        MetaMethod method = getMethodWithCaching(sender, methodName, arguments, isCallToSuper);
        MetaClassHelper.unwrap(arguments);

        if (method == null)
            method = tryListParamMetaMethod(sender, methodName, isCallToSuper, arguments);

        final boolean isClosure = object instanceof Closure;
        if (isClosure) {
            final Closure closure = (Closure) object;

            final Object owner = closure.getOwner();

            if (CLOSURE_CALL_METHOD.equals(methodName) || CLOSURE_DO_CALL_METHOD.equals(methodName)) {
                final Class objectClass = object.getClass();
                if (objectClass == MethodClosure.class) {
                    final MethodClosure mc = (MethodClosure) object;
                    methodName = mc.getMethod();
                    final Class ownerClass = owner instanceof Class ? (Class) owner : owner.getClass();
                    final MetaClass ownerMetaClass = registry.getMetaClass(ownerClass);
                    return ownerMetaClass.invokeMethod(ownerClass, owner, methodName, arguments, false, false);
                } else if (objectClass == CurriedClosure.class) {
                    final CurriedClosure cc = (CurriedClosure) object;
                    // change the arguments for an uncurried call
                    final Object[] curriedArguments = cc.getUncurriedArguments(arguments);
                    final Class ownerClass = owner instanceof Class ? (Class) owner : owner.getClass();
                    final MetaClass ownerMetaClass = registry.getMetaClass(ownerClass);
                    return ownerMetaClass.invokeMethod(owner, methodName, curriedArguments);
                }
                if (method==null) invokeMissingMethod(object,methodName,arguments);
            } else if (CLOSURE_CURRY_METHOD.equals(methodName)) {
                return closure.curry(arguments);
            }

            final Object delegate = closure.getDelegate();
            final boolean isClosureNotOwner = owner != closure;
            final int resolveStrategy = closure.getResolveStrategy();

            final Class[] argClasses = MetaClassHelper.convertToTypeArray(arguments);

            switch (resolveStrategy) {
                case Closure.TO_SELF:
                    method = closure.getMetaClass().pickMethod(methodName, argClasses);
                    if (method != null) return method.invoke(closure, arguments);
                    break;
                case Closure.DELEGATE_ONLY:
                    if (method == null && delegate != closure && delegate != null) {
                        MetaClass delegateMetaClass = lookupObjectMetaClass(delegate);
                        method = delegateMetaClass.pickMethod(methodName, argClasses);
                        if (method != null)
                            return delegateMetaClass.invokeMethod(delegate, methodName, originalArguments);
                        else if (delegate != closure && (delegate instanceof GroovyObject)) {
                            return invokeMethodOnGroovyObject(methodName, originalArguments, delegate);
                        }
                    }
                    break;
                case Closure.OWNER_ONLY:
                    if (method == null && owner != closure) {
                        MetaClass ownerMetaClass = lookupObjectMetaClass(owner);
                        return ownerMetaClass.invokeMethod(owner, methodName, originalArguments);
                    }
                    break;
                case Closure.DELEGATE_FIRST:
                    if (method == null && delegate != closure && delegate != null) {
                        MetaClass delegateMetaClass = lookupObjectMetaClass(delegate);
                        method = delegateMetaClass.pickMethod(methodName, argClasses);
                        if (method != null)
                            return delegateMetaClass.invokeMethod(delegate, methodName, originalArguments);
                    }
                    if (method == null && owner != closure) {
                        MetaClass ownerMetaClass = lookupObjectMetaClass(owner);
                        method = ownerMetaClass.pickMethod(methodName, argClasses);
                        if (method != null) return ownerMetaClass.invokeMethod(owner, methodName, originalArguments);
                    }
                    if (method == null && resolveStrategy != Closure.TO_SELF) {
                        // still no methods found, test if delegate or owner are GroovyObjects
                        // and invoke the method on them if so.
                        MissingMethodException last = null;
                        if (delegate != closure && (delegate instanceof GroovyObject)) {
                            try {
                                return invokeMethodOnGroovyObject(methodName, originalArguments, delegate);
                            } catch (MissingMethodException mme) {
                                if (last == null) last = mme;
                            }
                        }
                        if (isClosureNotOwner && (owner instanceof GroovyObject)) {
                            try {
                                return invokeMethodOnGroovyObject(methodName, originalArguments, owner);
                            } catch (MissingMethodException mme) {
                                last = mme;
                            }
                        }
                        if (last != null) return invokeMissingMethod(object, methodName, originalArguments, last, isCallToSuper);
                    }

                    break;
                default:
                    if (method == null && owner != closure) {
                        MetaClass ownerMetaClass = lookupObjectMetaClass(owner);
                        method = ownerMetaClass.pickMethod(methodName, argClasses);
                        if (method != null) return ownerMetaClass.invokeMethod(owner, methodName, originalArguments);
                    }
                    if (method == null && delegate != closure && delegate != null) {
                        MetaClass delegateMetaClass = lookupObjectMetaClass(delegate);
                        method = delegateMetaClass.pickMethod(methodName, argClasses);
                        if (method != null)
                            return delegateMetaClass.invokeMethod(delegate, methodName, originalArguments);
                    }
                    if (method == null && resolveStrategy != Closure.TO_SELF) {
                        // still no methods found, test if delegate or owner are GroovyObjects
                        // and invoke the method on them if so.
                        MissingMethodException last = null;
                        if (isClosureNotOwner && (owner instanceof GroovyObject)) {
                            try {
                                return invokeMethodOnGroovyObject(methodName, originalArguments, owner);
                            } catch (MissingMethodException mme) {
                                if (methodName.equals(mme.getMethod())) {
                                    if (last == null) last = mme;
                                } else {
                                    throw mme;
                                }
                            }
                            catch (InvokerInvocationException iie) {
                                if (iie.getCause() instanceof MissingMethodException) {
                                    MissingMethodException mme = (MissingMethodException) iie.getCause();
                                    if (methodName.equals(mme.getMethod())) {
                                        if (last == null) last = mme;
                                    } else {
                                        throw iie;
                                    }
                                }
                                else
                                  throw iie;
                            }
                        }
                        if (delegate != closure && (delegate instanceof GroovyObject)) {
                            try {
                                return invokeMethodOnGroovyObject(methodName, originalArguments, delegate);
                            } catch (MissingMethodException mme) {
                                last = mme;
                            }
                            catch (InvokerInvocationException iie) {
                                if (iie.getCause() instanceof MissingMethodException) {
                                    last = (MissingMethodException) iie.getCause();
                                }
                                else
                                  throw iie;
                            }
                        }
                        if (last != null) return invokeMissingMethod(object, methodName, originalArguments, last, isCallToSuper);
                    }
            }
        }

        if (method != null) {
            return method.doMethodInvoke(object, arguments);
        } else {
            return invokePropertyOrMissing(object, methodName, originalArguments, fromInsideClass, isCallToSuper);
        }

Ok, before you get a headache, you’ll just focus on one thing : most of this code deals with the Closure case. What if your class is not a closure ? I guess that’s true for about 90% of method calls. For example, in the following code :

def myObject = new MySuperFastJavaObject()
myObject.mySuperFastMethod()

Here, you just want the method call to behave exactly as if it were made from pure Java. If you don’t write a custom metaclass, the metaclass that will be used by Groovy will dispatch your method call through the upper algorithm. You have understood that there’s no need to deal with the closure case here. So, you could write your own metaclass that removes everything from the closure case in the invokeMethod method :

package groovy.runtime.metaclass.com.mypackage;

public class MySuperFastJavaObjectMetaClass extends MetaClassImpl {
 ...
        @Override
 public Object invokeMethod(Class sender, Object object, String methodName, Object[] originalArguments, boolean isCallToSuper, boolean fromInsideClass) {
  checkInitalised();
  if (object == null) {
   throw new NullPointerException("Cannot invoke method: " + methodName + " on null object");
  }

  final Object[] arguments = originalArguments == null ? EMPTY_ARGUMENTS : originalArguments;
  MetaMethod method = getMethodWithCaching(sender, methodName, arguments, isCallToSuper);
  MetaClassHelper.unwrap(arguments);

  if (method != null) {
   return method.doMethodInvoke(object, arguments);
  } else {
   return invokeMissingMethod(object, methodName, arguments);
  }
 }
}

Several things to notice :

usage of the groovy.runtime.metaclass prefix in your package will guarantee that Groovy will automatically load your metaclass and assign it to your class
the name of your metaclass is the name of your class plus the MetaClass suffix
overriding some methods only can lead to huge performance improvements

Now, say your object doesn’t require dynamic behaviour like methodMissing. Why would you end up calling the invokeMissingMethod method ? If you take a closer look at this method, you’ll notice that the default implementation does many things, and, in your case, will always fail since you don’t have defined any methodMissing method. That’s really a waste of time. So you can directly replace it with a missing method exception :

package groovy.runtime.metaclass.com.mypackage;

public class MySuperFastJavaObjectMetaClass extends MetaClassImpl {
 ...
        @Override
 public Object invokeMethod(Class sender, Object object, String methodName, Object[] originalArguments, boolean isCallToSuper, boolean fromInsideClass) {
  checkInitalised();
  if (object == null) {
   throw new NullPointerException("Cannot invoke method: " + methodName + " on null object");
  }

  final Object[] arguments = originalArguments == null ? EMPTY_ARGUMENTS : originalArguments;
  MetaMethod method = getMethodWithCaching(sender, methodName, arguments, isCallToSuper);
  MetaClassHelper.unwrap(arguments);

  if (method != null) {
   return method.doMethodInvoke(object, arguments);
  } else {
   throw new new MissingMethodException(methodName, sender, arguments);
  }
 }
}

Note that I also had the opposite case : if I were on a missing method, the target method in my Java class was always the same. So I could hijack the meta-object protocol so that it doesn’t try a tons of things before dispatching to always the same method missing implementation :

@Override
 public Object invokeMissingProperty(final Object instance, final String propertyName, final Object optionalValue, final boolean isGetter) {
  Layer layer = (Layer) instance; // Layer metaclass, it's ALWAYS a Layer
         return layer.java_propertyMissing(propertyName);
 }

Another case was dealing with the following bug (?) : https://jira.codehaus.org/browse/GROOVY-4495. In a specific case, static method invocation always lead to the longest execution path possible. However, in my case, the class used is an utility class which inherits another, the two written in pure Java, and for which every method is static. In another words, it’s a toolbox. Why would I want dynamic method dispatching on this class ? There’s no reason. So, I implemented my own metaclass which directly delegated calls to the appropriate methods AND worked around the bug by adding the method defined by the parent class in the method cache.

Expected improvements

So, doing this, how much improvement can you expect ? Well, I’ll talk about my case. As I’ve already said, the critical parts of code are all written in pure Java. So most of Groovy is used as a DSL. After having added a single metaclass, I reached a 10% improvement in execution time. I was so surprised by the result that I added a few metaclasses, and I reached up to 25% execution time improvement depending on the application. The only thing I can say is that it is really important to take this in consideration when you want to make the best of Groovy. Try it yourself !

Experience feedback on Groovy

Tue, 27 Jul 2010 00:00:00 +0200

(lecteurs francophones, vous pouvez consulter la version française ici)

On the occasion of the 3rd anniversary of this blog, I wanted to write a special post. It appears that it will be special in two ways. First, it’s the first one to be published in both French and English. Second, this article is about a language which slowly took a large place in my everyday work for the last years, I mean Groovy. I hope this post will interest you, and will help those who hesitate on using this language industrially to make their choice : I’ll try to show off the pros and cons, and as usual, discussion is open : I encourage you to write comments. I will focus on the industrial consequences of choosing Groovy more than the technical issues, though both are related.

Once upon a time

Groovy adoption at Lingway wasn’t made in one day. I joined the team in a special circumstance, as the company I was working for was bought by Lingway. At this time, we already used to integrate scripting languages to allow easier customizations, but their usage was rather limited. I had made the choice to integrate both Mozilla Rhino and BeanShell, but not Groovy. Why ? Well the reason is quite simple : I had missed it.

At Lingway, in the context of rewriting Perl components to Java and improving them, I was driven by the idea of using scripting languages to make things easier to customize. Particularly, there were two main use cases :

the need to parametrize data acquisition workflows for our main product, Lingway Knowledge Management
the design of a new content extraction engine for documents written in natural language

It was by that time, reading posts about closures (the lack of) in Java that I discovered Groovy. It was the beginning of a long love story ! What Groovy allowed to do perfectly suited my requirements :

a language on the JVM
able to use Java libraries, and, very important, usable from Java libraries (also known as cross compilation)
a dynamic language supporting closures
ability to write DSLs (Domain Specific Languages)
a clear and simple syntax which would allow people not aware of development to write scripts

Those two use cases, which lead to a global adoption of Groovy in our components/products, will allow me to illustrate the pros and cons of this language in an industrial context.

Syntax matters

If there were one thing to highlight as a key factor of Groovy’s adoption, it would definitely be syntax. Groovy offerts over other JVM languages like Scala or Clojure, major advantages :

a syntax 95%-compatible with Java : it greatly reduces the learning curve since any Java developer will be able to code in Groovy without having to know nor understand the subtleties of this language. In a reduced team where there’s not much time allowed for learning, it is very important.
precious add-ons : utility methods/classes which simplifies the usage of standard JDK Apis and makes code more compact
usage of closures, which allows us to focus on algorithmics more than syntax
a dynamic type system, which allows newbies to avoid thinking about types

Let’s come back on the last point : I have already said that I planned to use Groovy in edge cases where users'' were not computer aware''. While this term is quite unfair, it does hide what I consider now as a great success in the introduction of Groovy. The challenge was dared, but it worked : at Lingway, we have three profile types in our technical team : first, developers - like me -, coming from software engineering. Second, consultants, who are trained for programming, but have lower pure technical skills : their main abilities are transforming customer needs into parametrization. Those are the people who write our workflows. Last but not least, linguists, who for most only know about computers as tools : they don’t know anything about programming languages and software engineering. However, thanks to Groovy, all those three profiles are able to collaborate on a single platform, a single language. The ability for Groovy to simplify at most syntax and the ability to create DSLs is amazing.

Let’s take a real example, in the context of acquisition workflows A java programmer would have written (correctly) the following :

Map inputMap = new HashMap()
inputMap.put(com.lingway.lkm.db.corpus.bean.fields.DublinExtendedKind.Title, "Hello, World !");
inputMap.put(com.lingway.lkm.db.corpus.bean.fields.DublinExtendedKind.Body,"Groovy is cool !");
inputMap.put(com.lingway.lkm.db.corpus.bean.fields.DublinExtendedKind.Language,com.lingway.lkm.db.corpus.bean.languages.LanguageKind.ENGLISH);

Usage of Groovy in workflows allows us to simplify it to :

inputMap = [
   title: "Hello, World !"
   body: "Groovy is cool !"
   language: "en"
]

This way of doing allows the workflow ``code'' to be much more readable. It allows us to focus on what to do, not on how to do. This is extremely important and allows us to save time : it is easier to read and maintain. Moreover, it is not required to be an expert to understand what it does : no need to know what a HashMap is. No need to even know that you actually need to instanciate one. Let’s see another example where we would like to sum up the numbers contained in a flat file (one number per line). Our Java developper would have written the following :

File file = new File("/tmp/data.txt");
int total = 0;
try {
 BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(file), "utf-8"));
 String line;
 while ((line=reader.readLine())!=null) {
  if (line.length()>0) total += Integer.valueOf(line);
 }
 reader.close();
} catch (IOException e) {
 // this must never happen, and if it does, I don't know what to do
}
System.out.println("total = " + total);

In Groovy, you’d just write this :

def total = 0
new File("/tmp/data.txt").eachLine("utf-8") { line ->
   if (line) total += line as int
}
println "Total : $total"

From 13 lines, we fall to 4. Those 4 lines only focus on what to do. At worse, you’ll have to explain to your interlocutor what eachLine or as int does, and you’re done. Likely, Groovy is the perfect candidate for simplifying the usage of Java APIs. One of my favorite example is taken from the Gaelyk documentation, a simplified web framework based on Groovy :

mail.send sender: "app-admin-email-AT-gmail-DOT-com",
   to: "recipient-AT-somecompany-DOT-com",
   subject: "Hello",
   textBody: "Hello, how are you doing? -- MrG",
   attachment: [data: "Chapter 1, Chapter 2".bytes, fileName: "outline.txt"]

Here’s an example of a ``mini-DSL'', dedicated to sending e-mails. How could one imagine something simpler ? When you actually know about the verbosity of the Javamail API, it’s no match…

So, thanks to Groovy’s adaptability, we were able to :

make our workflows readable
write a DSL dedicated to linguistic extraction rules. This language is the core of our internal data extraction tool, and is mainly used by our linguists

About that, the strengths of Groovy in such a tool are multiple :

it allows non developers to write rules which are compiled to bytecode then executed by the JVM
when the DSL is not sufficient, linguists may ask the developers for help. The latter would then write chunks of Groovy code which perform complex operations

Therefore, what is possible is not limited to what the DSL allows. It is something particularly important to understand : if we had chosen to write a classical DSL, a rule based engine which would use its own syntax, then we would probably have achieved a higher level of readability, but we would also have had to :

either write an interpreter (simple solution) or a compiler (complex one) for our rules
develop new versions of the language as new needs are discovered

With Groovy, you just skip those steps, and you just earn an extra : it’s just code. Even if linguists actually write rules, there’s nothing that prevents us from writing regular code inside. The whole language is usable…

A funny thing is that as time passes by, linguists show an increasing curiosity towards the ``code'' part of rules. They naturally aim at factorizing rules : the language becomes structuring and leads to better code quality !

The barriers

So far I’ve been particularly enthusiast about Groovy. However, it’s not that simple, and there are things that are get complicated. I’ll split the barriers into two categories : technical barriers and humain barriers. Don’t neglect any of them.

Technical barriers

The first technical barrier we encountered was performance. Release after release, Groovy becomes faster and I can tell you that the current versions are really fast. However, don’t expect miracles. In particular, in the context of our extraction engine, we had a very important performance expectation. The objective, for example, was to be able to perform a complete resume parsing and data extraction (name, surname, personal data, experiences, trainings, …) from a binary document (Word,…) within a second. If the core of our engine had been written in Groovy, there’s no chance that we could have reached such a performance. That’s why we decided to write the critical parts in pure Java, while the domain code is written in Groovy (leading to a DSL). This way we have a good trade-off between performance and readability.

So, if writing code in Groovy is really easy thanks to its syntax, it is just also easy to write slow code. I remember a parser written in Groovy which read the XML configuration file of our engine. This code was written in Groovy because the XmlSlurper makes it really easy to read XML files. Whatever, the Java code was 20 times as fast as the Groovy version… (admittedly, almost 20 times longer). Another example about the curious default type used by Groovy in decimal computations :

def num = 1.1

The type associated to num is not, as one would expect, float nor double, it’s BigDecimal. As a direct consequence, every benchmark found on the web about Groovy falls into this trap. You just have to strongly type your code to make performance acceptable (and more). For a language which simplifies life by avoiding strong typing, it seems curious and just mystifies the principal of least surprise (for the curious, there’s an explanation for that, as the Groovy developers chose to apply the principal of least surprise to the result of computations more than on the types : using BigDecimal allows computations to be exact).

Using Groovy code from Java leads to another barrier : since the natural way of coding in Groovy is to weakly type, Groovy generated APIs only take Object as parameters. Those APIs are just unusable, so if your Groovy code is intended to be used from Java, you’ll have to make the effort to strongly type.

Another barrier is directly related to the global adoption of Groovy in the technical team. The success of Groovy makes that we wish to introduce it everywhere. However, when different components include different versions of Groovy, you take the risk of compatibility issues. Happily, unlike the Scala language for example, Groovy maintains binary compatibility between one version and another. I greatly reduces the risks when upgrading.

Human barriers

Curiously, the main barriers encountered during the adoption of Groovy were not technical but humain. And more curiously, the ones I faced did not come from people I expected. I expected linguists to rebel, it were developers !

To understand properly, you must understand that developers are Lingway all have nearly 10 years of experience. I can modestly say that they are good (if not very) developers. As good developers, they use good tools : I cannot understand when people use vi or Emacs to code : the main strength of Java has never been the language, but rather its tools. At Lingway, we use IntelliJ IDEA. This IDE is for me the best IDE available on the market for Java development. We’ve used it for long, and getting back to Eclipse would be worse than a curse for us. With a strongly types language like Java, and even more with the addition of generics, code is understandable (but rather noisy) : you actually know, reading the code, that this collection actually contains that type of objects. The compiler will complain if you try to use different content, and if you use an intelligent IDE, without having to compile, it will indicate to you what are the possible choices for what method, depending on the context. As time goes, the programmer develops what I call the ``completion frenzy'' : you actually pass most of your time pressing the CTRL+space or CTRL+Q key combinations. It’s no use to read javadoc, since my IDE will gently indicate me what the method expects at what position. When you intensively practice that, you may reach an incredible productivity.

In that context, the transition to Groovy looks like a regression : being widely sub-typed (and I’m fighting to make understand that weakly typed doesn’t mean untyped), it’s most of time impossible to know what a method/closure/map expects as a parameter without reading the documentation. However, when we come to documentation, we can find the best, like the worse. Even your IDE is useless : it doesn’t have enough hints to help you. Most of CTRL+space calls are headed for failure : it leads to an incredible frustration.

Therefore, developers just tend to come back the natural way : Java developers ``Java-ize'' their code, instead of Groovyfying it. We loose readability for ease of development. It’s quite paradoxical and I try to fight against this, but it’s difficult to challenge : I just think that unless you are a very curious and open developer, you’ll find it frustrating to fly visually. It’s just like getting 10 years back.

I must admit I have not succeeded to perfectly initiate the Groovy spirit to the team. Some make resist while others make efforts but it’s a question of feeling first : it’s very hard to fight against natural tendencies.

Recently, we started using Grails for an application of e-reputation analysis. This development is still in progress, and once again, I made a bet to try to developer faster. For now, I’m really satisfied of the result which leads to an unprecedented productivity. However, frustrations are not gone : the IDE support is far from perfect and widely insufficient : almost no completion, no differentiation between dynamic methods which are added by default to every object and service methods, for example. There’s not much more completion for render parameters. For taglibs, the standard completion is insufficient and there’s no way for the IDE to actually help because Grails taglibs definition miss metadata about required attributes and so on. You just actually have to open multiple Grails help web pages to get it right.

Conlusion

Through this post, I tried to show off the industrial usage of Groovy, integrated in many components which actually are in production (in our case, for more than 3 years). The global balance is positive, but you really don’t have to neglect the barriers of the integration of Groovy. Particularly, Groovy doesn’t escape one of the most complicated activities : driving change. A programmer which is too comfortable with Java will have difficulties to embrace the language and will sometimes be awfully insincere just to justify his own choices, guided by personal comfort : scarifying readability for tools. This is not unjustifiable, since I often find myself cursing the lack of completion from my IDE. This is not enough to change my mind about Groovy : this is surely one of the best thing that happened to Java for the last 10 years… A language to recommand, and I hope it’ll spread widely !

(for my english-speaking audience, my english is not perfect, do not hesitate to correct me)

Retour d’expérience : Groovy

Mon, 26 Jul 2010 00:00:00 +0200

(english speakers may read the english version here)

A l’occasion des trois ans de ce blog, je souhaitais écrire un billet un peu spécial. Il le sera à double titre, puisque tout d’abord publié en deux langues et ensuite parce que consacré à un retour d’expérience sur un langage qui a pris énormément d’importance dans mon travail ces dernières années : Groovy. J’espère que cet article vous intéressera, et aidera celles et ceux qui hésitent à employer ce langage daniqs leur entreprise à faire leur choix : je dresserai aussi bien les aspects positifs de son utilisation que ceux négatifs. Par ailleurs, comme toujours, la discussion est ouverte, et je vous invite à réagir en écrivant en commentaire. Mon objectif est de centrer ce retour sur l’utilisation industrielle de Groovy.

Un peu d’histoire

L’adoption de Groovy chez Lingway ne s’est pas faite en un jour. J’ai intégré cette entreprise d’une manière assez particulière, puisque l’entreprise pour laquelle je travaillais s’est faite rachetée par Lingway. A cette époque, nous utilisions déjà des langages dits de scripting pour faciliter la personnalisation de nos produits. Leur utilisation était alors très limitée. J’avais fais le choix d’utiliser Mozilla Rhino et BeanShell. Pourquoi pas Groovy ? Tout simplement parce que je n’avais pas entendu parler de ce langage.

Arrivé chez Lingway, la réécriture de nombreux composants Perl vers Java et l’amélioration de ceux-ci m’ont poussé à creuser l’idée de l’utilisation de langage de scripts pour la personnalisation de l’application aux besoins utilisateurs. En particulier, deux cas d’utilisation distincts :

la nécessité de paramétrer des workflows d’acquisition de données pour l’indexation de documents dans notre produit, Lingway Knowledge Management
l’écriture d’un nouveau moteur d’extraction de données textuelles dans les documents en langue naturelle

Cet à ce moment qu’au détour de lectures sur les closures en Java (ou plutôt, l’absence de closures), je fis la découverte de Groovy. Ce fut le début d’une longue histoire d’amour ! Ce que permettait Groovy correspondait exactement à mon besoin :

un langage fonctionnant sur la JVM
qui puisse appeler des librairies Java, et qui, très important, puisse en retour être appelé depuis Java
un langage dynamique supportant les closures
la possibilité de créer des DSL (Domain Specific Languages)
une syntaxe simple, permettant à des non informaticiens d’apprendre et d’écrire des scripts

Les deux cas d’utilisation principaux, qui ont mené à l’adoption générale de Groovy dans de nombreux composants/produits, vont me permettre d’illustrer les avantages et les inconvénients de ce langage, dans un contexte industriel.

La syntaxe est importante

S’il y a un élément que je dois mettre en avant comme étant l’élément clé de l’adoption de Groovy dans l’entreprise, c’est bien celui de la syntaxe. Groovy offre, par rapport à d’autres langages sur la JVM comme Scala ou Clojure, des avantages immenses :

syntaxe à 95% compatible Java : n’importe quel développeur Java peut écrire du Groovy, sans avoir besoin de comprendre les particularités de ce langage : celà permet d’avoir un temps d’apprentissage extrêmement réduit. Lorsque dans une équipe, on n’a pas beaucoup de temps à consacrer à la formation, c’est un plus indéniable
des ajouts précieux : l’ajout de nombreuses méthodes ``utilitaires'' simplifiant l’utilisation des API standard rend le code plus compact
l’utilisation des closures, qui permettent de se concentrer sur l’algorithmique avant la syntaxe
un typage dynamique, qui permet aux non initiés d’éviter d’avoir à se poser la question des types

Revenons sur ce dernier point. A plusieurs reprises, j’ai fait référence à la possibilité d’utiliser Groovy pour faire travailler des ``non informaticiens''. Si le terme est un peu abusif, il n’en résume pas moins ce que je considère avec le recul comme une grande réussite. Le pari était osé, mais il a fonctionné : chez Lingway, nous avons trois types de profils dans l’équipe de développement. Il y a d’abord les développeurs, comme moi, issus de formations en ingénierie informatique. Ensuite, nous trouvons les consultants, qui s’ils sont formés à l’informatique et au développement, ont un niveau de technicité moins élevé : leur expertise repose sur la compréhension du besoin client et au paramétrage de nos produits. En particulier, ce sont eux qui écrivent les fameux workflows d’acquisition. Enfin, la dernière catégorie correspond aux linguistes. S’ils sont formés à l’informatique, c’est avant tout en tant qu’outil : la plupart d’entre eux ne connaissent rien des langages de programmation. Pourtant, grâce à Groovy, ces trois profils peuvent cohabiter sur une même plateforme, un même langage. La capacité qu’on a, avec Groovy, de simplifier à l’extrême la syntaxe, et à écrire des langages de domaine est extraordinaire.

Prenons un exemple concret, dans le cadre des workflows d’acquisition de données. L’ingénieur Java, sans Groovy, aurait écrit ceci (et il aurait bien fait son travail) :

Map inputMap = new HashMap()
inputMap.put(com.lingway.lkm.db.corpus.bean.fields.DublinExtendedKind.Title, "Hello, World !");
inputMap.put(com.lingway.lkm.db.corpus.bean.fields.DublinExtendedKind.Body,"Groovy is cool !");
inputMap.put(com.lingway.lkm.db.corpus.bean.fields.DublinExtendedKind.Language,com.lingway.lkm.db.corpus.bean.languages.LanguageKind.ENGLISH);

L’utilisation de Groovy dans nos workflows permet de simplifier l’écriture ainsi :

inputMap = [
   title: "Hello, World !"
   body: "Groovy is cool !"
   language: "en"
]

Cette façon de faire permet de rendre le code du workflow beaucoup plus lisible, et de se concentrer sur ce que doit faire le workflow, non sur la syntaxe. C’est extrêmement important, et ça permet de gagner énormément de temps : la lecture est facilitée, la maintenance aussi. Par ailleurs, il n’y a pas besoin d’une expertise énorme pour comprendre ce que celà fait. Pas besoin de comprendre de qu’est une hashmap. Pas besoin de savoir, même, qu’il faut en créer une… Un autre exemple, concernant la lecture d’un fichier ligne par ligne sur lequel on souhaiterait faire la somme des entiers qu’il contient: notre développeur Java aurait écrit ceci :

File file = new File("/tmp/data.txt");
int total = 0;
try {
 BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(file), "utf-8"));
 String line;
 while ((line=reader.readLine())!=null) {
  if (line.length()>0) total += Integer.valueOf(line);
 }
 reader.close();
} catch (IOException e) {
 // this must never happen, and if it does, I don't know what to do
}
System.out.println("total = " + total);

En groovy, on se contentera d’écrire ceci :

def total = 0
new File("/tmp/data.txt").eachLine("utf-8") { line ->
   if (line) total += line as int
}
println "Total : $total"

De 13 lignes, on passe à 4. Ces 4 lignes se concentrent sur la seule chose qui importe : le traitement. Au pire, vous devrez expliquer à votre interlocuteur novice en Groovy ce que fait eachLine ou as int, et tout est compris… De la même façon, Groovy est un excellent outil pour simplifier l’utilisation des API Java. Un des exemples qui me plaît le plus est celui de l’utilisation de l’API Java mail (tiré de la documentation de Gaelyk, un framework web léger basé sur Groovy) :

mail.send sender: "app-admin-email-AT-gmail-DOT-com",
   to: "recipient-AT-somecompany-DOT-com",
   subject: "Hello",
   textBody: "Hello, how are you doing? -- MrG",
   attachment: [data: "Chapter 1, Chapter 2".bytes, fileName: "outline.txt"]

Il s’agit ici d’un mini DSL, dédié à l’envoi de mails. Comment imaginer plus simple ? Lorsqu’on connait la verbosité de l’API Javamail, il n’y a pas photo…

Ainsi, grâce à la flexibilité de Groovy, nous avons pu :

rendre lisible nos workflows
écrire un langage domaine dédié à l’écriture de règles linguistiques. Ce langage est au coeur de notre outil interne d’extraction d’informations, et est utilisé par des linguistes.

A ce propos, la force de l’utilisation de Groovy dans un tel outil est multiple :

il permet à des non informaticiens d’écrire des règles qui sont compilées en bytecode exécuté par la JVM
lorsque le DSL est insuffisant, les linguistes peuvent faire appel aux développeurs, qui écrivent des ``bouts de code Groovy'' qui réalisent les opérations complexes

Ainsi, le possible ne se limite plus à l’expressivité du DSL. C’est un point particulièrement important à comprendre : si on avait choisi d’écrire un DSL classique, un moteur à base de règles disposant de sa propre syntaxe, nous aurions certes sans doute atteint un niveau de lisibilité supérieur à ce qu’il est possible de faire en Groovy, mais nous aurions du :

soit écrire un interpréteur, version simple, soit un compilateur, version complexe, de règles
développer de nouvelles versions du langage au fur et à mesure que les besoins apparaissent

Avec Groovy, on s’affranchit de ces deux étapes, et on dispose d’un bonus non négligeable : il s’agit de code. Même si les linguistes écrivent des règles, il n’en reste pas moins qu’au milieu de ces règles, on peut faire appel à tout le langage, et donc faire toutes les opérations possibles et imaginables…

Chose amusante, nous nous rendons compte qu’avec le temps, nos linguistes montrent une curiosité grandissante envers la partie ``code'', et tendent naturellement à générifier leurs règles : le langage devient structurant, et appele à une meilleur qualité !

Les freins

Si je suis jusqu’ici particulièrement entousiaste, il n’en reste pas moins que tout n’est pas rose dans le monde de Groovy. Je classerai les freins en deux catégories : les freins techniques d’une part et humains d’autre part. Aucun des deux n’est à négliger.

Les freins technique

Le premier frein technique que nous avons rencontré concerne les performances. Au cours du temps, les performances de Groovy s’améliorent fortement (et je vous garantit que c’est déjà extrêmement rapide), pour autant, il ne faut pas s’attendre à des miracles. En particulier, dans le cadre de notre moteur d’extraction de données, nous avions un impératif de performance extrême. L’objectif, à titre d’exemple, était d’écrire un moteur d’extraction des données d’un CV (nom, prénom, informations personnelles, expériences, entreprises, formations, …) en moins d’une seconde en moyenne par CV. Si le coeur du moteur avait été écrit en Groovy, jamais nous n’aurions atteint de telles performances. Ainsi, pour ce projet particulier, le code critique est écrit en Java, et le code ``domaine'', autrement dit, ce qui correspond à la syntaxe, est écrit en Groovy. On dispose alors d’un excellent compromis : la syntaxe simplifiée Groovy avec les performances de Java.

Ainsi, si Groovy permet de simplifier énormément la syntaxe, il est aussi tout à fait possible d’écrire du code très peu performant. J’ai en mémoire une première version du code de parsing du fichier XML de configuration du moteur. Ce code avait d’abord été écrit en Groovy, parce qu’il permettait grâce au XmlSlurper de lire le fichier très rapidement. Seulement, la version Java était 20 fois plus rapide… L’autre exemple concerne le typage par défaut en Groovy. Assez curieusement (mais c’est un choix qui se justifie), lorsque l’on écrit ceci en Groovy :

def num = 1.1

Le type associé n’est pas float, mais un BigDecimal (merci Guillaume pour la correction ;)). Conséquence directe, tous les benchmarks publiés sur le web concernant Groovy démontrent son extrême lenteur. Hors, il suffit de typer'', autrement dit remplacer def'' par ``int'' pour obtenir des performances plus que raisonnables. Pour un langage qui nous affranchit de typer, ce cas est à la limite du principe de la moindre surprise prônée par les développeurs (pour les curieux, le type par défaut est ainsi choisi parce qu’il permet d’avoir des calculs exacts, permettant ainsi d’obtenir le principe de la moindre surprise du côté des résultats).

Un autre frein concerne l’utilisation de classes Groovy depuis Java. Puisque la façon naturelle de développer en Groovy est de sous-typer, on se retrouve naturellement avec des API Groovy qui ne prennent que des Object en paramètre. Une pure hérérie du point de vue Java, et surtout, des API inutilisables. La plupart du temps, si votre code est destiné à être utilisé depuis une classe Java, vous devrez faire l’effort de typer correctement, quitte à perdre la lisibilité du langage.

Un autre frein technique, de moindre importance est directement lié au succès de Groovy dans l’équipe technique : on a envie d’en mettre partout. Et lorsque différents composants dépendent de différentes versions de Groovy et qu’ils doivent cohabiter, il y a risque de conflits de version. Fort heureusement, contrairement à Scala, Groovy maintient ses binaires compatibles d’une version à l’autre, ce qui permet de grandement limiter les risques.

Les freins humains

Curieusement, les principaux freins rencontrés à l’utilisation de Groovy ne sont pas techniques. Ils sont humains. Et encore plus curieusement, ceux que j’ai rencontrés ne venaient pas de la population que j’attendais. Je m’attendais à ce que ce soient les linguistes puis les consultants qui râlent, ce furent les développeurs !

Pour bien comprendre, il faut savoir que les développeurs chez Lingway ont chacun près de 10 ans d’expérience en Java. Modestement, je puis affirmer qu’ils sont bons, voire très bons. En bons développeurs, ils utilisent de bons outils : je ne peux pas comprendre qu’un développeur Java digne de ce nom utilise encore vi ou Emacs pour développer : la force de Java n’a jamais été le langage, mais bien les outils. En particulier, chez Lingway, nous utilisons IntelliJ IDEA. Cet IDE est sans aucun doute le meilleur IDE Java disponible sur le marché. Nous l’utilisons depuis longtemps, et revenir à Eclipse constituerait pour nos une punition digne du pire supplice chinois. La force de tels outils, fin de parenthèse, c’est entre autres la complétion de code. Avec un langage fortement typé tel que Java, et a fortiori avec l’utilisation des génériques, à défaut d’être compact, le code Java est compréhensible : on sait, par exemple, que telle collection contient telle liste d’objets. Le compilateur plantera si vous tentez d’enfreindre la règle, et si vous utilisez un bon IDE, sans avoir besoin de compiler, il vous indiquera les choix possibles à chaque appel de méthode, en fonction du contexte. Avec le temps, se développe ce que j’appelle la ``completion mania'' : l’appel du CTRL+space ou du CTRL+Q devient un trouble compulsif. Pas besoin de lire la javadoc, puisque mon IDE va gentilment me dire ce que je dois coller comme paramètre. Lorsqu’on pratique ceci intensément, on gagne un temps incroyable en développement.

Dans ce contexte, le passage à Groovy apparaît comme une régression : en étant largement sous-typé (mais typé, je me bats pour qu’on comprenne que Groovy est un langage fortement typé MAIS dynamique) tant la pratique l’encourage, il est impossible, la plupart du temps, de savoir quoi passer en paramètre sans passer par la documentation de l’API. Et en matière de documentation, il y a de tout… Ici, votre IDE n’a que peu d’armes à fourbir : il est aussi perdu que vous : comment savoir quels sont les paramètre obligatoires, et quels types sont réellement attendus ? La plupart des appels au CTRL+space sont voués à l’échec, conduisant à une énorme frustration.

La tendance naturelle revient alors au galop : les développeurs Java chevronnés Java-isent'' leur code, au lieu de le Groovyfier, juste pour disposer de l’intelligence de l’IDE. On perd ainsi énormément en lisibilité, contre la facilité de développement. C’est assez paradoxal, et j’essaye de me battre contre cette pratique, mais elle est difficilement contestable : je crois qu’à moins d’être extrêmement curieux et ouvert, un développeur Java expert trouvera très frustrant de naviguer à vue''. L’impression de revenir 10 ans en arrière, avant les premiers IDE intelligents, est concrète et difficilement contestable.

J’avoue ne pas être totalement parvenu à imprénier ``l’esprit Groovy'' à l’équipe de développement. Certains font de la résistance, quand d’autres font des efforts, mais c’est avant tout une question de feeling : il est très difficile de combattre les tendances naturelles.

Récemment, nous sommes passés à Grails pour le développement d’une application d’analyse de la e-Réputation. Le développement est encore en cours, et cette fois encore, j’ai pris le pari de bousculer l’équipe pour réussir à développer plus vite. Pour l’instant, je suis extrêmement satisfait du résultat, une productivité inégalée. Pour autant, les frustrations n’ont pas disparu. Le support de l’IDE est largement insuffisant : presque aucune complétion, pas de différenciation entre les méthodes dynamiques communes à tous les objets Groovy, et les méthodes des services, par exemple. Pas plus, non plus, d’aide sur les paramètres des méthodes render, … Pour les taglibs, la complétion des paramètres standard est insuffisante, et celle des taglibs custom impossible à réaliser, puisqu’aucune métadonnée n’est présente permettant à l’IDE de déduire quels paramètres sont obligatoires ou non.

En conclusion

A travers ce billet, j’ai essayé de vous dresser un portrait de l’utilisation de Groovy en mode industriel, intégré au sein de produits en production (dans notre cas, depuis 3 ans). Le bilan est largement positif, mais il ne faut pas négliger les aspects négatifs de l’intégration. En particulier, Groovy n’échappe pas à une des activités les plus complexes : la conduite du changement. Un développeur trop bien installé aura du mal à s’y mettre, et aura même tendance, parfois, à user d’une terrible mauvaise foi pour justifier des choix qui ne sont que personnels : sacrifier la lisibilité, qui fait la force de Groovy, à l’utilisation des outils, et donc au confort. Cette lutte n’est pas sans fondement, et je suis moi-même souvent en train de pester contre le manque de complétion dans mon IDE, qui impose d’avoir constamment plusieurs onglets Grails d’ouverts. Ces points noirs ne sont pas anodins, mais ne changent en rien mon avis général sur Groovy : c’est la meilleure chose qui soit arrivé à Java ces 10 dernières années. Un langage a recommander absolument, et qui je l’espère gagnera en popularité !

Downloading sources and javadocs of dependencies in Grails

Mon, 28 Jun 2010 00:00:00 +0200

After having searched half the internet for a solution, here’s the simplest one I’ve found for downloading the sources and the javadoc of dependencies in Grails :

grails install-plugin eclipse-scripts
grails download-sources-and-javadocs

Note that you actually don’t need to use Eclipse for this to work (I don’t). Unfortunately, Jetbrains’ IntelliJ IDEA will not recognize sources and javadocs automatically, so you’ll have to attach them manually. No magic trick for that. For those interested in this feature, please vote for it.

I’m quite astonished this feature is not bundled with Grails. It’s much more complicated to actually deal with third party libraries without access to either the documentation or the sources…

[edit]Jetbrains have fixed the issue, but only in IntelliJ IDEA X. Hope it’ll get backported to 9.0.x.

Coding a Groovy closure in pure Java

Mon, 19 Apr 2010 00:00:00 +0200

Today I just wanted to share a simple trick that’s not well documented in Groovy : in a project which mixes Java and Groovy code, I’ve faced a problem where I wanted to use a Groovy method in Java which took a Closure as a parameter. Here’s how to do it. Actually, it’s quite simple :

Groovy

class Operation {
    def op(list, action) {
        list.each { action(it) }
    }
}

Java

public class MyJavaClass {
    private Operation groovy = new Operation();

    private void shouldBeUsedAsClosure(Object o) {
        System.out.println(o);
    }

    public void demonstrateClosureUsage() {
        // creates a closure which delegates calls to this Java object
        MethodClosure cl = new MethodClosure(this, "shouldBeUsedAsClosure");
        groovy.op(Arrays.asList("hello", "world"), cl);
    }

    public static void main(String... args) {
         MyJavaClass java = new MyJavaClass();
        java.demonstrateClosureUsage();
    }
}

This example is straighforward, however, this allows you to write your custom ``closures'' directly in Java. This is somehow interesting whenever performance matters and that critical portions of code need to be coded in Java.

Note that you can have multiple methods with the same name, Groovy will automatically dispatch the call to the appropriate delegate :

public class MyJavaClass {
    private Operation groovy = new Operation();

    private void shouldBeUsedAsClosure(String str) {
        System.out.println(str.toUpperCase());
    }

    private void shouldBeUsedAsClosure(int x) {
        System.out.println(x*x);
    }

    public void demonstrateClosureUsage() {
        // creates a closure which delegates calls to this Java object
        MethodClosure cl = new MethodClosure(this, "shouldBeUsedAsClosure");
        groovy.op(Arrays.asList("hello", "world","this","should","print","4 : ",2), cl);
    }

    public static void main(String... args) {
         MyJavaClass java = new MyJavaClass();
        java.demonstrateClosureUsage();
    }
}

This is an elegant way to avoid multiple instanceOf tests to check the type of the closure arguments. Last but not least, Groovy also provides a way to call static methods as closures, passing the class instead of an instance as the owner parameter :

   MethodClosure cl = new MethodClosure(MyJavaClass.class, "staticMethod");

Has the Java build tools war begun ?

Thu, 15 Apr 2010 00:00:00 +0200

At the beginning everything was simple

In this post I will not discuss the pros and cons of each and every build tool (mainly because I only worked with Ant and Maven and know and read about the others), but rather start a debate on whether so many tools is required : I’m uncomfortable with the increasing number of build tools available for Java. At the beginning, everything was simple : Ant was here, and did its job. However, while Ant is very good at build customization, it had the cons of its pros: for each project, you had to copy parts of your ant scripts from another project, or create an Ant plugin. However, in every organization, or worse, every developer had its own way of building projects, organizing source code and so you could rapidly come with projects which used the same build tool, but in a totally different manner.

Then came Maven

I’m quite happy with Maven 2, which allows us to get reproducible, standardized builds with dependency management. While Maven is not perfect, it has many advantages over Ant, the most important thing to my eyes beeing that the convention based idea forces well structured and understandable code layout. It makes switching from one project to another easy because every project ``look like'' another. You know where to find sources, test sources, resources and so on.

However, Maven 2 is far from perfect, and has many problems :

very poor documentation : there are tons of plugins, but documentation for each plugin is often insufficient.
build descriptors are verbose : that’s XML. XML is verbose.
builds are slow : each build requires parsing the XML files, checking for dependencies, downloading, … and incremental compilation is mostly broken
integrating libraries which do not use Maven, or worse, code compilers/generators which don’t have a Maven plugin is a pain (think of Flex, which lacked a decent plugin until very recently).

Starting from those (and other) problems, people started to create new build tools. To my knowledge, the first one was Ivy, which basically Ant plus dependency management. Great for guys who can’t stand Maven’s well structured (but conservative) build architecture but love the idea of dependency management introduced by Maven. Got to build Flex ? Ok, call the Flex compiler as an Ant task. Done.

Then came Groovy

I love Groovy. It’s easy to learn if you’re comfortable with Java, and it makes it very easy to write DSLs. The first thing to come was the integration of Ant into Groovy, and the creation of a Groovy DSL for Ant : Gant. It offers you the way to call Ant tasks from Groovy code easily, or, would you like it, build your project with Gant and Groovy code instead of XML files.

A few days ago, the developers announced that Groovy 1.8, the next major version, would use Gradle as a build tool instead of Ant. What it Gradle ? As far as I understand, Gradle can be thought as a Groovy DSL for both Ant and Maven 2, while it does not rely on Maven : it takes the ideas from Maven, combines them with the descriptive tasks from Ant which makes it easy to build customized builds. Why is it so easy ? Well, just because your Gradle builds is actually Groovy code…

That’s the idea. Ok, but I’m aware of different tools which do the same thing :

Buildr, from Apache, defined as A drop-in replacement for Maven 2.0, Buildr uses the same file layout, artifact specifications, local and remote repositories, but uses a dedicated language (not XML, nor Groovy).
Kundo, which just seems to do exactly the same thing as Gradle, but leaks a decent documentation
GMaven, which is a Maven plugin which allows you to compile Groovy code from Maven, compiling and using code from and to Java in Groovy, and allows you to write parts of your build process in Groovy too. This is the tool I’m currently using to build Groovy code from Maven.
Maven 3, yet to be released, which will allow you to abandon your XML files for a Groovy DSL, and use Groovy code in your builds. From what I read, it will likely allow us to to the same thing as Gradle and the previously mentioned tools

This list is based on tools I’m aware of, there must be some other tools available too. But the question is here : do we really need so many tools ?. For me, Ant is required, for its simplicity which allows building projects or libraries without taking care of the Java ecosystem. Maven is required for any larger project, or for OpenSource projects or libraries : thanks to conventional rules and dependency management, it’s quite easy to integrate any library. That makes two. For me, Gradle and other tools are more likely tools that make it easier to build complex projects than revolutionary approches : nothing was impossible to do with Ant or Maven. With Groovy support coming in Maven 3, I’m really wondering if it’s worth learning a new language at the time development teams become comfortable with Maven.

In another software development domain, it makes me think about version control : CVS, Subversion, now Git and Mercurial… And you, what do you think : has the war already begun ?

JLangdetect 0.2 released

Wed, 30 Sep 2009 00:00:00 +0200

JLangDetect 0.2 released !

This is a small update which includes the following features :

Ability to detect the language of a document using a subset of the languages used for training
Logs now managed by log4j

The ability to use a subset of the languages used for training is important if you know that a document must be written in french or english, for example, but the detector has been trained for more languages. Using a subset will ensure that the detector returns one of those languages.

Downloads

JLangDetect is licensed under Apache 2.0.

Binary : jlangdetect-0.2.jar
Source: jlangdetect-0.2-sources.jar
Javadoc: jlangdetect-0.2-javadoc.jar
Europarl pre-compiled corpus: ngrams-europarl.zip

Version Control

Project is hosted on Google code.

JLangdetect available at google code

Thu, 16 Jul 2009 00:00:00 +0200

Hi,

Several persons asked for it, so I took some time to create a google code project for JLangdetect. You’ll find it here :

https://code.google.com/p/jlangdetect/

I had not much time to improve it, so feel free to contribute. My ideas for future directions include :

remove ``irrevelant'' portions of texts from corpora to reduce the size of the n-gram trees
add ability to limit the tested text to a subset of test languages (useful if you know that your text is either in english or french, but your detector is configured with more languages)
add pre-filters to both learning and detection algorithms in order to address problems like case (if the corpus is a large well written text but the tested string is an uppercase title, then detection will likely be wrong)
improve detection thanks to pluggable add-ons like lexicon recognition, …

Take a look at my posts related to JLangdetect for more details.

Ubuntu Jaunty unusable for Swing debugging ?

Thu, 30 Apr 2009 00:00:00 +0200

Ubuntu 9.04 Jaunty unusable for Swing GUI debugging ?

I (and Thomas Singer) am having trouble with debugging Swing GUIs under Jaunty. The problem, described in this post doesn’t seem to be only related to IDEA, but is rather annoying : the whole desktop freezes when the debugger reaches a breakpoint. The only way to release the lock is to switch to a console with CTRL+ALT+F1 and kill the debugged process.

It looks like some already had the problem with Netbeans in previous versions of the JDK, and a workaround was available thanks to the AllowDeactivateGrabs option in xorg.conf. However, it seems that this option has been removed from Xorg. That’s a pity as it just doesn’t leave any workaround for this bug.

If anyone has a solution, please do not hesitate to comment this post, or join the discussion

[Update] Adobe releases a buggy flash plugin once again

Wed, 15 Oct 2008 00:00:00 +0200

Today more than ever, I’m really angry about Adobe. They have just released a brand new Flash 10 plugin with a very annoying bug which is there for months : Adobe hates Linux users, especially non-US users. Would anyone tolerate that a Windows user can’t write text in its favorite editor ? No, because if you’re not able to input text, you are not able to do anything with your computer. It’s just what’s happening with this Linux release once again : special characters (unicode) like accents, @, \{, … can’t be entered.

The bug has been reported in early April this year AND IT’S STILL NOT FIXED ! Furthermore, no one at Adobe deigns to answer. Adobe ignores 39 votes, placed 10th most voted in the bug tracker. You’ll also notice that the most voted is another Linux issue.

It’s really a pity that Adobe does not wish to take Linux users seriously. It just makes impossible to release Flex based applications for customers which use Linux. More, it makes our work harder since, like me, many developers develop on this platform.

Update : I had a comment from Adobe this morning, saying they are working on the issue. I hope this is good news for us, and that I’ll be able to reconsider my opinion soon.

Language detection : more thoughts

Thu, 25 Sep 2008 00:00:00 +0200

Following my previous post, some people asked me if JLangDetect was sufficient by itself. The obvious answer is no. There are many reasons for that. A good language detection should combine several heuristics. JLangDetect is one. In Lingway KM, we use several techniques to improve detection. For example, there are few steps that you must ensure, in order to maximize your chances.

JLangDetect does a good job because it requires a correct input :

have the inputstream is correctly read, meaning you should ensure that character encoding is correct. Java internaly encodes string in unicode, but your reader must know the encoding before reading the bytes. This makes it not as trivial as it first seems, and that’s why projects like JChardet exist. Read a iso-latin1 character stream as if it was unicode, and your language detection will be totally broken.
make sure you have raw text : HTML and XML is forbidden. Language detection is optimal on plain text. Tags are very noisy, and will result in wrong language detection.
Try to keep revelant parts of the input stream : if your text is issued from a web page, there are chances that you’ll get noise caused by advertisements, menus and so on. You should try to keep your detector focused on what you want to detect : the language of the text.

NLP in Java : A language detector

Sun, 21 Sep 2008 00:00:00 +0200

Check out the latest version of the project

Hi,

Today I’m introducing a simple yet already working full Java language identifier. In natural language processing, there are tons of cases where you need to identify the language of a document first. While you’ll find most implementations in Perl, Java seems to be lacking in this domain.

JLangDetect : a Java language detector

Introduction

JLangDetect is a pure Java implementation of a language detector. It provides a toolkit for training language recognition, and a simple implementation of a detector.

How to use it

To identify the language of a text, just pass in a string to the sensor. It will respond a language code :

String lang = detector.detectLang(text, false);

A detector is created this way :

LangDetector detector = new LangDetector();
detector.register("fr", frenchTree);
detector.register("en", englishTree);

How it works ?

JLangDetect is based on n-gram tokenization. Basically, texts are tokenized with different token sizes. For example, given the text ``cat'', n-gram tokenization for 1 to 3 token sizes will produce the following tokens :

The idea is to tokenize a large set of documents in a given language and record token statistics. When you need to identify a language, then you’ll tokenize it the same way, and you’ll be able to score the input string against several token stats.

A gram-tree

For now, JLangDetect stores token statistics in a memory-based structure called a gram tree. A gram tree will record the number of times a given n-gram is found in the document library (called corpus).

There are ways to obtain a more compact representation, that will be likely found in future releases, but this one works like a charm. I’ve made it Serializable so that storing/reading from file system pre-compiled gram trees is easy.

Testing JLangDetect

Choosing a corpus

For my own tests, I’ve searched a corpus which would satisfy the best conditions for the JLangDetect algorithm. It includes :

the fact that the corpus size should be rather the same for every language
if not, the characters used in the corpus should be rather different (for example, it’s easy to discriminate between french and russian, because they use different characters, however, if you want to differenciate french and english, you need more accurate n-gram statistics).

A ``parallel corpus'' is a good candidate for that : it’s a corpus for which the text is simply the same, translated in different languages. I’ve made my test using the European Parliament Proceedings Parallel Corpus. It covers the proceedings of the european parliament from 1996 to 2006, which is up to 40 million words per language.

Translations exists for the following languages :

danish (da)
german (de)
greek (el)
english (en)
spanish (es)
finnish (fi)
french (fr)
italian (it)
dutch (nl)
portuguese (pt)
swedish (sv)

You’ll find a precompiled version of this corpus at the bottom of this page, if you wish to use it with JLangDetect. On my computer with a quad-core processor (Q9450), it took less than 10 minutes to process the whole corpus, I’ve optimized the importer for multi-core systems.

Accuracy

JLangDetect, with the previous corpus, does really well with both short and long texts. However, it will do better with longer texts (if anyone can tell me in which language is written the word ``chocolate''…).

Here’s a simple output of JLangDetect. OK means language has been detected properly. Error means not.

langof("un texte en français") = fr : OK
langof("a text in english") = en : OK
langof("un texto en español") = es : OK
langof("un texte un peu plus long en français") = fr : OK
langof("a text a little longer in english") = en : OK
langof("a little longer text in english") = en : OK
langof("un texto un poco mas longo en español") = es : OK
langof("J'aime les bisounours !") = fr : OK
langof("Bienvenue à Montmartre !") = fr : OK
langof("Welcome to London !") = en : OK
langof("un piccolo testo in italiano") = it : OK
langof("een kleine Nederlandse tekst") = nl : OK
langof("Matching sur des lexiques") = fr : OK
langof("Matching on lexicons") = en : OK
langof("Une première optimisation consiste à ne tester que les sous-chaînes de taille compatibles avec le lexique.") = fr : OK
langof("A otimização é a primeira prova de que não sub-canais compatível com o tamanho do léxico.") = pt : OK
langof("Ensimmäinen optimointi ei pidä testata, että osa-kanavien kanssa koko sanakirja.") = fi : OK
langof("chocolate") = es : Error
langof("some chocolate") = it : Error
langof("eating some chocolate") = en : OK

Update Chocolate is an error as I tagged it as english in my test case, but one could expect it to be spanish. It’s here to demonstrate the limits of the system for very short texts. As for ``longo'', long time I’ve not written spanish. Bisounours is a french joke ;) Feel free to comment ;)

Downloads

JLangDetect is licensed under Apache 2.0.

Binary : jlangdetect-0.1.jar
Source: jlangdetect-0.1-sources.jar
Javadoc: jlangdetect-0.1-javadoc.jar
Europarl pre-compiled corpus: ngrams-europarl.zip

Java 7 idea : change default file encoding

Tue, 16 Sep 2008 00:00:00 +0200

Apart from closures I eventually doubt will be integrated into Java 7, there’s something I’d really like to see, althought it introduces a (small yet easily fixable) change : I think the default source file encoding should be the same for all platforms. Thumbs up for UTF-8, since it is able to represent most of characters for every language (for exotic characters, you still may use \uxxxx notation).

I still can’t understand (but there must be valid reasons apart from people creating files with notepad) why the default source file encoding is not the same on every platform. The biggest problem is when using multiple development environments on the same project : people using Windows create source files using cp1252 while people like me, programming under Linux, are using utf-8.

The problem occurs when people introduce special characters (very common in french, for example in string constants or comments) in source code : you won’t be able to build under another platform unless you specify the source file encoding flag, which is rather annoying.

The problem can be obviously fixed with your favorite editor (IntelliJ IDEA allows choosing the default file encoding), but I’d really like it to be fixed in the compiler, so that the builds are consistent accross multiple platforms (plus, if you use continuous integration, there are chances that your build server is not using the encoding you expect).

The strange Hibernate version numbers

Thu, 21 Aug 2008 00:00:00 +0200

I am fascinated with Hibernate version numbers. Have you noticed how uncommon they are ? In the open source community, almost every project follows the same rules :

project-MAJOR.minor.bugfix

For example :

commons-io-1.2.3

spring-core-2.0.0

If necessary, you may append release specific modifiers :

alpha version : -alpha1, -alpha2, …
beta version : -beta1, -beta2, …
release candidate : -rc1, …

Hibernate does not follow those conventions. You may find :

hibernate-3.2.0.cr1 : why not hibernate-3.2.0-rc1 ?
hibernate-3.3.0.ga : why not hibernate-3.3.0 ?
hibernate-3.3.0.SP1 : why not hibernate-3.3.1 ?
hibernate-search-3.1.0.Beta1 : why not hibernate-search-3.1.0-beta1 ?
…

If you are used to Maven, then you must have been annoyed with it, as the resulting artifacts never follow the same conventions (with or without dash, uppercase or not, …).

Pretty frustrating. As a widely used library, I would expect Hibernate to follow the most common versioning schemes…

XPointer support in JDK is incomplete

Wed, 16 Jul 2008 00:00:00 +0200

There are some days when you want to kill people who don’t document their code, and some where you just want to kill those who silently ignore specifications. Today, I’ve spent hours digging into the source code of the Sun JDK (1.5) in order to find out why one of my XML inclusions did not work. Worse, there wasn’t a single warning, nor a single error in the console…

The story

I’m using XInclude to split long XML files into smaller parts. Basically, it allows you to write things like this :

--

You may activate this feature using the following lines of configuration :

code,prettyprint

[source]

--

(note the *xpointer=``xpointer(//macros/*)''* syntax).

Well, if you came to this page thinking you’d find a solution to the fact that this does not work, I can’t help you. It doesn’t. While the documentation says that _XInclude_ support is implemented in the JDK, the truth is that it is incomplete. The JDK just supports the https://www.w3.org/TR/xinclude/#XPElement[element scheme] but not the https://www.w3.org/TR/xinclude/#XPointer[xpointer scheme]. Well, the specification of XInclude says that the implementation of the second one is optional, but the main problem with the JDK is that is never throws an _UnsupportedOperationException_. Worse, it silently parses your includes, and never throws any error (hence, the xi:fallback element is ignored).

[[]]
XPointer scheme support ?

If you know about a Java library which supports the xpointer scheme, please let me know, I haven’t found any yet.

gasp Dell Optiplex 740 bios upgrade failure

Fri, 6 Jun 2008 00:00:00 +0200

This morning, I wanted to upgrade my BIOS to check out if the changes made to the nforce chipset support were to improve disk performance on my Ubuntu box… First, the bad thing is that you need to boot Windows to be able to upgrade. Fortunately, I had a (very) small Windows XP partition for that kind of things. So I launched the upgrade and gasp, it kept spawning messages like this :

FAILURE !! FAILURE !!
Click ok to try to reflash BIOS

Ok, BIOS was cleared, no new BIOS installed… After fighting up several minutes, I managed to get the upgrade complete. For those who would experience the same mess, here’s the trick : increase the priority of the BIOS upgrade process to ``higher than normal''. Back to Ubuntu ;)

Simplifiying java.util.concurrent with Groovy

Wed, 2 Apr 2008 00:00:00 +0200

Ever wondered how to simplify java.util.concurrent usage with Groovy ? Here’s a simple example of what can be easily done using closures. The following code executes several closures in parallel and waits upon completion before resuming the script.

def doInParallel(Closure... closures) {
 def service = java.util.concurrent.Executors.newFixedThreadPool(closures.length)
 def latch = new java.util.concurrent.CountDownLatch(closures.length)
 closures.each { cl -> service.execute({
  try {
   cl.call()
  } catch (e) {
   throw e
  } finally {
   latch.countDown()
  }
 })}
 latch.await()
 service.shutdown()
}

Here’s how to use it :

doInParallel(
   { println 1+1 },
   { callVeryLongProcess() },
   { throw Exception() } // demonstrates the correct handling of exceptions
)

println "This line of code will only appear after every closure in doInParallel is completed"

Fun, isn’t it ?

Moving Lucene a step forward

Thu, 20 Mar 2008 00:00:00 +0100

Introduction

At Lingway, we’ve been using Lucene for a few years now. For those who are new to Lucene, here’s its bottomline : Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java..

Before criticizing, I must admit that Lucene is a very good high-performance full text search engine. For years, Lucene has been considered as a first class citizen when looking for an embeddable search engine written in Java. Its reputation has grown fast, and it is still now the best open source Java search engine available. There’s nothing to say about that : Doug Cutting has done a great job. However, it’s development has been going very slow those late months, and I think Lucene will most likely not keep in touch with today’s document processing needs. Don’t mess up : I am no search engine developer, I am a developer which leverages search engines in order to provide high relevance information retrieval technologies.

This post is about why Lucene may not be the best choice for future developments if nothing is done, and why the situation may not be close to change. In our situation, we push Lucene to its limits, although we make it work quite good. It’s a reason why we made some suggestions and submitted a patch to Lucene (which does not cover everything listed here) : Lingway uses semantics to generate complex queries where proximity matters. For example, if you are looking for documents on conflicts in middle east, you’ll probably also want to find documents talking about war in Iraq. In that case, war and Iraq are called expansions of conflict and middle east respectively. We provide a technology which analyzes your query in order to deduce the most relevant expansions, and generate queries for them. Yet, in order to get relevant results, this is insufficient : Google-like ranking or term frequency scoring like implemented in Lucene do not suit semantic scoring needs. For example, a document which contains both middle and east terms but separated by more than 1 word are most likely not what you want to find. Moreover, we should attribute lower scores on expansions than on the regular words. For example, we’ll give a better score to conflict in middle east phrase than in war in Iraq.

At Lingway, we think this kind of document retrieval technology is the future of search engines. Google is good at finding documents, but what we want is to find the most relevant ones. However, most (if not all) of current search engines have not been thought to perform such complex queries… Lucene is used by wikipedia, and you’ll notice that if you try to find more than a single word, most results are irrevelant…

Here’s a capture of the upcoming Lingway KM 3.7 interface, which demonstrates the requirements. Here, we write a query in french, which is used to find documents in english talking about the same subject. Note that this is more than plain translation, we call it cross language mode :

Note the matches in green : chanteur becomes singer, but we also find matches about singing. Same for pop which expands to blues… Now for the technical part:

6 ways of improving Lucene

\6. No built-in support for clustering. If you want to create clusters, either write your own implementation of a Directory, or use Solr, or Nutch+https://hadoop.apache.org/[Hadoop]. Both Solr and Nutch leverage Lucene, but are not straight replacements. Lucene is embeddable, while you must leverage on Solr or Nutch. Well, I think it is not very surprising that Hadoop idea emerged from the Lucene team : Lucene doesn’t scale out. It’s internals makes it (very) fast for most common situations, but for large document sets, you have to scale out, and as Lucene does not implement clustering at the core level, you must switch from Lucene to another search engine layer, which is not straightforward. The problem with switching to Solr or Nutch is that you carry things you probably won’t need : integrated crawling for Nutch and indexing server for Solr.

\5. Span queries are slow. This may be interpreted as a problem specific to Lingway where we make intensive use of span queries (NEAR operator : ``red NEAR car''), but the Lucene index structure has been updated to add this feature, which was not thought at first. The underlying implementation leads to complex algorithms that are very slow, especially when some term is repeated many times in a single large document. That’s why I tend to say that Lucene is a high-performance text search engine only if you use basic boolean queries.

\4. Scoring is not really pluggable. Lucene has its own implementation of a scoring algorithm, where terms may be boosted, and makes use of a Similarity class, but soon shows limitations when you want to perform complex scoring, for example based on actual matches and meta data about the query. If you want to do so, you’ll have to extend the Lucene query classes. The facts is that Lucene has been thought in terms of tf/idf like scoring algorithms, while in our situation, for linguistic based scoring, the structure of Lucene scoring facilities don’t fit. We’ve been obliged to override every Lucene query class in order to add support for our custom scoring. And that was a problem :

\3. Lucene is not well designed. As a software architect, I would tend to make this reason 1. Lucene has a very poor OO design. Basically, you have packages, classes, but almost no design pattern usage. I always makes me think about an application written by C(++) developers who discover Java and carry bad practices among them. This is a serious point : whenever you have to customize Lucene to suit your needs (and you will have to do so), you’ll have to face the problem. Here are some examples :

Almost no use of interfaces. Query classes (for example BooleanQuery, SpanQuery, TermQuery…) are all subclasses of an abstract class. If you want to add a feature to those classes, you’ll basically want to write an interface which describes the contract for your extension, but as the abstract Query class does not implement an interface, you’ll have to constantly cast your custom query objects to a Query in order to be able to use your objects in native Lucene calls. Tons of examples like this (HitCollector, …). This is also a problem when you wan’t to use AOP and auto-proxying.
Unnatural iterator implementations. No hasNext() method, next() returns a boolean and refreshes the object context. This is a pain when you want to keep track of iterated elements. I assume this have been done intentionally to reduce memory footprint but it makes once again algorithms both unclear and complex.

\2. A closed API which makes extending Lucene a pain. In Lucene world, it is called a feature. The policy is to open classes when some user needs to gain access to some feature. This leads to an API where most classes are package protected, which means you won’t ever be able to extend it (unless you create your class in the same package, which is quite dirty for custom code) or you’ll have to copy and rewrite the code. Moreover, and it is quite related to the previous point, there’s a serious lack of OO design here too : some classes which should have been inner are not, and anonymous classes are used for complex operations where you would typically need to override their behaviour. The reason invoked for closing the API is that the code has to be cleaned up and made stable before releasing it publicly. While the idea is honourable, once again it is a pain because if you have some code that does not fit in the mainstream Lucene idea, you’ll have to constantly backport Lucene improvements to your version until your patch is accepted. However, as the developers want to limit API changes as long as possible, there’s little chance that your patch will ever be commited. Add some final modifiers on either classes or methods and you’ll face the problem. I don’t think the Spring framework would have come so popular if the code had been so locked…

\1. Lucene search algorithms are not adapted to grid computing. Lucene has been written when hardware did not have memory that much and multicore processors didn’t exist. Therefore, the index structure has been thought and implemented in order to perform fast linear searches with a very little memory footprint. I’ve personally spent lots of hours trying to rewrite span query algorithms so that I could make use of a multithreaded context (for dual/quad core cpus), but the iterator-based directory reading algorithms make it hardly impossible to do so. In some rare cases you’ll be able to optimize things and iterate the index in parallel, but in most situations it will be impossible. In our case, when we have a very complex query with 50+ embedded span queries, the CPU is almost not used while the system is constantly calling I/Os, even using a RAMDirectory.

Any alternative ?

I think the last point is the more problematic : Lucene reaches its limits when it goes to searching large datasets (with many operators or not) on modern hardware. That’s why I’ve been looking for an alternative to Lucene. After reading blog entries and a discussion about Wikia, I found that there were not so many alternatives. However, I finally came to a very promising solution : MG4J. It has a very good object design, excellent performance on search (indexing is slower than Lucene), a small memory footprint, is up to 10x faster than Lucene on my span query benchmarks, and is nativelly designed for clustering. It also has built-in support for payloads, while in Lucene it is a very recent addition which is still experimental. However, MG4J still misses some features such as easy incremental indexation (indices ARE clusters, but there’s no idea on performance issues), document removal and an easier to use indexing process. What made me happy is that I was able to reproduce the customizations I made on Lucene in a few hours where it took me days on Lucene.

I think there’s room for a new open source search engine which is not thought in terms of a single computer indexing a collection of documents with limited memory, but in terms of transparent distributed indexation and searching in order to provide fast answers on large datasets (think of Terracotta or GridGain as repartition frameworks) : you should not have to leverage an application to gain access to clustering features. Lucene has an excellent implementation of the first category of search engines, but I think this is not adapted to what we require now : beeing able to find the best answer to a question in a reasonable amount of time. tf/idf based search algorithms and Google rank are not the future of search engines. Finding the most relevant answers implies complex queries involving metadata on documents and semantics, which is basically what Lingway does (with Lucene and other backing search engines), but it requires more power and an underlying technology which supports modern hardware.

A good reason to choose Lucene

Whatever the reproaches I have to make about Lucene, it is still the best java open source solution available for what we are doing, and it performs quite good ;-)

Tip: Find try/catch performance bottlenecks with structural search in IntelliJ IDEA

Fri, 29 Feb 2008 00:00:00 +0100

Advanced Java programmers are aware that try/catch statements cost many CPU cycles, so it is generally a bad idea to catch exceptions in loop statements. Moreover, even with profilers, it may be hard to find out that a try/catch statement is responsible for a performance bottleneck : you have to take a look at your code, and figure out whether it should be optimized or not. In many cases, you should definitely avoid using try/catch statements in loops.

Fortunately, IntelliJ IDEA has this powerful structural search feature. Let’s explain how to use it in order to find those possible bottlenecks. Structural search is powerful as it allows you to describe ``patterns'' of code to match. No matter the indentation, no matter the statements, you’ll just lookup for structural matches.

In our case, you need to look for something that ``looks like'' this :

for () {
  try {
  } catch () {
  }
}

The corresponding structural search pattern will be :

for ($decl$;$condition$;$increment$) {
 $beforeStatements$;
 try {
  $TryStatement$;
 } catch ($exceptionType$ $ex$) {
  $CatchStatement$;
 }
 $afterStatements$;
}

To use this, either click on the ``search structurally'' item in the search menu, or press Ctrl+Shift+S and enter the pattern :

+
+
Then you need to tweak the variables in order to specify the cardinalities. Click on ``Edit variables'', and update :

beforeStatements : minimum count 0, maximum count unlimited
afterStatements : minimum count 0, maximum count unlimited
tryStatement : minimum count 1, maximum count unlimited
catchStatement : minimum count 0, maximum count unlimited

The other variables may stay as defaults (min 1, max 1).

Now click on ``Find'' and you’ll probably find some of those nasty constructs. The previous patterns works for classical for loops, you need to tune it to work for Java 5 for each constructs, and don’t forget while loops too.

Java BGGA closures proposal and the Groovy syntax

Fri, 29 Feb 2008 00:00:00 +0100

While there’s still no decision made on the integration of closure to the Java language (AFAIK), I think the BGGA proposal is the most promising candidate. I’m quite confident it will eventually be integrated to the language.

However, I was wondering about the syntax of the proposal, mainly because more and more programmers are using closures in Groovy. Basically, you’ll write a closure in Groovy like this :

{ x,y -> x+y }

In the BGGA proposal, the syntax is more verbose because Java is strongly typed :

{int x, int y => x + y}

Everything would be fine, but I was wondering about the ⇒ separator, which can be confusing :

{int x, int y => x >= y }

IMHO, it would be better if the proposal followed the syntax that’s getting known by more and more people, that is to say the Groovy one : replacing ⇒ with →

What do you think ?

Groovy, functional programming and the curry method

Fri, 15 Feb 2008 00:00:00 +0100

I’m sure you have heard of functional programming, at least when you were sill a student (if you are, then you’re probably digging into it). The Groovy language, built on top of Java, provides means of dealing with functional programming. Moreover, it offers a curry method which allows powerful recursion mechanisms.

For example, in maths, you’ll want to define the sum function of a list as the sum of the first element and the (recursive) sum of the rest of the list. Going further, you’ll want to define a meta function which defines the behaviour of the sum method :

f(x1, x2, ...) = f(x1, f(x2, ...))

That’s currying. Here’s a basic example which demonstrates this feature in Groovy. First, let’s define a reduce closure which takes as the first argument the function to be applied on the list of arguments :

def reduce
reduce = { fn, params ->
    if (params.size()==1) return params[0]
 fn(params[0], reduce(fn,params[1..

Note that for the reduce function to be known inside the closure, you have to define it before (click here for details).

Then we may define the sum() function as this :

def sum = reduce.curry({ x,y -> x+y })

The sum method is a closure passed as a parameter to the curry method. Technically, it is equivalent to write a function where you replace the first parameter of the reduce closure with the sum closure. Now let’s see the result :

println sum((1..10))

--> 55

More interesting, as the parameter types are not specified, you may also use our sum function as a string or list concatenation :

println sum(("a".."z"))

--> abcdefghijklmnopqrstuvwxyz

println sum([(1..10),(11..20)])

--> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

I love Groovy ;)

Addendum: You should also use this with caution, as it costs much in invocation time, and you should not use it for so simple functions (summing is a very bad example as the cost in time of the sum operation is much less that the method invocation cost). Furthermore, don’t forget that it is recursions, so too many parameters will drive you too a stack overflow error ;-)

Apache CXF + Maven + Javamail + Log4J (update)

Thu, 14 Feb 2008 00:00:00 +0100

I had much trouble trying to figure out why a project we moved to Apache CXF (for managing web services) suddenly started to send malformed emails. Basically, we use Commons email for sending emails to the administrator whenever a monitored service goes wrong.

Everything went good until we moved our web service layer to the powerful Apache CXF. Instead of sending a well formed e-mail, the server suddenly started sending emails without subject nor correct sender. As we use Maven for building our project, and CXF makes use of Apache Geronimo Javamail/activation implementations, I immediatly suspected a buggy implementation of javamail.

Instead of sending, in the DATA section of the email the following :

DATA
From: "I am the sender"

Geronimo javamail implementation only sends what was actually set as the mail content :

DATA
the content of my mail

I don’t know if it is a feature or a bug, but it really sucks (update : it’s neither feature nor bug, it’s a classpath messup, see the latest paragraph). To get rid of this mess, we had to exclude the geronimo dependencies from both cxf-rt-core and cxf-rt-databinding-jaxb modules. And, if like us you have a dependency on another module using CXF too, don’t forget to exclude from the parent module too.

Basically, you shoud write something like this in your pom.xml file :

   org.apache.cxf
   cxf-rt-core
   ${cxf.version}


     org.apache.geronimo.specs
     geronimo-javamail_1.4_spec


     org.apache.geronimo.specs
     geronimo-activation_1.1_spec




   org.apache.cxf
   cxf-rt-frontend-jaxws
   ${cxf.version}


   org.apache.cxf
   cxf-rt-databinding-jaxb
   ${cxf.version}


     org.apache.geronimo.specs
     geronimo-javamail_1.4_spec


     org.apache.geronimo.specs
     geronimo-activation_1.1_spec

Then, add the Sun javamail/activation implementations instead :

   javax.mail
   mail
   1.4

To the CXF team : please make those dependencies optional.

To the Maven team : please add the global exclude feature so that we won’t have to specify excludes on each dependency !

Update: After filing a bug report on Geronimo, I’ve made further investigations. The problem is more complex than I first thought : if there were only the Geronimo dependencies, I would not have had the same error. Instead, I would have got a message telling me that it could not find the ``SMTP transport''. I figured out that the problem was our log4j dependency, which depends on javax.mail. The Geronimo Spec mail API does not implement the SMTP transport : it is implemented in a separated artifact. So the SMTP transport that was used in my case what the one in the Javamail transitive dependency from log4j. The problem that seems to arise is that Geronimo is not compatible with the transport layer from Javamail : if you just order the classpath differently, all goes fine. It all depends on the library which is first loaded…

Alternative to delegate pattern with cglib

Sat, 9 Feb 2008 00:00:00 +0100

This week I had some third party class I needed to override, but as in many cases, I had to use the delegate design pattern in order to be able to expand the class without modifying the 3rd party library.

However, I came to a problem when the base class used the visitor pattern. The following snippet explains it all:

public class MyBaseClass implements AnInterface {

...

public void accept(Visitor visitor) {
   visitor.visit(this);
}

...

The problem is that if you delegate the doSomething() method, the object which will be passed to the visitor will be the delegate, not your implementation :

public class DelegatingClass implements AnInterface,AnotherInterface {
...
public void visit(Visitor visitor) {
   delegate.visit(visitor); // ok, assume you can't do visitor.visit(this), that would be too easy. Imagine the delegate does more stuff or imagine another method like doSomething() { something.call(this); }
}
}

This leads to situations where the visited object is not the one you’d like to, and it breaks your algorithms where you assume the visited object is your implementation (which adds several features). Futhermore, there are situations when you can’t even delegate. For examples, when extending a class with final methods, or when you try to expand a class which is package protected (there are tons of examples like this in Apache Lucene). I’ve come to a solution using the cglib library. The idea is to create a proxy and dynamically extend the base class with your features. I’ve written a simple utility class which does it all :

package com.lingway.cglib;

import net.sf.cglib.proxy.Enhancer;
import net.sf.cglib.proxy.MethodInterceptor;
import net.sf.cglib.proxy.MethodProxy;

import java.lang.reflect.Method;

/**
 * This utility class generates cglib proxies which delegates missing methods to instanciated
 * objects.
 * User: Cedric
 * Date: 9 févr. 2008
 * Time: 08:35:20
 */
public class AdvancedEnhancer {
    /**
     * Creates a new instance of a delegating object.
     * @param aSuperClass super class of the generated object.
     * @param someInterfaces interfaces generated object will implement.
     * @param delegates delegates which implements the "missing" methods.
     * @return Object</> delegating object
     */
    public static Object enhanceWithInterfaces(
            Class aSuperClass,
            Class[] someInterfaces,
            Object[] delegates) {

        Enhancer enhancer = new Enhancer();
        enhancer.setSuperclass(aSuperClass);
        enhancer.setInterfaces(someInterfaces);
        enhancer.setCallback(new InterfaceMethodInterceptor(delegates));
        return enhancer.create();
    }

    /**
     * A convenient creator which assumes a single interface to be implemented
     * by a list of delegates. This allows removing explicit class casting.
     * @param aSuperClass super class of the generated object.
     * @param aTargetInterface target interface
     * @param delegates delegates which implements the "missing" methods.
     * @return T</> delegating object implementing the T interface
     */
    @SuppressWarnings("unchecked")
    public static

And here’s an example of how it works. Pretty easy !

package com.lingway.cglib;

/**
 * A proof of concept of the "cglib delegate pattern"
 * User: Cedric
 * Date: 9 févr. 2008
 * Time: 14:06:56
 */
public class ProofOfConcept {
    public interface ISelf {
        ISelf self(); // basically, returns "this", which is just for the POC :)
    }

    public interface IEcho {
        void echo();
    }

    public interface IMixin extends ISelf,IEcho {}

    public static class Self implements ISelf {
        public ISelf self() {
            return this;
        }
    }

    public static class Echo implements IEcho {
        public void echo() {
            System.out.println("echo !");
        }
    }

    public static void main(String[] args) {
        IEcho echo = new Echo(); // these are the "additional features"
        IMixin mixin = AdvancedEnhancer.enhanceWithTargetInterface(Self.class, IMixin.class, echo);
        System.out.println("mixin.self (shows this is the proxy) = " + mixin.self());
        mixin.echo();
    }
}

In the previous example, I ``enhance'' the Self class with the features of the Echo class. This is really easy, and perfectly solves our problem. Note that technically, the delegate and the delegator are switched.

Enjoy !

The facts about annotations inheritance

Fri, 18 Jan 2008 00:00:00 +0100

Following my post about the behaviour of annotations in an inheritance model, Mike Kaufman made a very interesting comment. The truth, as I said, is that annotations are technically inherited only if the annotation definition itself is annotated with @Inherited.

I must admit that while this is technically true, the facts are not so simple. Let me explain why I came to this test case. I had written a web service using the Metro (JAX-WS RI) project. With project Metro, web services are defined with annotations. For example, you would write :

@WebService(serviceName="myService")
public class MyService {
    @WebMethod
    public int sum(int x, int y) { return x+y; }
}

This is a very convenient (and easy) way of building web services. But services should always be defined through interfaces. Therefore, here’s I would like to do :

public interface IMyService {
   void sum(int x, int y);
}

..

@WebService(serviceName="myService")
public class MyService implements IMyService {
...
}

That’s a good starting point, but I like to say that the MyService class is nothing but a default implementation of my service. Moreover, annotating a class leads to readability problems when you mix several annotation frameworks. So I wanted to be able to annotate the interface.

@WebService
public interface IMyService {
  @WebMethod void sum(@WebParam(name="x") int x, @WebParam(name="y") int y);
}

Then implement like this :

@WebService(serviceName="myService")
public class MyService implements IMyService {
   public int sum(int x, int y) { return x+y; }
}

This has multiple advantages :

separation of interface and implementation (should always be done)
the interface may be shared for both server endpoint and client stubs

Note that I must copy down the @WebService annotation just to specify the service name (as multiple services may implement the same interface). There, you should already see a problem, but let’s continue : Metro does not allow this. You cannot annotate interfaces and hope subclasses will be ``annotated''. Bas point, but it’s rather logic :

package javax.jws;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target( { ElementType.TYPE })
public @interface WebService
{
    String name() default "";

    String targetNamespace() default "";

    String serviceName() default "";

    String wsdlLocation() default "";

    String endpointInterface() default "";

    String portName() default "";
};

The @WebService annotation is not itself annotated with @Inherited. Then, some weeks later, I moved to Apache CXF. To my surprise, although CXF uses the very same JAX-WS annotations, those annotations can be set on interfaces rather than classes ! How could it be ? Are annotations inherited or not ? My previous post gave you a technical yet simple answer : in that particular case, annotations are not inherited.

That’s where Mike’s comment is interesting : it is a very complete explanation on why different frameworks did build their own annotations inheritance rules. To be clear, if an annotation has its retention policy set to runtime and is annotated with @Inherited, then the subclasses with have this annotation to their annotations list too. But this does not prevent a framework from building its own inheritance policy.

In CXF, when you declare a class as a webservice, CXF looks for superclasses and interfaces to get those annotations. This is a framework dedicated annotation inheritance rule.

With this second example, here you see the problem : although the technical truth is that annotations are not inherited by default, the facts are that frameworks define their own rules… Thanks Mike for giving me a great transition ;)

The truth about annotations inheritance

Thu, 17 Jan 2008 00:00:00 +0100

I’ve read tons of articles about the subject : are annotations inherited or not ? Java 5 annotations are powerful, but you must understand what you do. Annotating a class does not necessarily mean that a subclass will keep the annotation. So, are they inherited or not ? The truth is that if you want your annotations to be inherited, you must specify it. Here’s a simple JUnit 4.4 test case to show you the concept.

package test.annotations;

import org.junit.Test;
import static org.junit.Assert.assertEquals;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Annotation;
import java.lang.annotation.Inherited;

public class AnnotationTest {

 @Retention(RetentionPolicy.RUNTIME)
 public @interface SimpleAnnotation {
  String value() default "test";
 }

 @Retention(RetentionPolicy.RUNTIME)
 @Inherited
 public @interface InheritedAnnotation {
  String value() default "test";
 }

 // two simple classes that will show that the annotation is *not* inherited
 @SimpleAnnotation
 class BaseClass {
 }

 class SubclassDoesNotInheritAnnotation extends BaseClass {
 }

 // two simple classes that will show that the annotation *is* inherited
 @InheritedAnnotation
 class InheritedBase {
 }

 class ChildInheritsAnnotation extends InheritedBase {
 }

 // proof
 @Test
 public void testAnnotations() {
  // not inherited
  Annotation[] annotations = BaseClass.class.getAnnotations();
  assertEquals(1, annotations.length);
  annotations = SubclassDoesNotInheritAnnotation.class.getAnnotations();
  assertEquals(0, annotations.length);

  // inherited
  annotations = InheritedBase.class.getAnnotations();
  assertEquals(1, annotations.length);
  annotations = ChildInheritsAnnotation.class.getAnnotations();
  assertEquals(1, annotations.length);
 }

}

The secret is the @Inherited annotation, which will ensure that subclasses will be annotated too.

IntelliJ IDEA’s local history saved my life !

Thu, 17 Jan 2008 00:00:00 +0100

One of my coworkers came to me quite stressed after having accidently'' deleted source files from his project. The problem was that he had not done it from our editor, IntelliJ IDEA, but rather directly via windows explorer. Furthermore, he had also pressed shift+delete'', which, for those who are not aware of that combination, means ``delete without possibility to recover''.

Damned. Not commit for the late two days (!), Windows XP, NTFS, shift+delete. My first thought was that he was screwed. Then we thought about the local history feature of Jetbrain’s editor. If there was a place where he could recover his files, there it was.

So we went to the View → Recent changes'' menu, and IDEA showed us a popup showing external changes''. We clicked and thanks Jetbrains, the files were marked as deleted. We just had to select them, click on ``revert'' and voilà !

Ok, Jetbrains did not saved my life. They saved his life ;)

P.S : I’m using Ubuntu Linux every day, and I have to admit the problem would have been the same : that’s not a windows related problem :)

Separating configuration from deployment in web applications

Wed, 26 Dec 2007 00:00:00 +0100

Since I write web applications, I’ve always been struggling with a very basic problem. Basically, I refuse to write, and tell my coworkers too, applications which require a compilation for each customer. This seems common sense, but it is really critical when building applications which require a high level of customization. The second problem is about upgrading applications.

The weakness of web applications containers

Take Tomcat. Anyone who has ever deployed a .war file knows that: deploying a war erases everything in the exploded directory. This means that if you have made customizations in files located in WEB-INF directory (for example to configure your database), upgrading the application will make you loose all your configuration data. This has two implications:

for the developer, this requires writing down a burn-proof upgrade guide which mostly consists of saying ``backup all your data and do what you’ve done for the first version another time''.
for the customer, this leads to situations when you prefer not to upgrade because you fear the upgrade procedure : you cannot allow breaking your system

I’ve personally seen both situations : as a developer, for my applications, and as a customer (upgrading Jira/Confluence), and lead me to find an alternative solution : allowing separation of configuration from binaries in a web environment in a system independent way. Basically, the application should work out of the box, and you should never have to rewrite configuration files again.

Classloader magics

Using Java, I’ve not found any cleaner way of reading configuration/resources any easier than using the classloader : it is an elegant system independent way of retrieving data. For example, you could get your configuration file this way :

public static final String PROPERTIES_FILE="/CONFIG/configuration.properties";

// ...

InputStream in = this.getClass().getResourceAsStream(PROPERTIES_FILE);
Properties props = new Properties();
try {
   props.load(in);
} catch (IOException e) {
   theLogger.warn("Unable to read configuration file " + PROPERTIES_FILE, e);
}

This assumes that a file called configuration.properties is accessible in the CONFIG directory at the root level in the classpath. I think web environments miss some top level classpath directory convention like my CONFIG directory. We have the WEB-INF and META-INF directories, but both are bundled with the application binaries(war file). I’d like to have, someday, a JSR which dynamically adds a CONFIG directory in the application classpath. Application servers would be encouraged to separate this directory from the exploded one. For example, in a Tomcat environment, it would lead to the following directory structure :

tomcat/
tomcat/webapps
tomcat/webapps/my-app
tomcat/webapps-conf/
tomcat/webapps-conf/my-app

The usual tricks that you do in the WEB-INF/classes directory would, therefore, be done in the webapps-conf/my-app directory. As this feature/standard is missing, I’ve been obliged to find a workaround using the Spring framework (but this could be reproduced in any web application).

An implementation of configuration separation using Spring

Concept

Basically, I have implemented a fallback mechanism which allows fine grained configuration data finding procedure. When the application is bootstraping, it will look for its configuration files. If nothing is found, it will create a configuration directory and copy the default configuration files into it, then continue the boostrap using those files. I’ve chosen the following fallback mechanism (feel free to code your own), which targets at locating the application home directory:

if a properties file is accessible in the classpath, the application will look for a property in it which contains the application home directory
if a system property is set, then the application will read its home directory from it
the application creates its home directory in the user home (that is the user which actually runs the application server

This mechanism allows multiple configurations : if you want to deploy your web application more than once on a single server, choose the first solution (agreed you’ll have to write one file in the WEB-INF/classes directory). If you don’t need it, just export a system property before running the web application server. And finally, the simplest case is the last one, but in this case I strongly recommand that your web application user home directory is separated from the server binaries too. For example, under Linux, I usually run my Tomcat application server with its own user/group) (tomcat), but the tomcat user home must not be the tomcat home. You should have something like this : /opt/tomcat : the application server /home/tomcat : the tomcat user home directory Doing this, upgrading the application server itself is easier, and you’ll never erase your configuration data. By the way, the solution I propose also allows you to share configuration across multiple web applications.

Spring implementation

The Spring implementation of this lookup algorithm makes use of the Spring property placeholder concept: a property placeholder allows Spring to define properties programmatically which can be used in its configuration files. For example, you’ll have the following XML code :

In the previous snippet, _$\{myproperty}_ is a property. The idea is to read all those properties from a custom property placeholder configurer and use them into our Spring configuration file. The good thing is that you can actually *use those properties for importing other Spring configuration files*, for example:

code,prettyprint---- code,prettyprint




...

And the application configurer magic. In this case, we create two Spring configuration files, one called applications.xml and the other named generated.applications.xml. Both are real Spring configuration files :

code,prettyprint

 code,prettyprint
public class ApplicationConfigurer extends PropertyPlaceholderConfigurer {
 private final static Logger theLogger = Logger.getLogger(ApplicationConfigurer.class);

 public static final String APPLICATION_HOME_PROPERTY = "application.home";
        public static final String CUSTOM_APPLICATIONS_FILE = "applications.xml";
 public static final String GENERATED_APPLICATIONS_FILE = "generated.applications.xml";

 private static final String APPLICATION_HOME = ".mycompany"+File.separatorChar+"myapp";
 private static final String APPLICATION_PROPERTIES_FILE = "/myapp.properties";
 private static final String SAMPLE_APPLICATIONS_FILE = "/com/mycompany/conf/applications.sample.xml";


 public ApplicationConfigurator() {

  // output generic version data
  theLogger.info(Version.getVersion());
  theLogger.info("(c) MyCompany 2007");

  String fultyHome = null;

  // search for application.properties file
  InputStream in = this.getClass().getResourceAsStream(APPLICATION_PROPERTIES_FILE);
  Properties props = new Properties();
  if (in != null) {
   try {
    props.load(in);
    setProperties(props);
    fultyHome = props.getProperty(APPLICATION_HOME_PROPERTY);
   } catch (IOException e) {
    in = null;
    theLogger.warn("Unable to read properties file " + APPLICATION_PROPERTIES_FILE, e);
   }

  }

  if (in == null) { // search for system property
   fultyHome = System.getProperty(APPLICATION_HOME_PROPERTY);
   if (fultyHome == null) { // create default home
    fultyHome = System.getProperty("user.home") + File.separator + APPLICATION_HOME;
   }
   props.put(APPLICATION_HOME_PROPERTY, fultyHome);
   setProperties(props);
  }

  theLogger.info("Using APPLICATION_HOME : " + fultyHome);
  File homeDir = new File(fultyHome);
  createDefaults(homeDir);

 }

 private void createDefaults(File aHome) {
  if (!aHome.exists()) {
   theLogger.info(aHome + " does not exist. Creating default files");
   aHome.mkdirs();
  }
  File appFile = new File(aHome, CUSTOM_APPLICATIONS_FILE);
  if (!appFile.exists()) {
   theLogger.info("Configuration "+ CUSTOM_APPLICATIONS_FILE +" does not exist. Using defaults file.");
   theLogger.info("Please edit " + appFile.getAbsolutePath() + " then restart the application");
   URL url = this.getClass().getResource(SAMPLE_APPLICATIONS_FILE);
   File src = new File(url.getFile());
            copySampleFile(appFile, src);
            copySampleFile(new File(aHome, GENERATED_APPLICATIONS_FILE),src);
        }
 }

    private void copySampleFile(File aAppFile, File aSrc) {
        try {
            FileOutputStream out = new FileOutputStream(aAppFile);
            FileInputStream in = new FileInputStream(aSrc);
            in.getChannel().transferTo(0, aSrc.length(), out.getChannel());
            in.close();
            out.close();
        } catch (IOException e) {
            theLogger.error("Unable to copy default applications file",e);
        }
    }


}
-------------------------------------------------------------------------------------------------------

That’s all ! This class creates a single property called _application.home_ that may be used in your Spring configuration file. As I said before, you could read more that one property and create a whole set of properties to be used in your Spring configuration file, but as I also split my Spring configuration files and the lack of property exports to child contexts, the only property I export is the application home, so that *other beans can programatically create Spring contexts*.

[[]]
Conclusion
----------

This post showed you that you could, at level design, avoid some lacks to the Java web application frameworks which makes it rather complex to separate configuration data from the application itself. This is a low cost implementation which demonstrates that it is not necessarily costly to think about it, and that it is, to my mind, something that *should always be done* because it *simplifies maintenance*.

GFreeMarker 0.2 released

Fri, 21 Dec 2007 00:00:00 +0100

I have released a new version of GFreeMarker, an integration of the FreeMarker template engine in Groovy. This release has no functional change, but has been refactored :

Built made by Maven 2
Compatibility with JDK 1.4
Uses Groovy 1.5

FreeMarker for Groovy

Fri, 3 Aug 2007 00:00:00 +0200

After finding out that Groovy templates would be great as a replacement for Velocity, then thinking about security issues, I came to a third solution.

FreeMarker for Groovy

Some of you told me that I could take a look at FreeMarker, as a practical replacement for Velocity. I went on, but seen nothing that would fit my needs : beeing able to dynamically add tags'' or plugins'' to a customer template. I mean I would not need to rebuild and redeploy an application just to get a simple plugin working. Then I thought I would be able to create a transform for FreeMarker that would make the benefits of the FreeMarker template engine available for Groovy. Basically, it would fit my needs :

Beeing able to dynamically add plugins for a customer
fix does nasty security issues by making the web designer unable to directly write Groovy code (the programmer creates plugins, then makes them available to a customer)

How does it work ?

Imagine your customer requires an URL Encoding text transform, and that FreeMarker does not offer this possibility. Then, you would just need to :

Create a plugin named urlencoder that implements the IGroovyFreeMarkerPlugin interface
Copy this plugin into the Groovy FreeMarker template engine plugins directory

Tell your customer how to use it :

<@groovy plugin="urlencoder">this is an expression that will be converted to its URL Encoding form

The FreeMarker engine will then call your plugin and return the transform. Note that you may embed FreeMarker tags and variables : the transform content does not need to be plain text. For example :

<@groovy plugin="urlencoder">this is an ${expression} that will be converted to its URL Encoding form

Now take a look a the plugin code itself:

import groovy.text.freemarker.IGroovyFreeMarkerPlugin

class urlencoder implements IGroovyFreeMarkerPlugin {
 String transform(Map params, String content) {
  URLEncoder.encode(content);
 }
}

Here’s a sample groovy script that will show you how you can use the template engine :

import groovy.text.freemarker.FreeMarkerTemplateEngine

def tpl = '''
Hello, ${user.name}
<@groovy plugin="urlencoder" mode=user>this is a test ${user.name}'''
def engine = new FreeMarkerTemplateEngine("plugins")
def binding = ["user" : ["name":"cedric"]]
println engine.createTemplate(tpl).make(binding)

Default implementation bonuses

I’ve built a rather simple implementation, but it still has some additional features:

Template caching: instead of using the createTemplate() method, use the createNamedTemplate() one first. It will create a template and put it into the template cache, then use the getNamedTemplate() method to retrieve a cached template.
Plugin loader: there’s a default implementation which reads plugins from a directory. Feel free to implement the IGroovyFreeMarkerPluginLoader if you want more complex loaders
Plugin parameters: if the <@groovy> tag content is not sufficient, you may add parameters to the transform, and use them in the plugin thanks to the params map. For example : <@groovy plugin=temperatureconvertor'' source=celcius'' dest=``kelvin''>112

Grabbing the code

Binary: gfreemarker-0.1.0.jar
Javadoc: gfreemarker-javadoc.zip
Sources (with IntelliJ IDEA project): GroovyFreeMarker.tar.gz

You will also need the latest freemarker.jar. I will not have much time to work on this, so feel free to contribute !

Grails, Spring and templates security issue example

Thu, 2 Aug 2007 00:00:00 +0200

In my previous post, I was thinking about using Groovy SimpleTemplateEngine as a replacement for Velocity. The main idea was to be able to create arbitrary functions for use in the template without having to rebuild my application (and expose a utility class). I’ve been told about taking a look at FreeMarker, but the problem is quite the same : some operations may not be written by a web designer. I’ll take a simple example : for a customer, we needed to be able to take a simple string, and convert it in an URL encoded string. To do that, I had to expose and utility tool which was able to do URL encoding, and rebuild and application. As far as I have seen, FreeMarker functions do not allow that kind of manipulation either (I may be wrong, feel free to correct me).

Security issue

However, the goal of this post is to show you why mixing code with templates is generally a bad idea, however powerful the technique may be. As best example, I’ll show you a major security issue with using both Spring and Groovy. I’ve chosen Spring because most of my applications use this framework, and I’m quite confident yours too.

First, we’ll write a simple service, which should never be exposed to the end user. Its goal is to retrieve my list of credit card numbers, in order to be able to choose which one I’ll use to buy my PS3 system.

package com.example;

public class MyPrivateService {
    private String[] theCreditCardNumbers;

    public String[] getCreditCardNumbers() {
        return theCreditCardNumbers;
    }

    public void setCreditCardNumbers(String[] someCreditCardNumbers) {
        theCreditCardNumbers = someCreditCardNumbers;
    }
}

And the following Spring configuration :

Here’s a sample Groovy script that shows you how this could be used in your application :

import org.springframework.beans.factory.BeanFactory
import org.springframework.beans.factory.xml.XmlBeanFactory
import org.springframework.core.io.ClassPathResource


// setup Spring Application Context
def beanfactory = new XmlBeanFactory(new ClassPathResource("com/example/applicationContext.xml"))

// print credit card numbers
def creditCardService = beanfactory.getBean("creditCardService")

creditCardService.creditCardNumbers.each {
 println it;
}

which outputs :

1114-554112-01115
1151-45454-454490

Imagine my service is for pure internal usage, and that it must NEVER EVER be exposed to the public. However, in some other part of your web application, there’s a front end page which may be customized (themed) thanks to the Groovy template engine. This allows the customer to create his own theme, granted he writes a good looking HTML page. The user could write something as simple as :

Hello, M. ${user.name} !

You may think that your system is secure, because you only exposed the User bean to your template :

import groovy.text.SimpleTemplateEngine

class ProofOfConceptController {

 def index = {
  def template = '''
   <%  def content = new StringBuffer();
    def beanfactory = new org.springframework.beans.factory.xml.XmlBeanFactory(
     new org.springframework.core.io.FileSystemResource("spring/resources.xml"))
    beanfactory.beanDefinitionNames.each {
     content << "Bean named $it defined in application context";
     content << ""
     def bean = beanfactory.getBean(it)
     bean.properties.each {
      content << "$it"
     }
     content << ""
    }
   %>
            $content
  ''';
  def binding = ["user" : "cedric"]
  def engine = new SimpleTemplateEngine()
  ["rendered" : engine.createTemplate(template).make(binding)]
 }
}

Here’s the output :

Bean named creditCardService defined in application context

    * metaClass=org.codehaus.groovy.grails.commons.metaclass.ExpandoMetaClass@ea7776[class com.example.CreditCardService]
    * transactional=true
    * creditCardNumbers=[Ljava.lang.String;@4e94a4
    * class=class com.example.CreditCardService

Using this Grails controller, the user will be able to browse all beans defined within the traditional ``resources.xml'' Grails file !

How does it work ?

This proof-of-concept assumes that the user knowns about the underlying technology : Grails. Using this, you can grab the user defined application context (resources.xml) in order to list all beans. With the previous template, the hacker could find out interesting beans and properties. Now he should just adapt the template to read the creditCardService property named creditCardNumbers… Well, I must admit this is rather tricky : the hacker has to know about the Grails internals, but sure this example could be adapted to other web applications. Spring provides a useful but risky class called WebApplicationContextUtils which allows, given a session, retrieving the application context. Therefore if the session (that’s the tough part) is avaible to the template, one could hack the system. Fortunately, it is unlikely that the programmer makes the session available to the template, and we must encourage not to do so.

Conclusion

I’ve just showed you how one could use Groovy templates in order to hack a system or steal information. The problem resides in the mixed environment where the hacker has access to the same static utility methods as the programmer, in the same JVM. Therefore, if you cannot isolate the template engine from the rest of the system (I wonder how it could be done), you must really be careful about what you do…

Replacing Velocity with Groovy = JSmarty ?

Wed, 1 Aug 2007 00:00:00 +0200

Templating languages as first choice technology for front-end customization

I’m not fond of JSF-like front end technologies. I mean, for anyone who’s used to templating technologies such as Smarty for PHP, using JSF is masochism : crying things like you can’t just iterate over a simple collection. With JSF, you must understand the underlaying philosophy and the technology before you succeed in doing something as simple as echoing a set of records. Too complicated for simple designs, and, nonetheless, unusable by a web designer (I mean someone who’s able to write HTML, but knows almost nothing of programming).

That’s why I like templating technologies : simplicity, and with a good application design, you won’t break the MVC model. Customizing a web design for the needs of a customer would not require anything else than changing a few front-end templates. For this I’ve been using Apache Velocity in Java, while in PHP I was used to Smarty.

Velocity’s main drawback

But Velocity does not offer the softness Smarty has : imagine your web designer needs a date format tag in order to pretty print a date variable. In Smarty, you just need to write a plugin (ok, for dates it is just included ;-)), and it is straight available in your templates. With Velocity, you must imagine all usages of the variables or objects you expose before you make a template. I mean that you must think that a web designer would like to format a date, then expose the date utility to the template.

In this case, in order to be sure you won’t forget anything, you could simply expose all of your utility classes to the template, but this is fondamentaly unnecessary, ugly and you could possibly never imagine everything a template writer could need. The problem is right there : with Velocity, if you did not think of something at the design time, then it will not be possible to do at runtime, just because template variables and utility classes are exposed programmatically. It is just right for variables, but a problem for utility classes. That’s the main drawback I found to Velocity.

Then came Groovy and its SimpleTemplateEngine. Basically, it does the very same as Velocity for templating, but its main advantage is that you can embed groovy code into the template :

import groovy.text.SimpleTemplateEngine

def tpl_src ='''
<% def formatDate(date) {
    new java.text.SimpleDateFormat("EEE, MMM d, ''yy").format(date)
}%>

The date today is <% print formatDate(date) %>'''

def binding = [ "date": Calendar.getInstance().getTime() ]
def engine = new SimpleTemplateEngine()
println engine.createTemplate(tpl_src).make(binding)

Just as simple as that ! You won’t need to rebuild an application in order to make an utility class, which has fundamentally nothing to do with the application logic, available to a template (front end logic). That’s where I liked Smarty so much, and makes me think I should actually think of replacing my Velocity templates with Groovy ones.

Should I move ?

Well, before moving, I’ll have to double check things like :

performance : how does Groovy templates compare to Velocity (Smarty has this caching thing)?
template inclusion : how can I manage template inclusions like velocity does ?
security : beeing able to run arbitrary code in the template may not be a so good idea…

Well I have a semi-answer for the last one : GSP pages, built upon Groovy, seem to have a tag, but GSP’s are part of the grails framework, which is too large for this particular need.

What do you think ?

Dynamic database models with Groovy and Hibernate

Thu, 26 Jul 2007 00:00:00 +0200

The problem

I’ve been working some time with Hibernate and been quite happy with it. However, for one of the projects I’m working on, I’ve been facing a problem Hibernate could not manage alone. The main idea was just to migrate an existing database from MySQL to an ORM DB independent through Hibernate.

Independently of the various compatibility problems which I should not address now, this project had a major difference with traditionnal database modelling : the application works with a different database for each customer, and each database has its own model (at least for some tables). Deploying for a customer does not require compiling a new application. Let’s come with an example.

In documentary management, a document can be described as a set of fields :

 Document(title, abstract, subject, body)

While some of those fields fit for every customer, working on specific documents require more fields. For example :

 Document(title,abstract, subject, body, category)

where category is a custom field.

Generally, if you are used to Hibernate, you may think the problem can be solved with a simple bean where all fields are stored as a map :

 public class Document {
    private int theID;
    private Map theCustomFields;
    /* ...  here comes the getters/setters */
}

+
There are two problems with this model :

types of properties are defined at compile time, and cannot be changed (property name type is String, but the property value is also. What if you want to store a boolean ?)
- Hibernate will generate two tables : one for the document, and the other one for the custom fields values.

This is surely not what we want : the existing model just consists of a simple document table for which all columns refer to a document property (aka custom field). This is both much simpler to read and does not require any join in order to read all the fields of a single document.

This ``limitation'' of Hibernate just meant that I would not be able to migrate the database without changing its schema. In my case, this was just problematic as I could not upgrade the database schema, too many different components, written in several languages (Java, Perl, …) could write or read this database.

Groovybernate !

Therefore, I’ve been searching for a solution, and thought about dynamic bean generation. I’ve had read about the dynamic language Groovy, and decided to take a look. As it produces 100% Java compatible bytecode, I thought it would just fit for Hibernate. It took not very long to find out that I would be able to do what I wanted : achieving maximum database flexibility thanks to Groovy and Hibernate. This means runtime modification of the model, and optimum performance (no joins whatsoever for just bean properties).

How does it work ?

From Groovy 1.1 (still in beta stages), you may use annotations. That sounded good for me as I’ve always been using Hibernate Annotations in my projects. The idea was just to dynamically create an Annotated Java Bean from a description file, then make Hibernate aware of this class and voilà !

The following consists of a simple Proof-Of-Concept. Instead of using a configuration file for describing the model, we’ll just do it programmatically. Then we’ll show how to use this model in order to dynamically create a Groovy class, create instances of it through Java and, eventually, persist it via Hibernate.

Why not pure Groovy ?

One could say that if I’ve managed to do this with Java and Groovy code mixed up, then I could’ve done it in pure Groovy. True. But I still remember that Groovy is a dynamic language loosely typed, which, therefore, does not offer the same compile-time security one could have with pure Java. I’m using IntelliJ IDEA every day for developpement, and not beeing able to say if your code is valid statically with Groovy is just a source of bunches of bugs, especially when you are using plents of third party libs. The following POC has been made with IntelliJ IDEA and the latest Groovy 1.1 beta 2, which allows compiling both Java and Groovy code at once, mixing up both languages without caring about cyclic dependencies. To be honest, my first trial was made with the beta 1 which did not offer this, and it was just more complicated : you had to separate 3 trees of source code : one for pure Groovy, one for pure Java, and the latest one is an interoperability set of Java interfaces and enumerations that allows the other two to communicate. Hopefully the Jetbrain guys went off this and provided us a new version of the compiler, and you should not have to do those (dirty but necessary) tricks. Note that using IntelliJ IDEA is not necessary. However, as the latest EAP offers an outstanding Groovy support, I strongly suggest that you take a look ;)

To conclude, mixing up Java and Groovy code is just a mean of having the best of the two worlds : security of strongly typed Java and flexibility of dynamic Groovy. This just makes the best thing I’ve ever programmed in my life, that’s truly groovy !

Preparing the model with meta description

If I wish to be able to dynamically generate beans, the best idea is to have a meta description of my bean. If you are aware of Groovy, you could think about the ExpandoMetaClass which allows to dynamically add properties/methods on an object. However, we could not manage to use it in this case, because we needed to dynamically annotate the class and its properties. Groovy 1.1 supports annotations, but is not yet able to dynamically add annotations to classes or members (if I’m wrong, tell me now !).

So I went by creating my own simplified meta model. Here’s the code (Java) :

public class MetaClass {
 private boolean isEntity = true;
 private String theParentClass;
 private String theName;
 private String theTableName;
 private Map

This meta class allows describing a bean which should be persisted by Hibernate. We need to be able to describe the class fields too (Java) :

public class MetaField {
 private String theName;
 private boolean isPK;
 private MetaType theType;


 public MetaField() {
 }

 public MetaField(String aName, boolean aPK, MetaType aType) {
  theName = aName;
  isPK = aPK;
  theType = aType;
 }

 public String getName() {
  return theName;
 }

 public void setName(String aName) {
  theName = aName;
 }

 public boolean isPK() {
  return isPK;
 }

 public void setPK(boolean aPK) {
  isPK = aPK;
 }

 public MetaType getType() {
  return theType;
 }

 public void setType(MetaType aType) {
  theType = aType;
 }

}

Then eventually, a set of meta types, with their Hibernate translation :

public enum MetaType {
 STRING("String"),
 INT("Integer"),
 LONG("Long"),
 DOUBLE("Double"),
 DATETIME("@Date Date"),
 DATE("@Temporal(TemporalType.DATE) Date"),
 TEXT("@Column(length = 2147483647) String");

 String theHibernateConversion;

 MetaType(String aStringValue) {
  theHibernateConversion = aStringValue;
 }

 public String getHibernateConversion() {
  return theHibernateConversion;
 }
}

Ok, let’s see now how we programmatically describe a bean :

MetaClass meta = new MetaClass(cl);
meta.setName('test');
meta.addField(new MetaField("title", false, MetaType.STRING));
meta.addField(new MetaField("date", false, MetaType.DATE));

Easy, isn’t it ? You may be wondering why we must provide a GroovyClassLoader. This is something that I’ve been working with since the beginning, without cleaner solution : in order to be able to get a class instance which is both recognized by Groovy and Hibernate, Groovy must be the parent class loader…

Generating a groovy bean

Ok, this is the core trick : now that we have a meta description of a class, we need to translate it to a real Java Class object. This implies compiling our meta class into real bytecode. As we cannot use the ExpandoMetaClass, we’ll do it through dynamic code generation. As this kind of stuff is much easier to both write and read in the Groovy language (get it ? the best of the two worlds !), we’ll write it in Groovy. That’s where the old cyclic dependencies come : notice how we’ll use the MetaClass class which we’ve written in Java, while the Java code will have to be able to access this bean generator class, written in Groovy… Ugly enough, before beta 2, I had to write interfaces for all those objects which are shared between the two (and you’ll reckon that writing interfaces for beans is somehow… ridiculous). So here’s the core groovy bean generator (Groovy) :

import MetaClass
import groovy.text.Template
import groovy.text.SimpleTemplateEngine
class GroovyBeanGenerator {
 public String getBeanSource(MetaClass meta) {
  def tpl_src = '''import javax.persistence.*;

<% if (meta.isEntity) { %>
@Entity(name="${meta.tableName}")
<%}%>
class ${meta.name} <% if (meta.parentClass!=null) { %> extends ${meta.parentClass} <%}%>{
<% for (entry in meta.fields) {
 def field = entry.value
 print "    ";
 if (field.isPK) print "@Id ";
 println "${field.type.hibernateConversion} ${field.name}"
}%>
}
  '''
  def binding = [ "meta": meta ]
  def engine = new SimpleTemplateEngine()
     def template = engine.createTemplate(tpl_src).make(binding)
     return template.toString();
 }
}

It works by defining a Groovy template, which is just a Groovy bean class code. Note that this code could surely be written shorter thanks to closures, but I’d rather keep it simple for newcomers. The goal of this class is just to generate source code for our bean. This bean is annotated with Hibernate Annotations. Now, we’ll write the simple Java bunch of code which will allow us to create instances of this bean and register them in Hibernate.

GroovybernateUtil for dynamic registration of beans

This class will do two things (ok, we could have splitted them apart) : create instances of beans using the groovy bean generator and manage the Hibernate session.

public class GroovybernateUtil {
 private String theApplicationName;
 private AnnotationConfiguration theConfiguration;
 private boolean isModelUpdated;
 private SessionFactory theSessionFactory;
 private GroovyClassLoader theClassLoader;

 public GroovybernateUtil(GroovyClassLoader aClassLoader, String aApplicationName, String aDialect, String aDriver, String anUrl, String aUsername, String aPassword) {
  theClassLoader = aClassLoader;
  theApplicationName = aApplicationName;
  theConfiguration = (AnnotationConfiguration) new AnnotationConfiguration()
   .setProperty("hibernate.dialect", aDialect)
   .setProperty("hibernate.connection.driver_class", aDriver)
   .setProperty("hibernate.connection.url", anUrl)
   .setProperty("hibernate.connection.username", aUsername)
   .setProperty("hibernate.connection.password", aPassword)
//   .setProperty("hibernate.generate_statistics", "true")
   .setProperty("hibernate.hbm2ddl.auto", "create-update");
//   .setProperty("hibernate.hbm2ddl.auto", "create-drop");
        isModelUpdated = true;
 }

 public Session getSession() {
  if (isModelUpdated) updateSessionFactory();
  return theSessionFactory.openSession();
 }

 private void updateSessionFactory() {
  theSessionFactory = theConfiguration.buildSessionFactory();
  isModelUpdated = false;
 }

 public String getApplicationName() {
  return theApplicationName;
 }

 public void setApplicationName(String aApplicationName) {
  theApplicationName = aApplicationName;
 }


 public void registerGroovyBean(Class aBeanClass) {
  theConfiguration.addAnnotatedClass(aBeanClass);
 }

 public void registerGroovyBean(String aBeanClass) throws ClassNotFoundException {
  theConfiguration.addAnnotatedClass(theClassLoader.loadClass(aBeanClass));
 }

 public void registerGroovyBeanSource(InputStream aBeanSource) {
  theConfiguration.addAnnotatedClass(theClassLoader.parseClass(aBeanSource));
 }

 public Statistics getStats() {
  return theSessionFactory.getStatistics();
 }
}

Anyone who’ll need to work with a dynamic database model should use this simple utility class in order to create an Hibernate session that will be aware of dynamic beans. We’re done !

Make it run together

Ok, you’ve read a bunch of code, and now, I assume you’ll want to test it by yourself, and, surely, you’ll want to see how you’ll be able to create instances of thoses beans, set their properties and so on. Hopefully, each object instanciated in groovy inherits the GroovyOject interface, which provides the getters/setters we need.

Here’s the simple test code. It reads a large CSV file which should be persisted in database. The CSV columns refer to bean properties, while rows are instances :

public class Groovybernate {
 public static void main(String[] args) throws IllegalAccessException, InstantiationException {

  GroovyClassLoader cl = new GroovyClassLoader(Groovybernate.class.getClassLoader());
  Thread.currentThread().setContextClassLoader(cl);
  GroovybernateUtil util = new GroovybernateUtil(cl, "test",
    "org.hibernate.dialect.MySQL5Dialect",
    "com.mysql.jdbc.Driver",
    "jdbc:mysql://localhost:3306/hibertest",
    "test",
    "password");
  MetaClass meta = new MetaClass(cl);
  meta.setName("test");
  meta.setTableName("test");
  meta.addField(new MetaField("id", true, MetaType.INT));
  meta.addField(new MetaField("c1", false, MetaType.STRING));
  meta.addField(new MetaField("c2", false, MetaType.STRING));
  meta.addField(new MetaField("c3", false, MetaType.STRING));
  meta.addField(new MetaField("c4", false, MetaType.STRING));
  meta.addField(new MetaField("c5", false, MetaType.STRING));
  meta.addField(new MetaField("c6", false, MetaType.STRING));
  meta.addField(new MetaField("c7", false, MetaType.STRING));
  meta.addField(new MetaField("c8", false, MetaType.STRING));
  meta.addField(new MetaField("c9", false, MetaType.STRING));
  meta.addField(new MetaField("c10", false, MetaType.STRING));
  meta.addField(new MetaField("c11", false, MetaType.STRING));
  meta.addField(new MetaField("c12", false, MetaType.STRING));
  meta.addField(new MetaField("c13", false, MetaType.STRING));
  meta.addField(new MetaField("c14", false, MetaType.STRING));
  meta.addField(new MetaField("c15", false, MetaType.STRING));
  meta.addField(new MetaField("c16", false, MetaType.STRING));
  meta.addField(new MetaField("c17", false, MetaType.STRING));
  try {
   util.registerGroovyBean(meta.getBeanClass());
  } catch (ClassNotFoundException e) {
   e.printStackTrace();
  }
  Session session = util.getSession();
  Transaction tx = session.beginTransaction();
  long sd = Calendar.getInstance().getTimeInMillis();
  BufferedReader reader = null;
  try {
   reader = new BufferedReader(new FileReader("resources/large.csv"));
  } catch (FileNotFoundException e) {
   e.printStackTrace();
  }
  int loop = 0;
  try {
   String line = reader.readLine();
   while (line != null) {
    try {
     loop++;
     String[] values = line.split(":::");
     GroovyObject bean = meta.newInstance();
     bean.setProperty("vid", loop);
     for (int i = 0; i < 17; i++) {
      bean.setProperty("c" + (i + 1), values[i]);
     }
     session.save(bean);
     if (loop % 10000==0) {
      System.out.println("Committed "+loop+" records...");
      session.flush();
      session.clear();
     }
    } catch (ClassNotFoundException e) {
     e.printStackTrace();
    }
    line = reader.readLine();
   }
  } catch (IOException e) {
   e.printStackTrace();
  }
  tx.commit();

  long ed = Calendar.getInstance().getTimeInMillis();
  double duration = ((ed - sd) / 1000);
  double persec = ((double) loop / duration);
  double perhour = 3600 * persec;
  System.out.println(MessageFormat.format("Processed {0} elements in {1} seconds ({2} records/h)", loop, duration, perhour));
//  Statistics stats = util.getStats();

 }
}

Conclusion

To conclude, we’ve been able to perform what we wanted to do : to dynamically generate a bean from a meta description, which could be easily read from a configuration file, in order to achieve maximum flexibility in database schema customization. This allows customizing some objects of an application for customer needs, without needing to update an existing database schema (our need), with human readable database objects (one table per bean, instead of two tables, one for instances, the other for its properties) and very good performance. I hope this has given you some ideas about how you can go beyond the ``limitations'' of a static Hibernate object model without breaking the logic behind. I mean that with this solution, what should be pure configuration (database customization for customer needs) remains configuration without breaking the rules of ORM. If I had wanted to perform such without Groovy, I would have been obliged to compile a dedicated application for each customer, which is both unnecessary and completely illogic. Moreover, any change in the model would require building a new application…

Comments welcome ;)