How AI Can Learn To Paint Watercolours Using The TRL Framework And OpenEnv

🔍 Read the full analysis: How AI Can Learn To Paint Watercolours Using The TRL Framework And OpenEnv on ThorstenMeyerAI.com

An independent engineer has published a complete, open-source reproduction of Surya Narreddi’s viral watercolour painting AI, utilizing the TRL framework and OpenEnv environment on Hugging Face. This release includes datasets, training scripts, models, and the reinforcement learning environment, making the project fully accessible for research and development. The original project, which gained over 1.5 million views in August, trained a language model to generate JavaScript code that produces watercolour-style images, emphasizing aesthetic quality over verifiable correctness.

The reproduction follows Narreddi’s approach by training a Qwen model with reinforcement learning against a composite reward function. This reward combines a compilation check, code length, a style judge based on human-curated references, and an open preference model called HPSv3, which scores images based on human aesthetic preferences. The training process involves 110 steps, 240 episodes per step, and eight generations per episode, all orchestrated via Hugging Face’s infrastructure. The project’s key contribution is the fully open release of all artifacts, including datasets, code, and models, under an open license, facilitating community engagement and further experimentation.

This work aims to explore whether reinforcement learning can optimize models based on aesthetic taste, rather than traditional verifiable metrics. Unlike most recent RL work focused on problem-solving or code correctness, this project emphasizes subjective aesthetic judgment, aligning with older RLHF (Reinforcement Learning with Human Feedback) traditions. The project’s output—editable, readable code that produces watercolour images—offers transparency and inspectability absent in pixel-based models, which tend to generate more uniform, ‘perfect’ images.

Implications for AI Art and Aesthetic Optimization

This project demonstrates that reinforcement learning can be applied to optimize AI models based on subjective human aesthetic preferences, not just objective correctness. The open release of datasets, models, and environments lowers barriers for researchers to experiment with aesthetic-driven AI training, potentially advancing creative AI applications. It also highlights a shift toward more transparent, interpretable AI outputs—since the model produces code that can be read, edited, and re-run—offering new avenues for understanding AI creativity and decision-making. The approach could influence future developments in AI art, emphasizing style and subjective quality over traditional metrics.

Evolution of Generative AI and Artistic Practice

The project situates itself within the lineage of early generative AI art, from DeepDream (2015) to neural network portraits by Mario Klingemann and datasets curated by artists like Anna Ridler. Historically, AI art has ranged from debugging tools turned into art to experiments probing the boundaries of machine creativity. Narreddi’s approach mirrors these traditions by manually curating a reference pool and training models against it, emphasizing the importance of datasets and subjective judgment in artistic AI. The original project gained viral attention in August through a video showcasing the model’s watercolour paintings, which appear loose, imperfect, and handmade—contrasting with the highly polished outputs typical of mainstream image generators.

Prior work in this domain has often lacked open artifacts, limiting reproducibility. This new reproduction fills that gap by providing all necessary components openly, enabling broader community engagement and further technical exploration. The project also draws inspiration from earlier experiments in training models on hand-labeled datasets, highlighting the importance of human curation in artistic AI development.

“This open reproduction aims to test whether reinforcement learning can truly optimize for aesthetic taste, not just correctness or utility.”

— Thorsten Meyer, project author

Open Questions on Model Performance and Aesthetic Alignment

It remains unclear how closely the reproduction’s outputs match the original in quality and style. The project presents visual median outputs for different reward mixes but does not provide a definitive quantitative comparison or final verdict on which reward configuration yields the best results. The full technical report from Narreddi, which could clarify these points, has yet to be published. Additionally, the extent to which reinforcement learning over aesthetic taste can generalize beyond this specific task remains an open question, as does the potential for broader application in diverse artistic styles.

Future Directions for Open Aesthetic Reinforcement Learning

The next steps include awaiting Narreddi’s full technical report, which is expected to provide detailed evaluations and insights. The open artifacts released by the reproduction enable community-led experiments to compare reward mixes, test different datasets, and explore other aesthetic criteria. Researchers may also investigate how well the approach generalizes across different artistic styles or whether it can be extended to other subjective domains such as music or literature. Further development could involve integrating more sophisticated visual models or refining reward functions to better capture nuanced aesthetic preferences.

Key Questions

What is the main innovation of this open reproduction?

The project fully releases datasets, models, scripts, and environments, enabling transparent, reproducible research into AI aesthetic learning using reinforcement learning over subjective taste.

How does this work differ from traditional AI art models?

Unlike pixel-based models, this approach produces readable code that creates watercolour images, allowing inspectability, editing, and understanding of each decision, emphasizing aesthetic preference rather than pixel perfection.

Can this method be applied to other art styles?

Potentially yes, but further experimentation is needed to determine how well reinforcement learning over aesthetic preferences generalizes beyond watercolours and the specific datasets used.

What are the limitations of this project?

Current uncertainties include the final quality comparison with the original, the generalizability of the approach, and the lack of a published full technical report to validate the results and methodology.

Where can I access the open artifacts?

All datasets, models, and scripts are available on Hugging Face in a dedicated collection, accessible for community use and further research.

Primary source: Hugging Face · via ThorstenMeyerAI.com