Benjamin Jörissen

Educational Scientist

Software

Software Designs for Critical Educational Practice and Research

As a UNESCO Chair dedicated to research in digital culture, we are concerned with the effects of digitality, and of artificial intelligence in particular, on the ongoing formation of world- and self-relationings within the disruptive global transformations of the present. In recent years we have examined generative AI closely and found that this technology, while challenging education in many respects, also offers new means of critical educational and scholarly articulation. Having worked for many years on Medienbildung, design, power and subjectivation, and on an educational understanding of digitality and digital environmentality, we have taken the opportunity not only to research and write about digitality, but to do theory in code. The result is a growing series of software that emerges from participatory design processes and enters educational settings as a co-actor rather than a tool. All of it is released either as open source or as source-available.

Our software projects

As AI is transforming society and the world of work, it is increasingly becoming a subject of education. The project that created ATELIER – AI4ArtsEd (Artificial Intelligence for Arts Education) – explored the opportunities, conditions of application, and structural limits of participatory artificial intelligence (AI) in culturally diversity-sensitive artistic-pedagogical settings in Arts and Cultural Education.

ATELIER was developed to foster critical and creative collective engagement with generative AI in culture, arts, and media education. Generative AI models are powerful tools, but they are also “black boxes.” A central goal of the project is the critical exploration of generative AI as an actor in cultural, artistic, and societal contexts. Generative models are not neutral – they carry cultural, social, gender-related, and ethnic biases inherited from their training data. These distortions must be made visible and critically examined in order to understand what kind of actor generative AI actually is. In short, ATELIER seeks to offer many ways of interacting with generative AI – except the one we have been trained to adopt: that of mere “users” of a powerful technology.

As a platform designed exclusively for educational use, balancing the unpredictability of genAI outcomes with pedagogical care (safety and protection, while avoiding undue censorship) played a major role in the design process. While no technological solution provides absolute safety, ATELIER features a configurable multi-level safety system, GDPR monitoring, and EU AI Act compliance. Agents are resource-sensitive (meaning they have access to live energy data of the system the platform runs on, and will deliberately make use of this information where appropriate), and the platform has a context-sensitive help agent. It speaks 10 languages, individually configurable per device, and has a favorites system which allows for individual, but also collective use – sharing not only generated media, but also prompts and configurations, so that other members of a workshop may choose to pick up work from others.

Prompt Interception

ATELIER is based on the idea of Prompt Interception – a pedagogical method for interrupting the affirmative forms of subjectivation offered by AI through aesthetic and positional interventions. With genAI, these forms typically take the structure of augmenting, if not substituting, aesthetic, cognitive, and practical knowledge-related judgment through automated decision making: they are designed for modes of subjectivity that relieve its “user” of the processes of knowledge building and the formation of dispositions of judgment, thereby foreclosing them – while simultaneously producing results so elaborate and persuasive that they can easily generate an illusion of agency. This illusion structurally conceals the fact that the very processes in which knowledge and dispositions of judgment would be formed are being precluded.

Since every design – and software design in particular – inevitably produces affordances that anticipate certain ‘user’ subjectivities, which may take effect in affirmative ways, ATELIER strategically counters these with structurally resistive and heterogeneous modes of address: it addresses ‘users’ as collective, reflexive, entangled, critical, and explorative subjects.

early design of an intercepted genAI production flow
Early proof of concept using C.D. Friedrich’s “Wanderer Above the Sea of Fog” as an image to be automatically analyzed and transformed into different intercepted prompts.
Prompts intercepted on a technical level by manipulating the embedding vectors that “translate” language into numbers understood bei genAI image models. These images are results of the prompt “A house stands in a landscape surrounded by farmland, nature, and animals. A few people can be seen.”

Prompt Interception presets are represented as visual symbols of editable meta-prompts. They do not articulate styles (as common “prompt optimization” does), but rather historical and contemporary perspectives, positions, and attitudes.

ATELIER’s Modes

ATELIER consists of five different main modes available through the top menu

TOGETHER is constituted by non-egological agents (all agents are instructed to act as plural entities to avoid anthropomorphization) for dialogical support of AI-related teaching projects. They work in three phases: course preparation assistance, live course support (as a non-human co-actant), and follow-up. Its system prompt instructs Together to direct themselves to three forms of otherness, i.e. human collectives, situational materialities, and the machine and software framework (ATELIER) they are running upon. The agents are able to make use of an internal library (in order to enhance groundedness), document lesson plans and build importable and exportable individual workflows for reuse in live workshop situations (stored in the course instructor’s account). (As one of the latest additions to the platform, Together has yet to undergo thorough testing.)

CONTESTING is a resistant chatbot, built as an ‘anti-sycophant‘ agent, if you will. They challenge whoever speaks to them to articulate, develop, and defend creative ideas. CONTESTING autonomously and spontaneously generates media products of their own choosing – provided the ideas are deemed interesting in dialogue. In doing so, they subvert the assumption that genAI simply ‘does’ what users ‘command’. They are also programmed to abstain from any anthropomorphic emotional communication. First contact with CONTESTING feels decidedly matter-of-fact and may offer a glimpse into how this ‘alien subject‘ would probably communicate if it were not trained to please and retain customers in order to maximize profits.

FLOW is the heart of creative-artistic processes. It provides iterative, cross-medial artistic process trajectories oriented towards artistic action. It is basically an open workspace with an action circle. Whatever is drawn into the circle gets processed: “anything flows” is the principle here. Complex functions are visually represented on the canvas, offering detailed information on the reverse side of the visual objects. A header bar provides intelligent support and interactive visualization of the technical process.

LATENT LAB is a collection of comparative and deconstructive instruments inspired by current genAI research, designed for opening and exploring the genAI “black box.” Easily the most cognitively demanding mode of the platform, it rewards users with many opportunities for direct manipulation of the inner, usually hidden parts of genAI embedding and inference processes. Also, Latent Lab offers the option to train “Low-Rank Adapter” (LoRA) models using 10–50 of one’s own images, in a training sequence of about 3–4 hours (mind the resources), thus exploring the extent to which genAI models such as Stable Diffusion may be altered, e.g. to reproduce fewer stereotypes (spoiler: it is not very likely to succeed, but very educational to fail).

“Last but not least, CANVAS offers (almost) all ATELIER capabilities as combinable modules. CANVAS strives to make the structures of genAI accessible as modular elements while deliberately stripping away technical complexities that would disturb rather than support basic insights into the technology (for deeper technical exploration, anyone would use ComfyUI, which ATELIER uses as its main backend). Unlike anything on the market that we know of – including modular genAI environments – CANVAS supports nonlinear and recursive workflows (notice the red ‘fb’ feedback pipe in the screenshot below), in which evaluative nodes can refuse to pass on a prompt or image, giving the sending node feedback as to why its request was rejected. (The idea behind this is about the process rather than optimized results: what happens when different kinds of genAI models are played off against each other, correcting each other?)

ATELIER is “Source Available”: Non-commercial educational institutions may install, configure, and use the software in their educational offerings. Also, any person may reference and cite the software in academic publications, use excerpts for purposes of illustration, and install, configure, and use the software for research purposes. Commercial use only by agreement: Commercial use requires a separate written agreement with the author. The author may, upon request, grant written permission for specific modifications. For regular contributors, the author may offer a Contributor License Agreement.

ἀ·κρό·α·σις [aˈkro.a.sis]: listening, paying attention, hearing

akróasys is a sonic device that uses generative AI (genAI) very differently: not as a generator that returns finished audio, but as the oscillator of a playable instrument. The model sits where in an audio synthesizer a sine or a saw would sit in the signal path, and what it resonates the prompted meaning.

akróasys has two different genAI oscillators. The T5 Oscillator deconstructs the latent space of generative audio models before audio becomes generated. Two prompts mark two different points in the model’s semantic space, and the synth moves the point that rendered — between them, and well past both, into regions no phrase can name: textures, transients, patterns, field recordings, everyday noises, orchestral gestures, alien voices, human emotional expressions, impossible hybrids. Reaching what a prompt cannot reach is what this oscillator is for. The Language-Resonant Oscillator works the other way round: users describe an instrument, a language model writes the Csound orchestra for it, translating words into a veritable (also exportable) oscillator.

akróasys is Open Source: https://github.com/joeriben/akroasys/releases

https://soundcloud.com/benjamin-j-rissen/sets/akroasys

“Werkraum” is the predecessor of ATELIER: a lightweight ComfyUI Frontend with added pedagogical features, aiming to provide basic prompt and output safety and a very simple interface stripped of most technical parameters. Since ComfyUI is extremely powerful, tech-savvy educators may define their own workflows inside ComfyUI and export it for use inside Werkraum. Werkraum uses our own set of custom nodes, so make sure these are installed inside ComfyUI.

Werkraum is Open Source: https://github.com/joeriben/ai4artsed_werkraum

As we became curious about how complex dense theoretical thoughts could be and still be able to be implemented as software, we started to code a QDA-Software focused on the mapping paradigm of Adele Clarke’s Situational Analysis, but with a twist. We were inspired by Dewey/Bentleys Knowing and the Known, Karen Barads Agential Realism, and the critical thinking of Theodor W. Adorno’s Negative Dialectics, so we used core ideas of this heterogeneous collection as inspirations for the design of the database structure. In the spirit of Situational Analysis (we hope), every data object is “non-identical” in a certain sense: it is it’s own stack of “namings” (we do not use the word “code” here). This stack is not only as a memory of a succeeding formation of a naming, from cue to characterization, instead we like to think of these data objects as latent or superposed entities, with any concrete meaning (only) being one collapse of its semantic in-betweens. Oh, and there is no ontological fixation of “entities” vs. their “relations”: everything here is a way to look at/name namings, consequently, relations may be changed into namings and vice versa. Future will see of this proof of concept-Software is actually useful in real life research settings.

transact-qda is Open Source: https://github.com/joeriben/transact-qda