On the need to sustain your creative drive in the face of technological change

For a non-programmer, tell me about how you go about using a programming language to make generative art.

This will touch on something that is unusual about how I use programming languages in general. The common practice in the industry is one that involves very slow feedback loops, these things we call compile-test cycles: edit, compile, test. I tend to use programming languages that are much more interactive. This is the family of programming languages that come down to us from the communities of LISP and SmallTalk, primarily. In these languages, you’re engaged in a conversation with the computer — your program is running the whole time, you’re modifying it while it’s running, and you can inspect the state within the program to see what’s happening.

This is particularly good for exploratory programming, but also for art making. I can have a sketch running that is using a generative system I’ve created to produce some kind of visual effect. I could think, “What if this parameter were slightly different?” and instead of building a bespoke control panel to do that, I can execute a tiny snippet of code inside my editor that changes what’s happening in the program, so I’m still working in the same medium and I don’t need to switch to a different tool.

I might start with a blank canvas with a loop running that is redrawing something, but it doesn’t know what it’s redrawing yet. Then I will gradually add elements, and those elements may have some innate structure. They may be drawn from nature in some way. Often, in my work, I will start with some natural system I found intriguing, and I’ll think, “What would have to happen geometrically to create a thing that has a form like that?” Then I’ll try to build a system where I’m planting the seed, but the growth happens within the simulation.

I also do a lot of work that is inspired by different periods of art. Maybe there will be something Bauhaus-inspired; I’ll look at a pattern Kandinsky drew by hand and think, “What if I wanted an infinite number of those that were all as good as the one he did by hand? What would I need to tell the computer for it to know [how to do that]?” In that sense, my artwork is often at that meta level. I’m less interested in the single-object output than I am interested in the underlying system that makes things of that nature possible.

Your background is traditionally technical. How has that influenced your identity or your sense of aesthetics as an artist?

I don’t regard scientists and artists as fundamentally different kinds of people. In fact, I regard them as more alike than they are different.

The sort of division you see among people in modern American culture is, to me, a cultural artifact; it’s just an accident of education. I would say the same thing about athletics. The jocks versus geeks division is an entirely synthetic thing that arose in post-1950s America and spread in a diseased way to other parts of the world. There’s nothing about being good at using your nervous system to move your body through space that would make you bad at using your nervous system to reason about geometry.

Based on some early tests that show an aptitude or a proficiency, we’ve narrowly focused people into what we think is going to be the box in which they will perform, when we should be spending more time cultivating what people are innately and immediately good at but also filling in the rest of the profile. So if you’re somebody who finds mathematics easy but is intimidated by the idea of drawing classes, then you should be doing that. These things are all aspects of humanity, and it’s a mistake to leave any of them behind.

In your 2019 ClojuTRE talk on computational creativity, you gave a brief survey of historical definitions of creativity. After absorbing all of those, where do you net out? What grand unified theory of creativity do you subscribe to?

I think it’s the fundamental aspect that makes us human beings. Creative problem-solving is the thing that we do better [than any other species]. Communication is the other thing that we do better, which allows us to do creative problem-solving in groups. If you want to know why we’ve spread over the entire world and lived in every kind of ecosystem successfully, it’s because we’ve been able to creatively solve problems along the way. Without that, I don’t think we’re really people. Leaving aside your creative drives as an individual is a mistake, because it’s leaving aside your birthright as a human.

A question in the AI discourse right now is whether AI will ever be able to create the way a human does. Large language models can create reasonable facsimiles of mediocre writing and drawing, but that sort of path-breaking creative synthesis still seems to be uniquely human. As someone who has been in this field for a long time, what do you think is coming in terms of the influence of AI?

To touch on the first part of what you said, about mediocrity: when you have a big statistical model that is essentially taking the sum and then the average of the internet, whether it’s in words or pictures, then you can expect the output to be [average] by definition. Now, you can steer these models to get you somewhat surprising outputs, and that’s cool. I have some friends who train their own models and build complex workflows to come up with things that are very nice in terms of the outputs they achieve. For me, mostly, if I’m using a prompt to an LLM to generate an image, I can get an output that looks okay to good, because I word good and I have enough taste to pick the images that I think are okay. But after I’ve done that, I don’t feel like I’ve done anything, because I don’t feel like there’s any of me in the output.

I think a lot of where our good stuff comes from is actually from how the act of making the art changes us as individuals. Ages ago, I went to art school at night while I was doing a startup in Silicon Valley. I’d been a lifelong musician, and playing music my whole life meant that I heard everything differently. When I hear the leaves rustling, I hear the rhythm of the leaves rolling along the ground. When I hear the whistle on my kettle, I know what pitch it is. So I thought, “I’ll go to art school, and maybe it will change the way I see.” And of course it did. There’s no way you can learn to draw in charcoal and capture light and shadow without it changing the way you see everything for the rest of your life.

What if we take away the need to do any of those things to produce those outputs? Then we get an entire generation of people who do not transform themselves into having a higher level of perception. What does that do for our ability to discriminate between what is just AI slop and what is actually something amazing and beautiful? It’s leaving behind part of our birthright as humans, to outsource some of the best stuff we have going to the machines, even if the machines can do it.

Also, the more stuff there is, the more sifting has to be done to find the good stuff. Making a machine for the unlimited production of mediocre junk means that the signal-to-noise ratio is getting worse all the time, and I dislike this vigorously.

On the other hand, I think these technologies can become the components of amazing engineering solutions later on. An example of this, not in the artistic context, is that I took some LLMs and I attached them to a query apparatus for WikiData, the database version of Wikipedia. I was able to use the LLM to get the data into the system from natural language. Then I do a query against this fact database, and then I take the series of dry facts that it returns and have it reformatted as nice, flowing prose. So I get something that you can get into and out of with human language that doesn’t hallucinate any details, and this is actually immediately useful.

I think many things of that nature are coming. Artistic tools where the trained model is more like a paintbrush and less like an outsourced cheap artist are going to be extremely powerful. In cinema, I think we’ll see the cost of making movies drop to one-one-hundredth of the time and one-one-hundredth of the cost using these kinds of tools, because CGI is such an important part of film production already. In this sense, when the good tools come out of it, you will see actual artists be able to do more and better.

Have you been able to find a balance between the things you do to pay the bills and the things you do to satisfy an artistic impulse? Do you find the same amount of creativity and joy in your work at Applied Sciences as you do in the art you make?

Here, I have to start by saying that I’m in a position of ridiculous privilege. I came of age at a time when the things I liked to do for fun were among the most lucrative things you could do for a living.

Throughout my career, I have been able to work on only things I’m interested in and be paid very well for them, both on the science and programming side and also on the art side. Obviously, I make more money from the tech stuff than the art stuff. But in years when I’m more active, like in 2020, I made enough that I could have made a living in Berlin just from the art side. This is possible. It’s difficult and it requires a lot of luck, but it is possible. So I’m in the weird position where I don’t have to choose between the things I love and the things that pay the bills because everything I get paid for is also something I love. And I recognize the tremendous privilege of that statement.

What do you think it takes to do that, beyond luck? Are there things a person can do to be more likely to have that kind of outcome?

Having a very active daily practice, and never letting it get away from you, is incredibly important. Björk has a fantastic quote about not letting yourself get gummed up and only releasing something every seven years because it puts you out of the flow of creating: “Don’t hold your breath for five or seven years and not release anything, and then you’ve just got clogged up with way too much stuff… You lose contact to the part of you, your subconscious, that’s writing songs all the time, and the part of you that’s showing it to the world… That’s more important, to sustain that flow, than to wait until things are perfect.”

Whatever it is that you do, you have to really do it. If you have a choice between doing it for three hours on Sunday or doing it for 15 minutes a day for the rest of the week, do it 15 minutes a day, because what you do every day is what your brain is working on when you’re not paying attention. Your subconscious is making progress on the things you do constantly. There’s a bowdlerization of Aristotle’s Nicomachean Ethics that gets quoted a lot, which is that excellence is just a function of habit. It’s what you do repeatedly. Lean into it. Do the work.

And — this is the bad news, because many programmers or artists are not necessarily interested in standing up on a chair and shouting about themselves in public—if you do beautiful work and nobody sees it, you’re not going to have a good career. You have to find a way to surface what you’re doing.

If it were five to 10 years ago, I would say to get a Twitter account, communicate with the kinds of people who are interested in the kind of thing you do, post all of this work that you’re doing as your daily practice, and you will be noticed. Today, it’s a more complicated situation. Some arsonists have set fire to Twitter and it’s now full of smoke and dead bodies, so very few people you would want to find your work will go to that place. I think we’re in an interregnum where there isn’t a good public space to demonstrate excellence for most arts. But it is important that you find a way to do that, or you will likely go unnoticed.

I also wanted to ask you about your time AT&T Research, formerly Bell Labs. Bell Labs has a mythical place in tech lore. It was a hotbed of innovation and a Schelling point for practically every computer science pioneer you’ve ever heard of. Did that still penetrate the company’s DNA when you were there?

It was definitely a unique environment. First, as in any such situation, it was the people. You had a large concentration of brilliant people all in one place. That’s always a good thing.

The facility where I worked, the Claude Shannon Lab, was in a leafy suburb in New Jersey. We would go down to eat in the cafeteria, and there were floor-to-ceiling glass windows, and we would see deer outside. In my wing, the people in the other offices were Bjarne Stroustrup, the inventor of C++, and David Korn, who created KornShell. I used to ride in his minivan from downtown Manhattan, because there were a bunch of us who preferred to live in the city. So the vibe had mostly to do with the people, and then the facility itself being the perfect leafy campus environment, but tuned for grown-ups — well, eternally Peter Pan grown-ups.

We did some great work there, even though when I worked there it was after the heyday. Unix was invented around the time I was born, so I missed out on all those great things. But I know most of those people because I was very young when I got started in the industry, and they weren’t dead yet. Some of them are still walking around. So I have all the stories, I’m happy to report. The vibe, I think, was still similar, but obviously the level of work, while good, wasn’t as world-shaking as it was earlier in the 20th century.

Why do you think that was? Was it a function of something changing in the way the work was supported?

There were a couple of things. One is that the way research was supported changed. Here we come back to that idea of patronage. Both artists and scientists have in common that they do their best work when they are left alone and allowed to chase their own curiosity and their own aesthetics and their own feelings. The appetite and the surplus to allow that has decreased year on year since the middle of the 20th century.

There was a period where this was really celebrated, and it was considered a good use of funds to have people do things that may pay you back nothing but also may give you a whole different world. You would fund it with some faith in the fact that if the people are talented enough, something good will come out of it.

After the Reagan–Thatcher revolution, that became less of a thing. Ideologically, everything shifted to this idea that you should have a return-on-investment angle on what happens. And because you can’t predict the outcome of research, it is effectively impossible to have a return on investment attitude towards it.

A great example of this is the iPhone. The capacitive touch display was invented 25 years before that at Bell Labs by somebody who was just chasing their own interest. If that person hadn’t had the opportunity to plant those seeds, then Apple could not have reaped the benefits later. Right now, I feel like we’ve really shifted towards reaping, and left sowing to be somebody else’s problem. This will continue to harm us in the future, because if we keep doing basically the same things over and over again, we won’t have any new seed corn.

There are certainly little pockets where that focus on something other than ROI still exists. But I agree. It feels like everyone recognizes the value of something like Bell Labs, yet very few people have the risk appetite or long-term thinking to fund that anymore.

It’s not just the absence of a Bell Labs sort of thing. There are other social opportunities available that are not followed. For example, I was talking to some people who will remain nameless but who are very high in an organization that makes a popular search engine and browser. I wanted them to fund some improvements to a text editor called Emacs that I’ve been using for nearly 40 years. With a good team working on it and with some actual financial support, a lot could be improved. Around half of their employees use Emacs, so it seemed like it would even pay them back, in some sense. But they told me that the most their enormous, many-billions-a-year company could possibly [contribute] was funding for some student [project].

This kind of thing is insane. These are public goods that they consume, but they don’t see it as their responsibility to help support that commons. This is a problem with open-source software in general — it is insufficiently supported. It’s shared infrastructure, and shared infrastructure requires shared support.

If you could reshape the way the internet has evolved, where would you start?

I would try to prioritize [changing] some of the infantilizing drives of current products. It is very fashionable at the moment to believe that if a person can’t use something immediately on first seeing it, then it should be thrown away, because people are stupid and have no patience. This is a prevalent way of thinking about user interfaces. But if you look at the user interface of the violin, it’s terrible for quite a while. You have to put in some effort before you can do anything useful with the violin. But then you can do something that you simply cannot do with a tiny children’s xylophone. There are effects you can achieve if you’re willing to put in the work.

I feel like there’s a large area to explore of slightly more difficult things that have a higher ceiling. I believe you should raise the floor as much as you can, but you shouldn’t do it by lowering the ceiling.

I would like to make it more possible for people to, for example, automate things on their own; end user programming is the technical term for this. In a system like HyperCard, this was very effective. People could build systems to run their entire business inside of this very cool piece of software that you ran on a Macintosh. I don’t see a modern thing that is as good. There’s more we can do to democratize the programmatic aspects of owning a computer so that people have more power as individuals.

There have to be these open-box systems where you can play with the parts. Otherwise, you’re strictly a consumer. On Instagram, that’s exactly how I feel. I post my artwork there, but that’s the limit of what I can do. Someone else has decided the limits of my world. And I resent that.

At the end of your talk on creative computation, you give some recommendations for programmers who want to get in touch with their creative side: take an art course, meditate, take psychedelic mushrooms. I assume those recommendations still hold, but what else would you recommend to anybody who wants to connect with their creativity?

The important thing, and I tried to stress it in that talk, is that you can approach things as a reasoning and reasonable agent who is putting one fact in front of another and trying to be very orderly and systematic. That is an important way of being. But there’s another way of approaching things, which is to open yourself up to your own intuition and to feel your way through things. That’s no less important a way of being. You have to have both to be a complete human being. So whether a person is a programmer who isn’t as in touch with their intuition, or they’re an artist who is not as in touch with their ability to be analytical, I feel that whichever side you’re coming from, you should be trying to fill in the part at which you are the weakest so that you can be a more complete person.

For a lot of people, getting in touch with the intuitive side also has to do with the body itself, because many people are very disembodied. So, going to a yoga class, taking up meditation, doing things that allow you to realize that you are an embodied creature, and then starting to listen to how your body is feeling. Having a daily practice of checking in with yourself can automatically and immediately start to open you up to being able to do creative things. If you combine that with the daily practice of journaling or drawing or something else that allows you to focus those feelings and externalize them in some way, very quickly you’ll discover you have an artistic side you never knew was there.