At my house we have a Valentine’s Fairy 🧚🏼 (also a Christmas Eve Fairy, but that’s for another day)! This year, I have a child in my home for the first time and am excited to introduce her to this special fairy. As I was preparing, I decided I wanted to crochet a Valentine’s dog. After looking all over the internet for patterns or kits, I found an image I really liked but couldn’t find a pattern for it.
Isn’t it cute?? But I’ve never done my own pattern before. Could AI help?
After some basic prompting, asking for a crochet pattern from a picture, I wasn’t quite getting the results I wanted. I decided that I could start with having AI help me with the prompt itself then use that to get the pattern.
Monday I wrote about expertise and prompts. I was having ChatGPT help me write a prompt, and I noticed that I brought some expertise in teaching and learning into the process but ChatGPT also was adding its “own” expertise and filling in details, adding more specificity to my general statements.
Monday I was working in a domain that I knew well–and could see the expertise it was bringing to the conversation. I wondered, though, if someone without as much expertise in teaching and learning (such as a student training to be a teacher) would benefit as much or be able to produce a prompt with this embedded expertise.
Now was my chance to experiment with something I don’t have as much expertise in (though have some basic knowledge). My process was:
- Ask ChatGPT to write a prompt that will tell an LLM to create a crochet pattern for an uploaded picture
- Test and revise, using the original ChatGPT chat to help me modify the prompt (see image below; the prompt showing on the right was after several iterations)
Based on my initial experience trying to get a pattern, I noticed that it often missed parts of the dog, made lots of the parts a basic sphere, etc. I also have some basic prompt engineering knowledge and experience with LLMs. This knowledge impacted my starting prompt:
You are an expert prompt engineer. Use expert prompt engineering techniques to write an LLM prompt and open it in canvas. I want to write a prompt that will tell an LLM to create a crochet pattern from a picture. It should look really carefully at the image, including the shapes, ratios, sizes, and parts. It should think carefully and deconstruct the pattern* before creating it. It’s important it gets the proportions and shapes of the parts correct–it shouldn’t just make everything a standard circle.
*Note: I am not expecting that it will actually fully analyze shapes mathematically–but the hope is that it will cue it to pay closer attention to those details
The first prompted created by ChatGPT added more specifics to the ideas I had. For example:
I wrote (from my initial experimentation): It’s important it gets the proportions and shapes of the parts correct–it shouldn’t just make everything a standard circle.
ChatGPT included in the prompt: include any variations in structure such as tapering, asymmetry, or oblong shaping
I wrote (from my knowledge of prompt engineering): It should think carefully and deconstruct the pattern before creating it.
ChatGPT included in the prompt: Before generating the final pattern, take a structured, step-by-step approach to analyze and deconstruct the image thoroughly:
- Outline a preliminary pattern structure before proceeding to detailed row-by-row instructions.
- Break down the key components of the object into its basic shapes (e.g., spheres, cylinders, flat panels, cones).
- Assess stitch types and shaping methods required to maintain accurate proportions, including special stitches where necessary (e.g., bobble, loop, short rows).
- Consider construction order and assembly methods to ensure smooth transitions between sections and minimize seams.
What is a bobble? Apparently this.
ChatGPT added on its own:
Follow a structured format similar to professionally published amigurumi patterns.
- Include a Materials section listing the yarn type, hook size, and additional tools.
- Provide an Abbreviations section with standard crochet terms.
- Clearly format each pattern section, such as Head/Body, Ears, Legs, Tail, etc.
- Offer assembly instructions if the pattern includes multiple components.
And more I won’t go into here.
Explicitly Asking It For Expertise
At one point, I asked it if there are other things that I could add that could improve the prompt:
Is there anything else I could add that would make it more accurate?
So my knowledge–that ChatGPT may be able to offer expertise that I don’t have–allowed me to bring in more expertise.
Adding Sources
Eventually I realized that it might help to give it some sources to study and build on. The easiest way to do this is through a custom GPT, rather than just a prompt. At this point I decided to go ahead and do this, pasting the prompt into the “Instructions” box and uploading PDFs of sample patterns. I decided not to share it at this time (see the ethics discussion below).
Is This Basically Just Another Form of Prompt Engineering?
As I was working and observing what I knew and didn’t know, what it brought to the process, and what skills I was using, at some point I wondered how much basic prompt engineering knowledge was needed to do this. I have some basic understanding of what makes a good prompt. I also have a sense of how LLMs work and potential quirks.
But it felt more like I was designing, while someone else was doing the “engineering.” Like an architect that might make design in collaboration with engineers who work out the details.
Questions, questions
Is this Ethical?
I felt a bit of ethical fuzziness in this process. I was asking it to learn from what other experts have shared online–they shared them for free (at least the sources I shared, not sure about the training data), but I was putting them into another tool. I doubt they planned for this use.
The image itself was watermarked, but I couldn’t find it anywhere else on the internet and just really, really liked it.
As a human, if I crocheted (or studied) all the different patterns on the resource sites, I may get to the point where I could create the pattern myself. Is it fair to ask a computer to do the same thing faster?
Revising too many times?
At one point it seemed counter-productive to continue to revise the prompt. I tried opening a new chat to ask for revisions:
I created this prompt. It keeps having two problems: It creates numbered lists instead of bullet points when doing the patterns, and it doesn’t make arms and legs long enough. How can I fix the prompt? Please open in canvas.
This did help (with the bullet points), though I still struggled with making arms and legs long enough.
How much are LLMs doing this behind the scene (such as in the deep-thinking models)?
Is It Good?
This I don’t know yet. First, I have to share the picture with the Valentine’s Fairy. I’ll update you on how she does!
Update 1
I have just started to try out the pattern. I actually ran the prompt through Claude.ai as well as ChatGPT, curious on how they would compare. Overall, I liked Claude’s results better, but it missed many of the parts (the tail, ears, etc.) in the first run. Both LLMs seem to struggle with the irregular shapes–I’m looking at other patterns myself and making a lot of adjustments as I go. I was hoping it would be really good at this, as it is with recipes, but the translation from an image to a pattern doesn’t seem to be there yet…anyone want to invest in a new company that trains the model on lots of crochet patterns to make a better tool?
Update 2
I gave up on the LLM pattern. But I did find a great pattern on Etsy! Here’s the finished product.
