Process Engineering at a Furry Convention
- 35 min read - Text OnlyLines are slow
Long ago: Lines were long and slow.
Lines suck, and it is a reoccurring joke at nearly every convention.
Everyone comes for the fun:
Not for the lines.
September 24th, 2022: Branded merchandise that was intended for attendees to pick up at the April convention became available, and a serious effort was made to distribute outstanding owed merchandise to attendees through local community ambassadors. I became one of them and gave out bags and posters to those listed at a monthly bowling event.
January 24th, 2023: Fur Squared publishes hotel room block reservation links. Usually, conventions open attendee registration prior to hotel reservation. This time, registration was not ready. Others involved said in passing that the existing registration system was difficult and did not meet expectations.
January 26th: I sense that the head of technology is stressed and ask how they are doing. We end up in a Discord call and use VSCode Live Share for several hours to unblock registration.
Every year, this convention has a theme and a poster where, by selection lottery and an upgrade fee, attendees could have their character appear on the themed poster. Registration was not properly displaying and supporting lottery entry before I got involved.
January 29th: Pre-registration is officially opened, thanks to my work.
January 30th: I sign up for a hotel room. Later that day, I am on-boarded as Staff in the organization because attendees are able to register, and the revenue is enabling the convention to proceed with purchases and so forth.
Early February: Dealer sign ups is broken, and it needs to be fixed. Nearly every furry convention has a "Dealers Den" where independent artists, sole proprietorships, and even small businesses can have booth to vend to attendees. During this time, I get more familiar with the code base and find it insane.
The Foundation
Doing some volunteer software development for an event registration system.
We cant have nice things. Stripe acts differently between sandbox and production.
Also the original authors have multi-way multi-level django form object inheritance. This insanity also applies to all their view rendering code.
:dying-inside:
Each badge is generated dynamically, and this year's design has different positions for names than last year. This required an update to the canvas code. I also had a feeling that the printers would not faithfully print the images. So, I requested I have one or two printers to test with and calibrate the images and printers for the convention.
February 18th: Director of registration hands off two CR-80 card printers.
I spend literally days trying to restore and update a Windows handheld tablet, the only Windows device I have, with the driver for this printer. Wow, Microsoft has really sh*t the bed with their UX since I last tried. The Windows 11 upgrade thing is so bothersome, and I go through this extra download just to be told my hardware is not compatible.
February 20th: I finally get the printer ready to print images contained in a PDF and my worries were confirmed.
The greens in the above badge were not faithfully printed.
Ignore the bleed area on the digital image and the printing artifacts. The printer had a bunch of cat hair inside at first.
At first, I try to compensate by cooperating with the artist, Slade the Aqua Dragon, to adjust the background contrast. This helped, but Slade had enough after a few rounds and I never got the PSD to adjust it myself.
February 21st: Eventually, I find out that if I set up a Linux machine with an eleven year old printer driver, I have more options on calibrating the color intensity, sharpness, sublimation temperature, and so on. The results were far better. I also started toying with PDF417
Ideally, an operator could scan the badge when they give it to the attendee, and other processes could use it to track rewards that are given to attendees.
Then, I ordered the cheapest 2D code scanners on Amazon. They came with incompatible cords. With effort, I forced them to be compatible with the cords provided and ordered some more scanners to test with, including orb looking ones.
I accidentally ripped a pad off 🫠
So I tried the trick I saw on YouTube where I scratch the trace open and clear out the surrounding ground
Thankfully the pad nearby was floating so I could use that for something firm to affix a wire to.
Will try to fix the second unit some other way. This way was too risky. But at least I understand the problem now.
Hint the problem is lack of quality control in China where they mixed parts without checking wire compatibility
Another one fixed to work with the cable provided.
I think I could just do a keystone to short rewire on the outside too. Though the lock in rigid cable aesthetic is lost.
Early March: I searched and tried out many Text-Based UIs (TUIs). I briefly burned out while mocking up some. The next weekend I decide to do a simple json configuration file and a Python script that shells out to lp and lpstat to submit print jobs.
March 9th: I recommended we purchase more scanners and that we get YMCKOK ribbons. This will let us print with full color on the front and in black and white on the back. This way, we can print both sides without burning two-cards worth of color for each card, and it takes one fourth the time to print the backside.
I love and hate printers of all kinds.
Why arent you working?
Cups pending
Anything more helpful?
Cups pending
Dmesg? New usb found 40 mins ago!
Uh var log? Printer connected 40 mins ago
Unplug, plug back in, power cycle--nada
Please...
Delete printer
Re-add printer
Correct all the defaults
Submit job
Oh now you decide to work.
Wondering if it was just cups having a conniption
Wow. Incredibly, if the printer jams, the driver and printer can get into inconsistent states and the driver has to be restarted. Indeed, this problem also affected Windows, with a recent driver release. Unlike Windows, I did not have to restart the whole computer just because one printer jammed. I could just restart CUPS!
Truly, the world of printers is cursed, and few are interested in improving that experience.
March 12th: The print client I've been working on is able to distribute work to many printers over USB and I get aquatinted with Python AsyncIO. It reliably handles printers going offline and bouncing jobs mid-print and retrying individual cards onto other printers.
Tools meant for the job
Badge printing was the biggest choke point of the registration process. Now that it was automated, I could improve every dependant process. Badge check in and reward pickup were dependent on a cumbersome django administration editor.
March 16th: I began design of the registration check in application. Card stock and permanent markers are my go-to for UI design. They're like white boards that do not get erased. Quite handy.
March 18th: I finished an early version and I figured out how to effectively capture and handle USB HID 2D scanner input in a web application.
The application was built with some Tailwind CSS and Alpine JS. It really is quite simple to make a Point-of-Sale style interface.
I was still trying to keep the QR code that the artists added and instead rely on the PDF417 code. Cheap scanners are apparently bad at scanning one code among several, even with a configuration booklet. And PDF417 codes are... really really poor at information density, and did not scan well since the 2D code is wide instead of square to a scanner with a square sensor. I really wanted to replace the QR code above.
If operators or attendees had to cover up one code on their badge, the instructions, confusion, and compliance would take away all the time savings I could get.
March 19th: I returned the printers to the registration director. In past years, badges would be printed before going on site because of how stressful it is to print on site. I demonstrated the print distribution functionality and later that day, the director decided we would print on site instead of a week before the convention.
March 22nd: I created another frontend to track the convention store rewards. I borrowed a lot of code for the search and view transition functionality.
For local testing, I printed grey scale badges on card stock at the wrong size. At least it worked and I could test the workflows in physical space with scanners and touch screen laptops.
March 23rd: I proposed a modification to the badge where the QR code on the front moves to the back and the front QR code is made useful for the convention operation. The artists agreed with my proposal.
Became:
Everything is wrapped up and I made convention store rewards configurable for every membership.
March 26th: I figured out how to send MIME multipart/alternative emails with text and HTML content. After that, I added functionality to send a registration ticket to attendees. This email specifies which line someone should enter on site and includes a QR code.
The QR codes could have been smaller, but I found that some scanners might upper case or lower case inputs and I found that to be a risk I'd rather not face on site. Therefore, each unguessable token embedded in the QR code is encoded in Base 32.
March 29th: Registration ticket emails were sent out. I switched from SQLite to PostgreSQL, because I kept hitting contention with the print queue functionality and registration.
April 2nd: I fixed some upgrade cost inconsistencies. It would say $65, but actually charge $55. Also made it so an administrator could directly upgrade someone with credit or cash on site.
The Convention
April 6th: Zeroth day of the convention. Attendees can pick up badges, though there are no convention events to attend.
April 7th: First day of the convention. Badges were reorganized to be on-demand (newly registered as of that day) and registered as of the prior day. Although this decision made the email instructions inconsistent with the operation, it was not much of a problem with a simple sharpie-drawn sign at the fork.
I got on the poster again this year and they were available for pick up at the Dealers Den next to the rewards.
The operator was also the director of Dealers Den and had this to say:
Gotherine Foxx
We were functionally done at 3pm as opposed to 6:30 pushing 8
This honestly was a game changer.
We would normally have to stop at 5:30 and turn people away for a line in the morning at open.
We got 80% of swag passed out on Friday I thought that I undermade [swag] bags. I had to check attendance numbers.
I was like sh*t did I make enough bags? This is looking awfully low for like a number of bags that I made. I'm like did I get the data wrong? Please? Tell me I got it is the data wrong is the data skewed did we like add on like 50,000 super sponsors like last minute and I'm not getting the data in time because I was giving stuff out and I was like sh*t just going way too smoothly. I'm waiting for the failure, you know deficit to kick in where's the failure?
April 8th: Second day of the convention. Today, the shirts are available, and all the "super sponsors" are eligible to get one, which you'll see in the graph shared above. I finally experienced the pick up process myself, and it was light and frustration-free on the customer side. My transaction took 25 seconds in total. Scanning and record keeping was only a mere three seconds.
There is so little work to do that next year staff will likely have shorter shifts or even be reassigned. Everything worked so well because the people could focus on problems that only people could handle face to face.
On Thursday, around 125 people were processed in 30 minutes at the peak. That is 41 people per laptop, with an average of 43 seconds per attendee.
April 9th: Last day of the convention. Not much to report, everything continued to work. I helped tear down afterwards and pack things up. One of my scanners went missing, but it was the cheapest one I repaired.
The big changes
Here's the approach I took. We have a financial and operational obligation to check people in, to reward their attendance, and to spend as little time as possible doing busy work like record keeping.
There's a joke that we should pretend our users are drunk. In this case, it is not only hungover, it is with four hours of sleep, and possibly with disinterest in the process. A furry convention is meant to be fun. For some that involves drinking; for others it is talking with friends until 5 AM. Staff are there to have fun too, and as operators, the software has to accommodate their circumstances: tired, drunk, and interested in other things.
Attendees are bad at following instructions. Especially verbal instructions that have to be remembered a few minutes later. Requirements must be obvious, up front, and directions must be clear, concise, and easy to find. If an attendee has to be told to do something surprising, that takes time for them to process and follow an instruction.
For example, when I was testing with PDF417 codes, I could not see us saving this much time if the operator had to tell the attendee to cover the QR code on their badge. Also, the PDF417 codes just did not scan or print reliably.
The operators had touch screens and 2D scanners. Just about everyone knows how to point and press a button to scan something they know: a QR code. By eliminating ambiguity with only one QR code on the front, operators could smoothly scan the QR codes to mark the badge as handed out or to search the badge to see what rewards they are owed.
I also put the scanners in registration on stands and configured them to continually scan. No trigger button press required. This way attendees could swipe their phone under the scanner and the operator never had to touch the scanner or the computer. They could just scan the badge then hand it to the attendee. The physical transmission of a card from one party to another has to happen. I merely piggybacked on that process to quicken the record keeping. Work with your natural requirements to make a fluid and intuitive process.
Furthermore, zero ambiguity reduced training and enabled operators on shifts I never saw in action to effectively use my software.
I also freed up several people from the printing process. Once all existing attendees were queued to print, I turned on auto-enqueuing. This way, anyone that registered as of that moment would automatically have their badge printed without delay. This was an entire person's job: monitor and print newly registered attendees. The person who was doing that could now focus on people, not technology.
Another job: managing the printers. By making a fault tolerant and gracefully-degrading system that load balanced cards among the printers, during times of less traffic, literally hours could go by without maintaining a printer and everyone was still served.
Technology exists to make things easier, not to be paperwork but less flexible. Indeed, when it truly makes operations easier and become invisible, everyone involved can benefit.
Takt Time
It is not just about shortening processes, it is about shortening the ones that matter most. If the queue backed up to where those doing vaccine checks could not push attendees to badge pick up, they could not do their work. Likewise, if vaccination checks backed up and badge pickup were idle, there is wasted opportunity.
We must take into account how much time it takes for each work item divided by the amount of work that has to be performed, and allocate enough resources to balance the demand with prior and later stages of the process. This consideration is called "Takt Time" or just "Takt."
During the peak moment, we had 3 printers automatically printing badge cards, which took 40 seconds per card. Next, vaccination had 2 workers and vaccination check 20 seconds per person. Last, badge pickup had 3 stations and it took 43 seconds per person. Printing had a Takt of 13.3s, vaccination had a Takt of 10s, and badge pickup had a Takt of 14.3s. With the resources we had, these Takts are close and permitted a fluid process where little backing up or stalling was possible.
The same cannot be said for Midwest FurFest.
MFF could do better by allocating more towards vaccination checks and ensuring that internal processes for reward disbursement (like folding shirts) are balanced with the demand (handing out shirts).
I also believe that the semaphores that directed attendees to available stations could maximize station use by queuing at least two people per station and each station operator signal with a light when a station was ready to have more line up.
Conclusion
Checking in attendees and handing out rewards are the processes that I worked to improve. The results were measured and overwhelmingly positive. At every moment I designed, I considered the physical process that had to happen and made the most natural and least invasive solution to accompany that process. I considered the time it would take to convey and follow instructions between people, and reduced ambiguity. These processes all involve queues and dependencies on prior processes. I focused on reducing the time it takes to communicate and process people in a limited physical space. The results were clear: no queue backed up so far that it impeded prior queues (such as vaccination checks), where as this happened in prior years. Every staff member on shift were effective with their time and not once blocked on another.
My process was more effective per person than the biggest furry convention in the world. Printing took 40 seconds and could happen before or during vaccination checks, which happened under 20 seconds and with an average peak check-in time of 43 seconds to verify identity and give a badge, the effective blocking time for each attendee being serviced by staff is about 1 minute and each step had enough resources to have no choke points. Effective planning, allocation of resources, and empathy for every participant involved can help you land an effective process at your event too.
Bonus picture of "Ginji" ギンジ🐾ターラノー