AWS Chicago Summit - 2022-08-27

A few weeks ago, my employer's account manager invited my team to join them at the AWS Chicago Summit.

AWS Chicago Summit web page

We were given sufficient notice for the event (1 month) and so I talked to the CTO about it. He requested an expense estimate so I did my research and proposed it. I got approval and had no issue reserving rooms for myself and two others on the team.

Turns out we got the closest hotel to the event! In contrast, our AWS team instead were placed a mile-walk away.

Wednesday Afternoon

After several meetings over zoom we grouped together to car pool to Chicago. Some make a pilgrimage to the Chicago IKEA for the experience. Instead, we went to Micro Center.

Shelves loaded with cheap keyboards

If you walked into a Micro Center thinking they'd have stuff for a home lab, you would be mostly wrong. There was some Unifi stuff, but nothing I wanted. Otherwise tons of cheap switches that my ISP experienced friend said are the worst.

Not that I expected to find it, but I'm on the look out for this product. Just about everything is marked up 5-10x and this little thing is no exception.

overpriced unifi door bell chime

The rest of the store was... overpriced consumer junk, a section for apple hardware, a hobby center, PC parts, and vibrating gamer chairs.

Honestly it did not live up to the hype and we all left with nothing.

I get dropped off to check into the hotel but I get lost trying to find it. By the time I do find it, the others had finished parking.

Once I get into the hotel room I see a familiar thermostat. The trick I used at DEF CON worked here too!

Wednesday Evening

Our account managers invited us to a bar and we went up to some exclusive party floor. Drinks were free so I have a few hard seltzers and I talk to our account managers about what our interests and challenges are, what our business is facing in the economy and what our resources look like. I hear that there's some new Redshift instance type where compute is separate (finally?) from storage.

Time passes and it is time for dinner.

The account managers looooove when they can take customers out. The budget for "customer engagement" or whatever is super high. Dinner for 7 people? The bill with tip was like $495. Whoa.

I heard stories about how broadcasters are trying to use AWS to stream real time to PoPs. Someone from our side nudged on egress costs being high. They agreed under the cuff and mentioned enterprise contracts exist.

It is late, we head back to the room and the thermostat VIP mode reset while we were gone so we flipped that back on.

Just in case I leave this on twitter. Only 110 people saw it.

Cendyne

Anyone going to Chicago aws summit? I won’t be as obvious to spot but I will wear a badge saying “Talk to me about cryptography!”
If you want to meet up DM me

Aug 25 2022 03:53:28 UTC

Thursday Morning

One of the others went through registration and said it went quick. I and the other head out with a light load and check in.

Wow, absolutely wow. The registration space was maxed out. They absolutely wanted a high throughput horizontally scaled registration experience and it shows.

AWS Chicago Summit registration hardly has any people and plenty of free desks to check in at

When I entered, by the time I was queued there were only three people ahead of me.

I was blown away by that experience.

But the next one, big oof in my opinion. I scan my QR code, the scanner confirms with a chime. Nothing happens. Me and the desk person awkwardly wait and then she said to try manual. I try pushing the button to scan a qr code and then I succeed.

Amazon: I expect you to learn from your product experiences. Act on intent! Such rigid application flows show they haven't watched their product being used in production. By scanning a QR code it should have skipped that button press.

We briefly meet with our account managers at the startup corner. I get a nice thermal mug and we wander around a bit more for swag.

Afterwards we all head back to our room to put our stuff in the car since we are not staying an extra night. Then with a light backpack I head back to the convention space.

Turns out if you want to have a good time, you must install their app. The website does not offer any map information or location of specific event information.

AWS Keynote

We head towards the keynote, people with light sticks guide us to chairs and whoa that's an amazing wide display.

Cendyne

Space shuttles are constrained by the historic width of two horses.
Choose the place with the least constraints

Aug 25 2022 15:10:36 UTC

There was plenty of "build on us so you can do your thing, we won't limit you like horse butts will."

And every time they transitioned between speakers they played a light show program that blinded the audience. I was blinded four times.

Cendyne

AWS Chicago summit keynote lights are blinding the audience.

Aug 25 2022 15:28:25 UTC

I hear that events can get fined for pointing lights at the audience...

The talk by cenvent was interesting. They showed how they managed to pivot their in-person event business to support online event operations. It was kind of a sales pitch as well.

The other keynote speaker was from some mortgage and loans company and they did not review their slides for readability or flow like the smooth AWS speaker slides. In fact, some slides were solid red and our eyes just burned.

A very red slide which reads 1 peta byte of data

Panels at AWS Summit

The first one I went to "Securing 250 accounts: Maintain guardrails for self-serve cloud adoption". That story was "If we actually gave a platform team enough resources to enable an organization with 300 other developers, we would have a good time". As the creator of the platform team at my employer, I was excited to see how others did things and I was a little jealous that I could not realize the same scale they had. After all... at the peak (before the big workforce reduction), there were only 25 other developers and I had 3 to work with for platform needs.

I leave to the next one and I found the layout of this convention center to be really confusing. On the lower floors there's a bunch of empty restaurants and then more convention rooms and a big area in the back where I was headed.

I was not sure what to think of a bunch of these green glowing headphones on the chairs. I have to check in for this talk before heading in and they say to leave the headphones on the seat to know they must be cleaned after the presentation.

The speaker began to speak. There were no speakers. I was less than ten feet away but it was still a little difficult to hear.

So I put on my head phones and hear nothing.

alt

Half way through, the audio-visual (AV) guy puts up a new transmitter on stage and then I put the head phones back on. Now I can hear him... and a bunch of analog noise. The speaker's cell phone kept interfering with his transmitter and the AV guy's mixer had a ground loop that constantly buzzed our right ears.

Sure was distracting. But anyway the talk was about open source observability with OpenTelemetry. One of my team members was with me and he mentioned how honeycomb is using this stuff. Afterwards when he asked other Application Performance Monitoring (APM) vendors about it they had no idea what he was talking about.

Like any convention, this one required lots of walking. I head right back to the expo for my next one, a lightning talk of sorts on "Accelerating your business with protected amazon S3 buckets."

He introduces himself as a "Field Engineer" and goes on about this use case with Fox Sports. Third parties upload 8k footage to their bucket and they do not want to handle viruses when they scale down to 4k. Okay, so this isn't some IAM talk. The speaker is from Trend Micro so you can guess where this is going.

fox sports has partners upload to s3

glare
For PCI compliance, we have to show daily anti-virus (AV) scans of our production system. Running ClamAV inside of a container with cron.d is so hacky and totally unneeded. But we have to check a box.
How do hackers like me get into systems these days if you have AV there to protect you?
ceiling-watching
heard-you-were-talking
Server Side Request Forgery (SSRF) is the new attack on the block. The attacker does not need to get a file on the system to compromise what the system or application can access. By capturing and acquiring tokens from a Security Token Service (STS), an attacker can perform every action that process can on any resources it is given permissions to.
interesting
This is a silent attack. Logs for this kind of activity is usually a trace or info event, rather than a warning or error. That's why these other things like GuardDuty are catching on. The cloud provider may notice unusual patterns and reveal them to you–for a fee.
Thanks! I'll keep that in mind.
hackerman2
ych-bite
AWS does not do enough to encourage reduced access. Human provisioned IAM roles typically include permissive AWS managed policies. Many breaches can be attributed to permissive access controls.
shame
"Good practice" from AWS is that each developer gets their own AWS account, that every micro service has its own AWS account, that production and non-production are in separate AWS accounts. Many businesses do not follow this "good practice", so the blast radius for a compromise is quite wide.
Why do you only use one account if it is not good practice?
watching-through-window
corporate-drone
Because we were tied to a managed service provider that resold AWS services to us and they would charge $5000 per account per month for even the smallest alerting or cost optimization. That dangerous sales policy discouraged us from following good practice.

Fox's prior AV took minutes to run and this was a disaster for realtime events like sports. In my experience: updating AV definitions and starting ClamAV with 2.5GB of RAM takes over a minute to start up just to scan a single file.

What did TrendMicro do?

They "reduced the scans from minutes to seconds" by reducing the work performed on Fox's cloud.

trend micro shows how they introduced scanning

Literally the "scanner" is just streaming the file to a message digest and then submitting the hash to Trend Micro's web service.

i-guess
Is that really all AVs are? All it is doing is hashing a file and seeing if it is in some remote set of known hashes?
i-guess2
The best this will capture these days is the EICAR test file. I would think ransomware would be targeted and individualized to each victim. Even changing the date when something was compiled or packaged in its metadata will produce a new and never before seen hash.

After that's over I head to the next thing on my list. A "Chalk Talk" as it was called. I wander around until I figure out where it actually was. Again, the layout of this place was confusing.

There's no line in front of the room, the doors are closed, and there's people standing in front with a smile. I ask if I can go in and he says no, but I can try the exercise online. My team was inside so my company won't be missing much by my absence.

screenshot of slack where my team members report suffering from the container talk

Oh. Never mind.

A three hundred level panel turned out to be a baby talk about containers and how to make them.

They exit and I meet up with them and head back to the Expo.

I split up from them since I see that lunch things might be happening and I get the second to last lunch box available. Unfortunately one of my team members missed out.

alt

I head back with 10 minutes before the next event and... Oh no. It's a Chalk talk again.

I get in line and someone counts and marks the end of the line

When I get in line, half of us get through and someone else with a sign comes up and says "This is the end of the line, sorry but we are out of capacity."

exasperated
Registration was scaled to be the smoothest thing ever. Only the keynote and the quiet events (with those head phones) venue had enough space. Everything else was packed, noisy, and unable to fit everyone.

Frustrated, I head back to the expo for a backup talk, how to hire at amazon's scale. I could not hear three out of five words, even with me using my hand like an ear trumpet.

On my way to the next talk, I pass by the green stage again and someone is presenting to only four people. This started to feel sad.

Around the same time, one of my team members comments that less people are enjoying the AWS DJ than the fascist furry convention.

Meanwhile, here is some karaoke from said fascist furry convention.

ych-trashcan
I do not support fascism, bigotry, nazis, racism, or the like. I'm just pointing out that even a disgusting fringe community is having more fun than this AWS conference.

Anyway, I am heading to the next amazon talk on innovation at scale. It was interesting to hear some inside stories about how their processes evolved and the like.

Got to see a tangerine flywheel. At least it wasn't the infinity sign or some business exec eye candy that trivializes the work that happens.

Flywheel for amazon focuses on improving product and expanding products

One section talked about the customer and their needs. Then off the cuff he starts doing a total techbro ideation. It was so out of touch with reality as he focused on one of these stock photos. What if we could solve this imagined problem in a way that is completely disproportionate in cost to the end user?

Who is the customer

He goes on about micro services and hiring people, then mentions team size and how they go with some two pizza model. Then he clarifies that two pizzas means 10-12 people.

Two pizza team

kneading-dough
Over the course of my employment at my current employer, I have seen my company grow to 300% since my start date and then cut down to 40% of that after continual organizational mismanagement and market change. Given what I saw with how well people functioned at what counts of people, I do think there is merit in the "two pizza team" idea.
macro
Three re-organizations and many series of brainstorming sessions did not solve our core mismanagement problems. We were pressed into committing to 500% of the work with a 100% workforce increase.
drawing
After the workforce increase, half the engineering org was new and did not know the domain. Also half of the org were contractors in Ukraine with varying English competency. To work around the remote team's lack of independence and language skill, the product team shifted to micro management on Jira and called it "agile". This "agile" stuff required more communication.
todays-interesting-thing
With so many people talking to one another, supporting one another, it seemed like we could get more work done. So much peer to peer exchange was happening in our organization. Lots of new content flew through Jira and Github.
ych-noises
Every team member was subjected to so much inaccurate and useless noise. To solve this, management introduced more process. This raised performance for the remote team and reduced performance for our full time engineers. Eight out of ten of the team from before the workforce increase left because of growing dissatisfaction.
melting
We achieved the opposite of what we thought. Nearly every project stalled and only four of eighteen months were collectively effective. Only four months of collective work delivered anything! Much of that networking between peers lead to context switching, fatigue, employment dissatisfaction, and failure to meet individual and team expectations.
content
The two-pizza team size should constrain the noise and should bound the scope of what a team commits to. A happy team is one that meets its own expectations.
tired
Our failure was not setting constraints, saying yes to too many things, and completely draining existing resources to projects to train and enable the new workforce on the existing product, while expecting progress on existing projects.
ych-slap
We had no experienced resources leading new products. We had no support from leadership on anything that ran over three months. Management and leadership could not displace experienced resources from their existing responsibilities. Outside of my team, there is no redundancy in technical ownership because scope creep has compounded for years.
It sounds like scope management is an incredibly important skill. Oh and turning down the noise.
ded
nod
Both are the most important skills I have developed in the last two years as a manager.

Amazon learns from failures

Near the end of his talk, he mentions that failures happen too. At first Amazon tried to compete with ebay and that failed. Eventually Amazon arrived to the marketplace model we have today. Some failures do not get second chances though, like the Fire Phone.

Thursday Evening

I was wanting to go to one more, but I was invited to join our account managers at some taco pub called "Broken English."

While waiting between talks, I was reading an Overlord light novel. It was hard to read. I felt some anxiety building up because of the two setbacks I experienced and my disappointment with the conference's content.

I concluded that even if I attended another panel, I would not be in a state to learn from it.

My team met up with our account managers back at the startup booth and we left together in a few lyft / uber rides.

Street art of masked bears

At first I thought this was some street art, but actually inside the walls are lined with these masked bears–which I initially thought were furbies.

photo of inside the taco bar with more of the masked bears

The event was hosted by Circle CI and some other services company called Caylent. But we did not mingle much with either Circle CI or Caylent, mostly just our account managers.

I think I saw the tablet rack up to over $4800 in tacos and alcohol here. Well at least my team member who missed lunch got like 9 tacos.

The technical account manager talked about his experience with a community owned apartment building and how New York was instituting an environmental grading system which can impose fines. Just having lights in the hallways takes about 800/yr in electricity.

The other account manager was quite aggressive in getting others to eat tacos and drink. I declined a fourth margarita, I felt like I was at the edge of my safe limit. But they ordered one anyway.

So I drank it, and my vision and balance was very difficult afterwards. The next few hours I have to drink plenty of water to remain aware.

A jug of water and ice colored with pinks and purples from the mood lights

I took a good portrait of one of my team members here, but given how personal that is, I'll just share this water which had similar aesthetic value given the lighting.

We order an uber back to the hotel. I had to be very deliberate about my walking. Turns out our driver was in a Tesla. This was my first time ever riding a Tesla. It sure could accelerate.

We get out and start driving back home. Along the way we get some gas and things smell funny.

We get to the next gas station and it wasn't the fuel cap being cross threaded. I sit in the back still sipping water while conversing with someone about PQC in TLS libraries on Slack.

They buy a funnel and some coolant and top off the radiator. It seemed better after that.

By the time I get to my car, I was in good shape and I make it home around 10 PM. The next day I work.

Conclusions

This conference was very vendor and sales heavy. The ticket cost was $0 and like how Facebook is free due to ads, it shows. I did not write about it, but every conversation with a vendor leads to your badge being scanned. They scan your badge to get your email and phone number.

If you are a manager, a CTO, a CISO, then this conference is for you. If you don't know the vocabulary or technologies available in the cloud space, then this conference is for you. If you need a lead to solve your problem with a product, then this conference is for you. This was not at all a conference for developers.

I brought that up to our account manager and they might be able to get some Re:Invent vouchers for my team. But I would not be able to go.

Cendyne

Might get vouchers for reinvent, but it conflicts with MFF so I would not go.

Aug 25 2022 20:59:23 UTC

I saw that several vendors have really developed their products over the last 3 years since I last took a look at them. I think the technical sessions are there bootstrap an excited manager so they can come back to their team to know how to talk about these products. I do not think that these sessions were in any way meant to empower these managers to perform the work themselves.

It was nice to meet up with our account managers. But because we dropped that managed service provider (MSP), we will be losing the account managers we've gotten to know in person. They are tied to that MSP.

Cendyne

AWS Chicago Summit is over.
To my dismay, the event was targeted primarily to my job title and those above my job title.

300 level: Do you know what a container is????

My two devs left that one after 15 minutes. It was so painful for them to sit through an intro course.

Aug 26 2022 01:31:10 UTC

Cendyne

Since the admission price was $0 to the event (not the hotel), my guess is that this is just a sales and networking conference for management and those who show interest in migrating to cloud offerings.

Would I bring my dev team to this again? Probably not.
Me and my boss? Yes.

Aug 26 2022 01:32:42 UTC

Cendyne

Did anyone talk to me about cryptography?

No.

Did I see anything implementing competent cryptography?

Well, Trend Micro is hashing stuff in s3 to see if exists in their db. But that was it.

Aug 26 2022 01:35:42 UTC

If I had the opportunity to go again, I might; but I would invite different people.