PSN'ing Me Off - Sony's Failure to Secure The PlayStation Network

Let us establish first and foremost the basic position in which Sony now finds itself with the gaming public: they are fucked.

Some of you will doubtlessly protest my very strong language and crude imagery regarding the current situation with Sony's PlayStation Network.  Some of you probably expect nothing less.  Either way, I stand by my assertion.  The scenario that has played out couldn't possibly be conceived, even with assistance from LSD, salvia, shrooms, and mescaline all mixed together.  DDOSing the PSN, that sort of thing should be something that any network engineer, security oriented or not, ought to be factoring into their designs when they build something like this.  But this has gone way beyond a mere botnet or script kiddie attack.  Somebody, or a group of somebodies, didn't just shut down the PSN the way that Anonymous "accidentally" did a few weeks ago.  They broke in and made off with user data.  How much user data?

Try all of it.

There are, best estimate, some 70 million PSN accounts.  Those accounts contain names, addresses, and most importantly, credit card info.  And every last bit of that data was taken.  This is light-years above owning a box on Sony's network.  It's like the Great Train Robbery, only considerably worse.  What could you do with essentially unfettered access to 70 million credit and debit cards?  Depends on how smart you were about it.  The best part, from the perspective of the hackers, is that Sony has actually helped them get away with this.  How so?  By not owning up to the fact that they got hacked, and not owning up to the fact that personal data was lost.  Because Sony sat around with their thumbs up their asses, putting out milquetoast "updates" which informed without actually enlightening anybody, and ignored the rising degree of protests far longer than they should have, they essentially covered for the hackers.  Their prevarications have given those guys at least a week's head start to play around with other people's money.

One thing that should be kept in mind at moments like this is that it really is smart to avoid ascribing malicious motives to certain actions which can be better explained by basic stupidity.  Consider Patrick Seybold, the Senior Director of Corporate Communications and Social Media for Sony.  It's tempting to paint him as an outright villain, a corporate mouthpiece stooge who propagated a farrago of lies by repeating over and over, "we don't know how bad it really is" for six whole days.  But it's perhaps more accurate to look at him as being stupid.  The less flattering view would be the typical suit, a guy who is in the habit of talking a lot but not really saying much of anything, which might go over well in the boardroom but tends to make your customers start hauling out their pitchforks and torches.  The more forgiving perspective would be a man who was given the mushroom treatment by another segment of his company and used as a human shield for a week.  Continuing up the food chain, we have the engineers whose balliwick the PSN falls under.  Again, real tempting to paint them as evil bastards.  Again, much better to look at them as exercising gross stupidity rather than genuine evil.  In a corporate environment the size and breadth of Sony, the size of a problem is proportionate to the speed with which one's CYA reflex kicks in.  A tiny little problem, nobody will give it a second thought, just fix it and forget it.  A bigger problem, say an authentication issue for the East Coast for example, and you can be sure there's some CYA going on before the problem actually gets fixed.  When you've got a problem like the current one, everybody will be on the verge of panic trying to figure out how their posteriors can be sufficiently shielded, even as the small vestiges of their brains still capable of coherent thought inform them that there isn't a snowflake's chance in Hell they can make anything relating to the disaster look good.  Fiascoes like this one tend to lead upper management to demand people's heads, and heads will be served up one way or the other.  If the engineers weren't feeding Seybold any genuinely useful information, then it's certainly understandable why Seybold's blog posts weren't assuaging the public's discontent.

I would like to take a moment to address another example of stupidity, and one that has far more potentially damaging consequences.  It is the stupidity of complacency.  The stupidity of "don't worry, it's not a big deal."  To some extent, Sony gave us this brand of stupidity over the course of the last week, and it's turned out that we shouldn't just be worried, we should be all sorts of pissed off and justifiably scared.  An article on Ars Technica had some choice words from Michael Pachter, an analyst at Wedbush Morgan who has seemingly made the current stage of his career focused on "analyzing" the video game industry.  And by "analyzing," I mean "spouting mindless bullshit and getting paid six figures for it."  In the past, I've done my best to avoid giving much thought to Pachter and his inanities, but his pronouncements in regard to Sony and the PSN breach just cannot go unchallenged.  The first mistake is playing the "shit happens" card, stating that security breaches do happen and it sucks for customers.  Sony wasn't even stupid enough to try and use that gambit, which doesn't start Pachter off on solid footing.  Yes, security breaches happen, but in regards to the PSN, security breaches DIDN'T happen.  Outages, yes.  Authentication problems, more than Sony would probably like to admit.  I know that no system is 100% secure and no system can avoid being breached forever.  The PSN was probably the closest thing to an impenetrable system that Man has devised in the last decade.  When it finally was breached, it was ripped wide open and the really valuable data, the personal user data, not the games, was sucked out like marrow from a cracked bone.  The "hassle of tracking down whether somebody is fraudulently using credit info" which Pachter breezily dismisses isn't the sort of annoyance that can be dispensed with by clicking a mouse and re-entering some data.  Assuming for a moment that the spread of credit cards stored on the PSN is evenly split up between the 4 major credit card companies (Visa, MasterCard, American Express, Discover), then each of those companies is looking at dealing with seventeen and a half million cards that need to be cancelled and re-issued.  It's likely not such an even split, but the company who's only handling four or five million card cancellations probably won't be feeling suitably grateful for the distinction.  That's going to tie up massive amounts of resources which would otherwise be pointed towards day-to-day operations.  The ripple effect on the economy just from having to process all those cancellations beggars the imagination.  Even if it's handled at a lower level through local banks and credit unions, it's still eventually going to impact the operations of the credit card companies.

Pachter continues to show his absolute lack of anything resembling intelligent thought when he made the following pronouncement: "In my view, a serious hacker with evil intent would be better off hacking into a financial institution rather than a gaming network."  He continues to diminish the scale of the disaster by concluding that the breach is "not a serious security threat."  If I were a serious hacker with evil intent, directly hacking into a financial institution would be the last thing in the universe I'd want to do.  It wouldn't matter to me if it was the Last National Bank of Zimbabwe.  Shooting for a direct breach of bank data would be unbelievably stupid and ultimately profitless.  Banks have been directly robbed so many times in physical form over the centuries that they tend to design their computer systems much like they would their branches.  Lots of security fences, lots of redundancies, lots of alarms.  Banks expect people to try and straight out rob them, so they harden themselves accordingly.  True, they can still be breached, and user data can be obtained, but banks will go berserk the minute a breach happens and they will be locking down everything related to the breach very quickly.  If you're lucky, you'll have about 24 hours worth of use out of that data, then it's pretty much wasted hard drive space.  Rather than hit the banks directly, hacking a game network would allow somebody to come at them sideways.  Remember how I asked what you would do if you had 70 million credit card accounts, all the personal data associated with those accounts, and a week's head start?  If I were the smarter version of Pachter's hypothetical "serious hacker," I'd be making relatively small money transfers.  A cash advance here, a direct withdrawl there.  Keep the limit down to a C-note at a time.  Even if I could only pull it off one time each for 10% of the accounts that I snagged, that's still 7 million accounts, and a Benjamin from each one of those accounts would add up to some serious money.  Banks look for big money transfers into and out of individual accounts.  Somebody shows up with a hundred million dollars and says, "I'd like to make a deposit," you can bet there's a manager on the phone to the Feds before the ink's even dry on the deposit slip.  Small money transfers, on the other hand, it's background noise to a bank.  A modicum of caution while pulling money out and putting in, nobody would have any reason to suspect anything, certainly nothing that would justify filling out a Suspicious Activity Report.  And if I were being extra smart about it, there would be a mix of ATM withdrawals and electronic fund transfers.  Shift a C-note to the bank of my choice, pull it out a few hours later, and the cash is mine.  I could go on about how ATM cameras would be recording me, but if I'm smart enough to have planned and executed a plunder on this scale, dealing with ATM cameras would have been factored into my thinking and a suitable countermeasure developed.  Bottom line: a gaming network is the perfect vehicle to rob a bank, because nobody will see it coming.

As my high school forensics coach told me oh so many years ago, it's considered good form to concede at least one of your opponent's points during a debate.  And while I firmly believe that describing Michael Pachter as a halfwit is overly generous praise, his little chat with Ars Technica did produce one point which I can agree with.  "Over the long run, we'll all forget about this, unless it happens again."  Perhaps not entirely accurate, but close enough.  The brouhaha will eventually die down, people will be fired, and life will return to something resembling normal.  How quickly things return to almost-normal, and how close they come to the established benchmark of normal prior to the breach, depends very heavily upon what Sony does next.  The smart thing to do would be complete disclosure.  Let the world see how thoroughly they fucked up and how badly they got taken.  Make sure that the conditions and the environment which allowed the breach to happen do not recur.  Sony needs to be crawling on their hands and knees over broken glass coated in lemon juice and salt to win their customers' confidence back.  Even then, it may never quite reach the level of confidence that they once enjoyed.  The question is how to prevent a new breach from happening.  If the hackers got in through a hacked PS3, what would Sony do?  Update the firmware to further cut off functionality?  Brick every PS3 currently out in the world and make their customers buy all new ones just to rebuild the integrity of the PSN?  Both of those options would almost certainly exacerbate an already infuriated customer base, as well as give hacker groups like Anonymous more grist for their mills.  Until Sony discloses how the hack was pulled off, it's exceedingly difficult to say how best to proceed.  Continuing to do what they've been doing for the last week is guaranteed to make the situation worse.  "Proactive" measures which somehow result in a further diminished user experience for the PSN when it finally does come back up will have the same effect.  For the immediate future, Sony is fucked as far as their customers are concerned, because there is nothing they can do that won't piss people off even more.  Even SCEA's board committing seppuku on YouTube wouldn't make people happy.  Sony will just have to take their lumps and contemplate the scale of repairs needed not only to the PSN, but to their reputation and their customer base.

A Year Without A Paycheck

One year ago today, my job unofficially ended.  We were taken aside, one by one, told we were getting severance, and then got walked out of the building.  Officially, we were still employed till the 15th of November.  I can't say it wasn't a terribly big surprise.  And to be fair, we'd been essentially sitting on our asses and getting paid for the last couple of months prior.  Still working, but not nearly as much as we had been.

Funny how time slips away, isn't it?

A year later, I'm still out of work.  The job hunt has been a bigger challenge than at any other time I can think of, even worse than when I moved back to Phoenix ten years ago.  The economy is in the toilet.  The tech sector which I've had a career in has become a hell of a lot more picky about hiring.  In fact, every sector has gotten picky.  Even temp work is hard to get these days.  Over the last 365 days, I've had precisely one temp job lasting six hours, which was a couple weeks ago.  I've sent out more resumes and applications than I can easily count.  I've gotten dozens of form emails essentially telling me I didn't get the job.  I've been ignored by dozens more.  I've had headhunters tell me they can get me work, and I've heard a recruiter tell me I'm screwed.  There have been folks out of work longer than me who aren't getting work and there are folks out of work for less time than I have getting snapped right up.  Part of me would like to get out of the tech sector.  Part of me knows I have to get back into the tech sector before I can move out of the tech sector.

It hasn't all been doom and gloom, though it sometimes feels like it.  I met a wonderful woman, the Otaku Girl, who prodded me to put up this blog.  I've met a lot of very interesting people at events I probably wouldn't have met them at previously.  I got to see a lot of things that when I was working regularly I never could have seen.  In some ways, I've been living more in the last year than I did while I was working for a living.

Still, I'd much rather have a job.  Something that lets me have a good work-life balance.  I'll take the paycheck, but I want to be able to enjoy it as well.

The hunt goes on.

You Don't Know Me And That's How I Like It

Recently, Bitmob put out an article postulating what would have happened if Blizzard had pounded RealID through above the objections of its customers.  I like the fact that they called shenanigans on the weak arguments most people were bandying about against RealID, but I dislike the fact that they didn't commit an equal amount of effort to the weaknesses of Blizzard's arguments for RealID.  Allow me to make the arguments that Bitmob neglected to mention.

First, there is the implied argument that Blizzard is doing a mitzvah to their customer base with RealID by exposing the trolls, ostensibly shaming them into good behavior by revealing their real names.  As the webcomic Ctl-Alt-Delete so eloquently demonstrated in this strip, revealing the identity of a troll is no deterrent against the behavior of a troll.  The flaw in Blizzard's theory is that trolls are capable of feeling shame when it is amply demonstrated, time and again, that they are incapable of that.  There is not a single iota of evidence to suggest even the slightest hint of remorse, regret, or shame in the behavior of a troll.  They're petty, small souled, simple minded, and cretinous, which means that they go for the simple pleasures, the easy hit.  They don't care who gets offended or what they get offended about so long as somebody gets offended and knows that it was what the troll put up that caused it.  They feed off the recognition that they punched somebody's buttons.  They're bullies, and they're a particularly obnoxious form of bully because they can't be smacked down like the Neanderthals that shake down kids for their lunch money.  They're confident because they feel safe in the knowledge that they cannot be touched, and so they cannot properly suffer the consequences of their actions.  Being simple minded, when somebody does manage to somehow verbally slap a troll down, the troll will not just stop.  They're incapable of learning more than one lesson at a time.  They might fade back for a bit, then they'll be back punching buttons again.  Eleven million plus subscribers to World of WarCraft would have had their names exposed, the vast majority needlessly associated with their alter egos, in order to pursue a futile attempt to punish a tiny majority in a way that will completely fail to deter them.  There would have been no happy ending with that course of action, nor will there ever be a happy ending with that course of action.  Until the population of trolls genuinely outnumbers the population of decent folks on the boards, whatever monetary cost savings are made will be lost in terms of customer backlash, cancelled accounts, and future sales losses.  As strange as it may sound, not even Blizzard or WoW is immune to the masses.  All it will take is one issue, one position stated too strongly, one policy adoption that offends the common core of a large enough percentage of the subscriber base, and it will trigger an avalanche of defections that the company might not survive.  Consider the example of Facebook.  They've been pissing off a lot of people over the last year or so.  One too many changes, one extra little line, or one unclear clause buried the boilerplate of the TOS, and Facebook stands to lose not only subscribers but substantial revenue.  The same situation applies to Blizzard.  Making people believe your hype is a perfectly acceptable business move.  Believing your own hype is a recipe for disaster.  Despite what Blizzard and the rabid core of Blizzard's fan base might believe, WoW is not the only game in town, even if it currently is the biggest.

And now we come to my second argument that Bitmob should have thought to make.  While many would argue that even in an MMO, customers have a right to privacy, I will argue that one has a right to anonymity.  The distinction might seem lost on some folks, so allow me to elucidate.  As I've said before, MMOs are very much like amusement parks.  There are a lot of activities that you can do within the park, but you as a player are coming into an environment where you have no direct control over anything except your avatar.  You influence nothing within the game.  You can make changes to yourself which ultimately have no practical effect outside of how you look and what kind of rides you can go on.  Yet when you're at the park, other people know you're there because they can see you.  Other players are aware of, or can be made aware of, your presence.  Whether you're grinding mobs in The Barrens, spamming in trade chat in Ironforge, or simulating some Night Elf-on-Gnome action in the tunnels of the Deeprun Tram, awareness of your presence in the game simply cannot be completely hidden.  Proximity to other players, even in the shady corners of the Deeprun Tram, constitutes most players' awareness of each other.  Global chat channels, friend lists, and guild rosters further add to the sign every MMO player wears around their neck saying "Here I am!"  Privacy in MMOs, at best, is a relative sort of thing, and it's fleeting.

Anonymity on the other hand is a little different, and something that should not be in the hands of any company, not even Blizzard.  Anonymity is the choice we make to acknowledge our presence to other people within the MMO.  Consider Mila Kunis or Curt Schilling, very famous people who are avowed WoW players.  If they want to advertise the names of every toon they run, that's perfectly fine.  If they don't want to, also fine.  The critical component is that they choose if and when to connect their toons to their real identities.  Yeah, it's fun talking about Family Guy or the place of free agents in baseball while you're doing a ten man raid on Icecrown Citadel, but it's not why we fork over $15 a month.  The fact that we want that level of remove, that layer of insulation, between our virtual names and our real names isn't a reason for suspicion, nor does it indicate nefarious intent, nor does it even suggest we're trolls in player's clothing.  Of all the choices one can make in an MMO, the only one with any true significance is whether or not we give somebody our real name.  If somebody wants to put their real name in for their toon, whether for vanity or lack of imagination, fine and well.  If somebody wants to come up with a completely different nom de guerre, also fine and well.  Once you make that connection public, however, you're going to have to rely on the imperfect fleshy memory of people to forget that connection.  It for damn sure won't fade away on the Internet.  The ability to control our identities, for good or ill, is perhaps the fundamental right of the 21st Century.  The ability to moderate, granulate, and compartmentalize who we are goes right to the very heart of our concepts of self and identity, whether it's physically or virtually.  And Blizzard has no business trying to usurp that ability, nor do they have any basis to demand their customers surrender that ability just to play games they develop.  To an extent, they can and do refine that identity just a bit, but they do not have any commercial or financial justification for breaching the divisions we make between our real world selves and our virtual alter egos.

Thus ends my arguments.  Good job otherwise, Bitmob.

Stute Developers

When I was a kid, my folks picked up a copy of Rudyard Kipling's Just So Stories.  It was not the complete collection, and as I found out later it was a somewhat sanitized version of it, but a lot of the well known stories were there like "The Elephant's Child" and "How The Leopard Changed It's Spots."  After reading an interview with Jamil Moledina on Ars Technica, another of those stories comes to mind, "How The Whale Got His Throat."  In the story, the Whale gobbled up virtually every fish in the oceans except for one, a "Stute Fish," who suggested that the Whale try having Man for dinner, though he did warn the Whale that Man was "nice, but nubbly."  The Man which gets eaten by the Whale turns out to be a Scot and something of an engineer, and manages to not only get out of the Whale's belly but also manages to keep the Whale from ever eating any fish again by rigging up a grating in the Whale's throat made from a pair of suspenders and a rubber dinghy that the Scot was floating around on in the middle of the ocean (because his mom told him he could).  The Stute Fish goes and buries himself in the mud somewhere along the equator to hide from the Whale.

I'm reminded of this story because despite Moledina's impressive resume and the generally concise interview he gave to Ars Technica, I'm not convinced that EA's "EA Partners" program is anything even remotely helpful to the average indie game development crew, possibly because I have the distinct feeling that EA's definition of a indie game developer is considerably different than what most people would use.  How many truly indie developers are out there with dev kits from Sony, Microsoft, AND Nintendo?  If we're talking about the hand-to-mouth garage developer, the one who's using whatever freeware and open-source tools he can legally obtain, and probably pirate copies of 3DS Max or Maya if they're not feeling real picky, chances are that even getting one dev kit constitutes a major coup on their part.  For the small team still in college, pretty much the same story.  Once you're big enough to be able to get those dev kits, you're not really operating on indie cred anymore, and you've probably managed to make enough coin to afford to pay people a little money.  By EA's definition, "indie" seems to be synonymous with "not currently signed to or owned by a publisher."  And with that definition in mind, the concept of EA Partners gets ominous, because it feels disturbingly like an offer from a Mafia don.

"Sure, we'll help you get your product on to the Big Three.  But one day, we're gonna come to you with a favor, and that day, you're gonna owe us."

Moledina's evasion over the question of IP ownership with the EA Partners program sent up a great big red flag for me, and it should probably do the same thing for any developer who might be considering this. While the interview references a statement Moledina made at the Gamesauce conference, there is a gaping hole in the statement that sounds very strange coming from a guy who ostensibly knows as much about game development and the way the industry works.  The quoted statement was this:

""It's an odd thing, because we continue to see and hear from developers ... that they're being forced to give up the IP.  Publishers are not that good at taking advantage of the IP unless the original creative team is involved."

What's wrong with this picture?  Could it possibly be there is a paradoxical, or at the very least dichotomous, nature to the statement?  Or might it be the unspoken truth that whether or not publishers are good at taking advantage of IP, they'll still yank it away if it looks like it's making money?  While Moledina goes on to state that developers shouldn't be afraid to "keep what [they] deserve," it's deeply troubling that he will not acknowledge even the possibility that the publisher will behave badly and take over the IP against the wishes of the developers.  Without even a tacit admission of this reality, or even the potential for this reality to manifest within the EA Partners program, very serious doubt is cast upon Moledina's assurance that EA is "very developer friendly."

All of the arguments that Moledina puts forth seem to hinge entirely on the assumption that an indie developer has neither the resources, nor the ingenuity, nor the clout to get their game out onto the consoles.  While it's entirely possible that some developers would fail on all of those criteria, it's also possible that such developers were never trying to meet any of those criteria to begin with.  Some of those indie developers are quite happy to develop for the PC and not have anything to do with the consoles.  Moreover, it occurs to me that if Sony and Microsoft and Nintendo really are getting serious about trying to bring high concept indie titles to their respective consoles, the devs are the ones who are going to be holding the whip hand in any sort of negotiations.  While the Big Three might have a lot of potential sources for new titles, the fact that they're reaching out to a developer means the developer has what the Big Three are looking for, and the devs are the ones who have the ability to modify the terms to suit them.  A shrewd indie will strive for a win-win situation, which will doubtlessly give them clout, which will make future negotiations easier.  The fact that an indie developer doesn't necessarily have the marketing department EA has at it's disposal doesn't mean that they're doomed to the purgatory of bargain bins and penny ante PayPal sales.  Any halfway competent marketer who knows exactly how to work social media can generate a lot of buzz for a game on a very shoestring marketing budget.  While an indie developer might not be cranking out million copy blockbusters, they aren't relegated to single digit sales numbers either.

The Stute Fish in Rudyard Kipling's story avoided getting eaten by the Whale by swimming alongside the Whale's eye.  The Stute developer can prosper by doing the same thing: staying by the eye of the big whales but staying well away from their maws.