Hiring: False negatives in the bigcorp

October 29, 2006

I’ve written a couple of times about why I don’t believe in the “No False Positives” view of hiring. Among other reasons,

o I work in an area (doc classification) where we make precision/recall (false-positive vs false negative) tradeoffs all the time, and I know that there is _always_ a tradeoff

o I believe that many of the ways to reduce your risk of hiring someone you shouldn’t have hired will increase the conventionality of your hiring. You’re more likely to miss a great but unusual person.

o Some of the very companies that promulgate the no-false-positives idea have made some pretty stupid hires IMHO 😉

But most importantly,

o Since your positive mistakes (bad hires) stick around, but your negative mistakes (people you should have hired) disappear, it’s much harder to learn from the negative mistakes. So it’s natural to come to believe that only positive mistakes matter.

Unless, of course, you work for a bigcorp like I do ….

Last week I was at a two-day meeting of everyone at my company who is using machine-learning techniques. At one particular presentation, I found myself thinking “This is a really good talk! good stuff…. who is this guy? familiar name … where do I know that name? Oh, right! that’s the guy we passed on three years ago!” This isn’t the first time this kind of thing has happened, either.

Depending on how your bigcorp does hiring, it’s possible that some people you reject will find a place elsewhere in the same company, and you can get just a little visibility into how your “rejects” do. When the reject turns out to be a rockstar, it’s worth reminiscing for just a moment about why you passed, and (if you were sure) why you were so sure….


SPoFs and SPoCs

August 29, 2006

In systems design, a Single Point of Failure (SPoF) is a component that brings everything down if it fails – for example, a single master server machine that is your only way to communicate with a cluster of child servers. Having a SPoF in your design is supposed to be a Bad Thing.

In organizational jargon, a Single Point of Contact (SPoC) is a person who represents a group to another group with respect to some project or issue. Having a designated SPoC is supposed to be a Good Thing, as it cuts down on confusion about who to talk to.

But … isn’t a SPoC also a SPoF? Hmm.

Simulations as thought experiments

April 16, 2006

Thinking more about what annoyed me about Norvig’s post on hiring at Google, I realize that it’s not that involved a computer simulation of hiring – it’s that it pretended to take a simple simulation seriously, like it was, well, science or something.

I don’t have anything against quick-and-dirty simulations – I actually think that they can be one of the most fun kinds of computer programming. You do what is essentially a thought experiment, and then you get to see your assumptions played out in front of you. You say “Hey, what if A’s hired A’s, and B’s hired C’s, but A’s made the occasional mistake and hired B’s instead – what would happen?”

Of course, though, it’s very hard for simulations to prove anything about the world – all you are seeing are your own assumptions and their implications, brought to “life” in front of you. Simulation programs like this are to thought experiments what calculators are to mental arithmetic – they don’t change the activity, they just make it a lot more efficient. And the worst way to use them is when you already kinda know what conclusion you’d like to reach (like that your company’s hiring philosophy is a good one)…. because usually there are enough input parameters to juggle that if you don’t get the right answer the first time, you will eventually.

No doubt the root of my feeling of distaste is just that I spent a long long time in grad school studying old-school AI, and this is exactly the kind of program that AI specialized in for a while. Some parts of AI had an engineering focus, and the programs were judged on how useful they were; the rest was essentially computer-aided philosophy, with the thought experiments helpfully instrumented with print statements. I actually don’t have anything against this kind of philosphizing and model-building, and I do think that programs can play a really cool role in showing where your assumptions are leading you astray. It’s just closing the loop back to anything about the real world that’s problematic.

Silly Google post on hiring

April 4, 2006

I was disappointed to see this post by Peter Norvig on the Google Research blog – disappointed because it’s so silly, and because I have had feelings of near-reverence for Norvig in the past (based mainly on his AI Programming book from back in the day).

What’s wrong with the post? Well, it’s one of those facile little pieces that takes a complex reality (technical hiring) and assumes enough of the interesting bits away that the rest can be made to look like a simple physics experiment. It’s the kind of thing that makes you want to shout “SCIENCE!” (as in she-blinded-me-with….).

In the first part of the post, Norvig demonstrates with a simulation that a higher standard of hiring (hiring above the mean) leads to better employee quality than a lower standard (hiring above the min), when both strategies are given an inexhaustible supply of candidates, and “interviewing” has zero cost. The first question you have to ask is: why would anyone be surprised by this, or predict anything different? And does anyone seriously propose hiring-above-the-min as a good idea?

And I’m curious: if hiring-above-the-mean is better than hiring-above-the-min, then why hasn’t Google tried hiring-above-the-max (i.e. insisting that each new hire be better than every current employee)? My initial investigations show that this simple strategy can make the quality of new hires approach 100% very quickly (although the simulation runs are taking a really long time to complete for some reason…)

In the second part, Norvig does another simulation to demonstrate that doing away with hiring managers is a good idea (hires are made by the company as a whole and then assigned to teams and projects later). The assumptions underlying this one are interesting. First of all, hiring managers are assumed to be completely greedy hirers, who will never fail to fill a position even if they have no candidate that’s minimally suitable. That’s an … interesting model of trust, shall we say. (I would have to agree that any individual hiring manager who acts that way is probably bad for his/her company, so if we assume that all hiring managers act that way, then I would agree that they’ve all got to go. (Disclaimer: I’m a hiring manager.))

But the more interesting implicit assumption is: there’s no cost to removing all informational connection between projects/teams and hiring. For one thing, fitting candidates to teams or projects is a non-issue by definition (because candidates are reduced to a single “quality” score, equally good for any project or team). For another, there’s no connection between the amount of information you can share with candidates in job offers and your success in getting candidates to accept those offers. (I am happy to say that I know of at least one counterexample: a game-changingly inventive and productive guy, currently working at Yahoo!, who decided to take the Y! offer over a “better” offer from G, because Y! could tell him something about the projects and co-workers in advance…)

I’m not even saying that I’m sure that decoupling projects from hiring decisions _isn’t_ a win – I’m just sure that if you simulate an inherently tradeoffy proposal like that by including all the benefits and none of the costs, then your simulation will probably tell you that it’s a really good idea. (Or as Norvig puts it: “We’re pleased that these little simulations show our hiring strategy is on top.”)

Anyway, if you personally would like to have your talents, abilities, and interests mapped to a single score at application time, and then agree in advance to join a “pool” of interchangeable workers, with no prior information about what you’ll be doing or who you’ll be working with, you should know that the Google hiring philosophy can make your dreams come true. On the other hand (shameless plug time), if you’d like to talk to different teams doing significantly different sorts of work, and then make an informed decision about what you’d like to work on and who you’d like to work closely with, drop us a line at Yahoo! Search. Each to his own, right?

(Disclaimer: in case you couldn’t guess already, I work at Yahoo! Search, but am not speaking for Y! here.)

How wifi changes the workplace

February 26, 2006

Following up on both my last post and Jeremy’s, I’ve been amused and interested to see how wifi (and other connecting technologies) has changed my working environment – especially in ways that no one was anticipating when they constructed the building…

I work in one of those buildings that have floors full of cubicles, with conference rooms at the edges. There are very few offices (even very senior execs have cubes, and sometimes even share them). There are implicit assumptions in this architecture: that most people will be working away most of the time on semi-isolated work, but that groups of people will also congregate occasionally in the conference rooms and talk (aided by things like whiteboards and projectors). People in the cubes don’t want to be bothered by the talking, so the conference rooms have doors, etc.

I’m an engineering manager, so I spend more time lately in the rooms than in the cube. And very recently I started doing what all the other people in that situation have been doing for years: taking a wifi-equipped notebook with me everywhere, including to meetings. Maybe because it’s new for me, I’m seeing a lot of funny little phenomena – mostly good, some bad, but a couple just odd.

o It no longer annoys me at all when I show up on time to a meeting and no one else is there yet. I just work, sending email and whatnot, until there’s critical mass and things get started. (Good)

o If I’m one of the late ones, I’m likely to walk into a room holding a bunch of people, all looking at their notebook computers and tapping away, not talking to each other. These are people who have all agreed to meet at this time and place for the purpose of talking to each other. (Odd)

o Sometimes someone has to come to a meeting who is not really involved, but is there to answer questions about their special expertise. This person can just work (or… whatever they’re doing on that machine) until the expertise is needed, and then jump in. (Good)

o People who actually need to pay attention get sucked into their machines. They can be at least as tuned out as someone who is dialing in. Sometimes it’s necessary to say: “Time to close the notebook”. (Bad)

o When I meet with my manager, and he realizes as a result that he needs to send someone a 1-line email, he does it then and there. I’m doing this now too. (In a former life it would have been a reminder note-to-self on a scrap of paper.) (Good)

o If we’re in a room with a projector and everyone has a machine, you can swap the cable around (I know – it’s clumsy), and anyone can drive and show anyone else anything on the intranet, docs, personal email, their local disk. (Good)

o Every so often you pass by a conference room, and see a solitary person on the phone. What are they doing? Either having a confidential conversation (job interview?) or more likely “dialing in” to some meeting in a different building, city, or continent. Alternatively, you’ll see and hear someone sitting in their cube and talking loudly and at length as they dial in to a meeting. I don’t think anyone anticipated this when they built the office – neither cubes or conference rooms are the right thing. (Bad).

o Every so often, you pass by a conference room, and see a solitary notebook-owner in there, not on the phone. What are they doing? Working. It’s a nice, spacious, quiet office, though unfortunately temporary. (Odd)

Meetings, 2

February 26, 2006

In an earlier post (In Praise of Meetings) I ranted about the unfortunate fact that, although meetings undeniably suck, they’re also irreplaceable for certain kinds of decision-making. As a subrant, I griped about the fact that having people “dial in” to meetings is a lose, since the communication bandwith is just too low (this observation from a person who is usually at the meeting in question, not dialing in). The combination led me to be dubious in general about working remotely and telecommuting (at least if you have the kind of job where you need to be at meetings).

A recent post of Jeremy’s (Where Almost Doesn’t Matter) woke me up a bit – possibly I was blaming the victim? Jeremy argues (citing Andrei Broder) that the problem isn’t necessarily that people are dialing in rather than physically present – it’s that some are present and some aren’t. The people who are actually there are doing a lot of nonverbal communication which leaves the dialed-in people out in the cold. The proposed solution is to have meetings be either all-present or all-virtual (as Andrei says was the practice at IBM). This is one of those simple, brilliant, obvious-in-retrospect ideas.

Jeremy more generally proposes that maybe “where” doesn’t matter anymore, especially when you’ve got phones, wifi, Treos, etc. I dunno. It sounds like the people he cites are doing a lot of roving around, which is made possible only by their electronic connectedness. If so: why do they have to rove around at all? Why aren’t they all working from home all the time, and saving their gas money? My hunch is that what being unchained from your desk makes possible is talking to more people face-to-face in more various places, which is really about greater personal control of the all-important Where (and the all-important face-to-face contact), than it is about making Where disappear.

Michigan recruiting trip

November 25, 2005

So last week I travelled to Ann Arbor, Michigan to recruit for my company. I’ve pounded the table for a while about needing to broaden our college presence beyond the first two coasts, and so I got to put my time where my mouth was. I took this particular one because I grew up in Ann Arbor, and my parents live there.

First, though a rant about the university/private-sector interface. The career office at Michigan emailed us to tell us what the interview rooms are like. I emailed back to ask about whiteboards, wifi, etc. They replied saying “yes, we have whiteboards, but you have to bring your own markers and erasers. We’re not funded for that.” I was incredulous (talk about coals-to-Newscastle!), but I packed up our own personal whiteboard eraser into my checked-in luggage. (Don’t ask why we have a whiteboard in our dining room, OK?) I envision some underfunded university bureacrat scouring line items for _anything_ that hiring companies could be made to pay for instead. (And pay they do. Michigan charges a sum I will not disclose for the privilege of a campus recruiting visit — I much prefer the honesty of this to another public university which will remain nameless, who asks companies if they’d like to “collaborate” on faculty research, and then bars them from on-campus recruiting if they don’t figure out the game and cough up… but I digress.)

Tuesday: I know why airlines like to have you on the plane all ready to go when there are weather delays. But man, do I hate waiting on the tarmac. An hour of delay in SJC, then an hour in the plane before takeoff, then more delay in O’Hare, then two more hours on the tarmac waiting for the go-ahead. I finally make it to my Ann Arbor hotel at 2am.

Michigan in November … my colleague A. has cause to complain (having grown up in NW India), but I was _born_ there, and was once a hardy Northlander. Hardy no more – soft, weak, and whiny instead. Anyone complaining about a thirty-degree day in November deserves to be told “Well, enjoy these nice days, because _winter’s_ coming”. But there’s no substitute for a long slow autumn of getting used to it – step off the plane from somewhere warmer and you naturally think the residents are insane for living there.

Wednesday: Infosession at Michigan, and it is one tough room. Not tough like hostile, but just 40 or 50 unresponsive engineering students staring at us blankly, spread out over a huge classroom. The HR people go first, riffing on Y! in general, then my partner A. and I talk about web search. We have silly trivia questions and swag giveaways, which work a bit in breaking the ice. A’s and my questions are aggressively geeky (how many trailing zeroes in 100 factorial? – someone gets it about two minutes in). Finally the crowd loosens up, and we have good conversation about the search competition, the Valley, etc.

Then in the evening, a victory for the Treo + Yahoo! Local! A. and our two HR friends are looking for somewhere to eat, having had no time to hit the hotel and change out of our Y! T-shirts. I know the town, but from so long ago that all the restaurants are different. So we hit a random bar, and I ask Y! Local for a French restaurant nearby. Score! A good time had by all.

Thursday: Due to scheduling weirdness, this is an off day (i.e. work from the hotel). Also, coincidentally my Dad’s birthday. So I bring my good friend and colleague A. over to my parents’ for dinner. Now, you know how weird it can be when people you know from different parts of your life meet? You want them to like each other, but how sometimes it just doesn’t work out? Well …. it wasn’t like that at all! Everyone got on famously.

Friday: A. and I haven’t beaten the jet lag yet — still sleeping until 9PST/12EST, and this is the day that we pay for it. First interview at 8:30 AM (5:30 PST). My first one is 25 minutes late, and walks in saying “Sorry, man, 8:30 comes really early y’know?”. Tell me about it. But the asymmetry does help — I’m sorry, but interviewing 8 people in succession is just easier than being interviewed by 8 people… All in all a good crew of smart young people – A. opines that the quality is better than previous day-long visits to Stanford and Berkeley.