Saturday, December 27, 2014

Mazes, rooms, and pathways

You can tell it's the holiday season, because I've been playing games and thinking about games, a lot.

So I thoroughly enjoyed Robert Nystrom's recent essay: Rooms and Mazes: A Procedural Dungeon Generator.

Nystrom takes us through a variety of different approaches to generating dungeons for adventure games, because it turns out there are a lot of things to consider.

For example:

You could make a roguelike with perfect dungeons, and many simple roguelikes do that because generators for those are easier to design and implement.

But I find them less fun to play. When you hit a dead end (which is often), you have to do a lot of backtracking to get to a new area to explore. You can’t circle around to avoid certain enemies, or sneak out a back passage.

Nystrom proceeds to explore two basic approaches to dungeon generation:

  • Start with a maze. Trim back the maze to leave holes in the space, then turn those holes into rooms.
  • Start with a bunch of randomly-placed rooms. Flood-fill the spaces between the rooms with mazes, then connect things together

As he puts it:

Where Buck and Karcero start with the maze and then add the rooms, mine does things in the opposite order. First, it places a bunch of random rooms. Then, it iterates over every tile in the dungeon. When it finds a solid one where an open area could be, it starts running a maze generator at that point.

The best part of the article is the illustrations. Nystrom fills the article with live demonstrations of various techniques, which bring the code to life and make it easy to understand the alternatives and how they impact the resulting generated dungeon.

If you've ever tried your hand at generating a dungeon (and what game-playing computer programmer hasn't?!), I think you'll really enjoy reading Rooms and Mazes.

Who knows? It may even get you excited enough to start writing that game you've been putting off...

Wasteland 2: a very short review

I've been whiling away many an hour recently playing Wasteland 2.

There's no doubt about it: this is a very fun game.

You and your party find yourselves in a post-apocalyptic wasteland, loosely modeled on the territory and scenery of southern Arizona, in a chaotic world full of mystery and adventure.

You travel about from place to place, take on quests, unearth clues, unravel mysteries, defeat bad guys, and generally have yourself a wild-and-wooly rip-roaring good time.

The making of the game is rather interesting, as revealed by this Wired article: How One Guy Got Kickstarters to Give Their Profits to Other Campaigns

Fargo went on to launch other games, including the now celebrated Fallout series, and even became a game publisher himself, as founder of Interplay Entertainment. But Wasteland was his baby, and by the early 2000s, he wanted to create a sequel to the game that put him on the map, called Wasteland 2.

The problem was, the studios wanted nothing to do with it. “I got nowhere for another decade,” Fargo says.

Then, in early 2012, Fargo stumbled upon a still up-and-coming company called Kickstarter that let creators raise funding from their friends and fans.

But you don't have to care about how the game is made; it's more fun just to play the game itself.

Now, where was I? Oh, yes, back to trying to negotiate with the Servants of the Mushroom Cloud...

Wednesday, December 24, 2014

Christmas Eve link clearance

Get 'em while they're hot!

  • On the Sony Pictures Security Breach
    Reports say Sony Pictures had trouble attracting and recruiting security talent, which isn’t too surprising for a company known for its disdainful attitude toward technology. Being on the wrong side of issues like SOPA/PIPA couldn’t have helped—what technologist would want to work for a company that is trying to break the Internet?
  • Did North Korea Really Attack Sony?
    The agency's evidence is tenuous, and I have a hard time believing it. But I also have trouble believing that the US government would make the accusation this formally if officials didn't believe it.
  • The Case for N. Korea’s Role in Sony Hack
    The “silent” part of the moniker is a reference to the stubborn fact that little is known about the hackers themselves. Unlike hacker groups in other countries where it is common to find miscreants with multiple profiles on social networks and hacker forums that can be used to build a more complete profile of the attackers — the North Koreans heavily restrict the use of Internet communications, even for their cyber warriors.
  • What Does "Responsibility" Mean for Attribution?
    Using the Spectrum of State Responsibility, in my assessment, the US government's statements include a range of possibilities, from State-encouraged to State-integrated.
  • Stupid Costly Patent Nuclear War By Microsoft & Apple Against Android Averted
    In short, this is basically Google and Cisco (with some help from a few others) licensing these patents to stop the majority of the lawsuits -- while also making sure that others can pay in as well should they feel threatened. Of course, Microsoft, Apple and the others still have control over the really good patents they kept for themselves, rather than give to Rockstar. And the whole thing does nothing for innovation other than shift around some money.
  • Why String Theory Still Offers Hope We Can Unify Physics
    Most gratifying, the mathematics revealed that one of these notes had properties precisely matching those of the “graviton,” a hypothetical particle that, according to quantum physics, should carry the force of gravity from one location to another. With this, the worldwide community of theoretical physicists looked up from their calculations. For the first time, gravity and quantum mechanics were playing by the same rules. At least in theory.
  • Cerebros and the art of drug smuggling
    This is not the work of a pickaxe army of drug war foot soldiers. These are multi-million-dollar underground networks, created covertly with professional machinery under the guidance of top-end engineers or architects who have been pulled—willing or not—to the dark side.
  • The Future: A Cat Litter Box and DRM
    I did some Googling, and I found that the “Smart” in SmartCartridge is that it has an RFID chip inside of it to keep track of how much solution it has, and once it runs out, well, you can't refill it. I honestly did not believe this and tore one of the cartridges apart, and there it was, looking back at me, a tiny chip holding up it’s little metal finger.

    Seriously CatGenie, you added fairly sophisticated DRM to a litter box? I’m a tad hurt you spent my money on building in a restriction instead of figuring out how to avoid constantly cooking poop.

  • Drobo vs. QNAP vs. Synology vs. G-Drive and MORE! Which Storage Should You Choose?
    To me, the ideal size for a NAS device is at least 4 drives. You can get them with as few as two if you’re really never going to use much storage, and if you are going to do that I’ve got a recommendation for you in a minute, but with 4 or more you can have plenty of space without sacrificing redundancy. And I’ve got three different options for you here. The QNAP TS-470 Pro, The Synology 1513+, and the Drobo 5N.
  • The Top Metrics You Need to Track to Improve Operational Performance
    At PagerDuty we’ve thought hard about what you should monitor and why from a systems perspective, but what about monitoring data on your operations performance? We’d like to share some specific metrics and guidelines that help teams measure and improve their operational performance.
  • Interesting papers from NIPS 2014
    NIPS is the premier conference on Deep Learning. Given the accelerating state of the art, it’s interesting to see what is new.
  • Testing TokuDB's Group Commit Algorithm Improvement
    As part of our effort to verify the new Binary Log Group Commit functionality introduced in TokuDB 7.5.4 for Percona Server, we wanted to demonstrate the substantial increase in throughput scaling but also show the bottleneck caused by the skewed interaction between the binary log group commit algorithm in MySQL 5.6 and the transaction commit mechanism used in TokuDB 7.5.3 for Percona Server.
  • Are We Consistent Yet?
    Traditional systems provide strong consistency, where clients can immediately view updates. Some distributed systems relax their consistency model to allow greater availability or better performance. Eventual consistency manifests itself to clients as stale views of data.
  • The Softsel Hot List for the week of December 22, 1986
    Back in the days before Internet-based software distribution, heck back even before the Internet existed in a form resembling what it is today, one of the most important ways of keeping track of the consumer computing industry was to subscribe to the Softsel Hot List, a weekly poster of the top sellers in various categories. Here is the Softsel Hot List for the week of December 22, 1986
  • Our local fill
    The original wetland that became Lake Merritt was known as San Antonio Slough. From Oakland’s earliest days, the locals kept trying to “reclaim” it by turning it into dry land, just as they did all around the bay. The whole waterfront is reclaimed land. The basic technique was to haul dirt and rock and rubbish down to the water, shove it in and tamp it down. In Gold Rush San Francisco they’d use abandoned ships for fill, but Oakland’s founding fathers had advanced beyond such crude strategems.

For my Christmas Eve meal, my grand-daughter made me hand-made crab-and-shrimp sushi.

Life is good.

Friday, December 19, 2014

Some links for understanding CVE-2014-9390

This is a particularly interesting CVE for me, because it's "right up my alley", as they say.

  • Git 1.8.5.6, 1.9.5, 2.0.5, 2.1.4 and 2.2.1 and thanking friends in Mercurial land
    Git maintains various meta-information for its repository in files in .git/ directory located at the root of the working tree. The system does not allow a file in that directory (e.g. .git/config) to be committed in the history of the project, or checked out to the working tree from the project. Otherwise, an unsuspecting user can run git pull from an innocuous-looking-but-malicious repository and have the meta-information in her repository overwritten, or executable hooks installed by the owner of that repository she pulled from (i.e. an attacker).
  • [ANNOUNCE] Git v2.2.1 (and updates to older maintenance tracks)
    Fixes since v2.2
    ----------------
    
     * We used to allow committing a path ".Git/config" with Git that is
       running on a case sensitive filesystem, but an attempt to check out
       such a path with Git that runs on a case insensitive filesystem
       would have clobbered ".git/config", which is definitely not what
       the user would have expected.  Git now prevents you from tracking
       a path with ".Git" (in any case combination) as a path component.
    
     * On Windows, certain path components that are different from ".git"
       are mapped to ".git", e.g. "git~1/config" is treated as if it were
       ".git/config".  HFS+ has a similar issue, where certain unicode
       codepoints are ignored, e.g. ".g\u200cit/config" is treated as if
       it were ".git/config".  Pathnames with these potential issues are
       rejected on the affected systems.  Git on systems that are not
       affected by this issue (e.g. Linux) can also be configured to
       reject them to ensure cross platform interoperability of the hosted
       projects.
    
     * "git fsck" notices a tree object that records such a path that can
       be confused with ".git", and with receive.fsckObjects configuration
       set to true, an attempt to "git push" such a tree object will be
       rejected.  Such a path may not be a problem on some filesystems
       but in order to protect those on HFS+ and on case insensitive
       filesystems, this check is enabled on all platforms.
    
    A big "thanks!" for bringing this issue to us goes to our friends in
    the Mercurial land, namely, Matt Mackall and Augie Fackler.
    
  • 3f1509809e728b70ea7912e4e1b40f22965e45ee
    fsck: complain about NTFS ".git" aliases in trees 
    read-cache: optionally disallow NTFS .git variants 
    path: add is_ntfs_dotgit() helper 
    fsck: complain about HFS+ ".git" aliases in trees 
    read-cache: optionally disallow HFS+ .git variants 
    utf8: add is_hfs_dotgit() helper 
    fsck: notice .git case-insensitively 
    t1450: refactor ".", "..", and ".git" fsck tests 
    verify_dotfile(): reject .git case-insensitively 
    read-tree: add tests for confusing paths like ".." and ".git" 
    unpack-trees: propagate errors adding entries to the index
    
  • Git vulnerability with .git\config
    Normally the git client avoids ever overwriting that file. Even if you commit a .git\config file and push it to a shared repo, no one else’s Git client will check it out into their private repos. However, a bug was discovered where various permutations of the .git folder name (e.g. mixed case, gIT, GiT, etc, Windows filename shortening .git~123, Ignorable Unicode codepoints .g\u200cit\config, etc) were not caught be the Git client’s filtering logic. As such, if someone pushed a malicious config file with one of these permutations, other people’s Git clients would check them out, overwriting their personal config file and hijacking their Git commands. This affects, at least, Windows NTFS and Mac OS X HFS+ filesystems, both of which are case insensitive filesystems.
  • Vulnerability announced: update your Git clients
    Repositories hosted on github.com cannot contain any of the malicious trees that trigger the vulnerability because we now verify and block these trees on push. We have also completed an automated scan of all existing content on github.com to look for malicious content that might have been pushed to our site before this vulnerability was discovered. This work is an extension of the data-quality checks we have always performed on repositories pushed to our servers to protect our users against malformed or malicious Git data.

Some resources for trying to understand the SPE hack

If you're trying to make sense of it all, here are some fairly well-reasoned essays, from a variety of smart people:

  • North Korea and the attack on Sony Pictures Entertainment
    In HPSR Security Briefing Episode 16, we profiled North Korea’s cyber threat landscape, highlighting the regime’s known capabilities and deficiencies in cyberspace. In that report, we noted North Korea’s initial response to The Interview. Based on our previous research of North Korean cyber capabilities, it is difficult to discern whether the regime acted alone. It is plausible that the actors responsible for this attack relied on the assistance of an insider.
  • FBI: North Korea to Blame for Sony Hack
    While the United States government seems convinced by technical analysis and intelligence sources that the North Koreans were behind the attack, skeptics could be forgiven for having doubts about this conclusion. It is interesting to note that the attackers initially made no mention of The Interview, and instead demanded payment from Sony to forestall the release of sensitive corporate data. It wasn’t until well after the news media pounced on the idea that the attack was in apparent retribution for The Interview that we saw the attackers begin to mention the Sony movie.
  • Lessons from the Sony Hack
    This could be any of us. We have no choice but to entrust companies with our intimate conversations: on email, on Facebook, by text and so on. We have no choice but to entrust the retailers that we use with our financial details. And we have little choice but to use cloud services such as iCloud and Google Docs.
  • Did the DPRK Hack Sony?
    Drawing positive conclusions from the public evidence is incorrect. The NSA and the CIA may (or may not) have many other details they'll never disclose. The much-ballyhooed language setting, for example, is completely useless.
  • Why it's possible North Korea was behind the Sony hack
    NSA analysts might start with what’s known as “flow records,” which are very similar to Netflow. Each of these records details a communication between computers and the ports they used to initiate said communication. It might read something like this: "Computer A, port X sent K bytes of data to computer B, port Y between T0 and T1". Although flow records are the simplest data collected by the NSA’s wiretaps, it’s a powerful tool for tracking online activity.
  • What is happening?
    It's not about a movie or even Sony, at all. When you build a nuclear program, you have to explode at least one warhead so that other countries see that you can do it. The same is true with Cyber.
  • goodbye horses
    I suppose the more nuanced point Dave is trying to make, is not so much that this is an Act of War, or even that it should be. But more that this Sony thing is not about Sony, it's about the US necessarily having to prepare for a future where it will have to politically and publicly address and respond to a game that up until very recently, was only played in the shadows.

Hacker School publications are quite impressive

Somehow I stumbled across the inaugural issue of the Hacker School journal: Code Words: A quarterly publication about programming from Hacker School.

I haven't had time to read the entire issue yet, but so far I'm really pleased.

You know that you're really reading a publication targeted at programmers when the first paragraph of the first article is as follows:

Floating point is a giant mess. There are known best practices for most issues that come up in everyday use (e.g., using Kahan summation or adding stochastic noise to reduce aggregate numerical error), but there are still corner cases. Different libraries and implementations give different results because there’s no generally accepted standard.

"Numerical error"? "Kahan summation"? "Stochastic noise"?

Oh, yes, yes, YES! Bring it on!

And I also really enjoyed An introduction to functional programming, with its straight-forward, plain-english approach:

When people talk about functional programming, they mention a dizzying number of “functional” characteristics. They mention immutable data, first class functions and tail call optimisation. These are language features that aid functional programming. They mention mapping, reducing, pipelining, recursing, currying and the use of higher order functions. These are programming techniques used to write functional code. They mention parallelization, lazy evaluation and determinism. These are advantageous properties of functional programs.

Ignore all that. Functional code is characterised by one thing: the absence of side effects. It doesn’t rely on data outside the current function, and it doesn’t change data that exists outside the current function. Every other “functional” thing can be derived from this property. Use it as a guide rope as you learn.

They seem to be off to a great start, so I'm hoping that Hacker School continue to find success, and continue to publish more issues of their journal.

Meanwhile, reading the journal led me to the Hacker School blog, which seems pretty great too; I particularly like their "Read Along" feature, in which they select a research topic each week, post a pointer to the paper with some introductory thoughts, and encourage the audience to read along with the paper and contribute their own reactions.

If you're looking for some good reading to strengthen and extend your programming skills, give the Hacker School publications a try!

I suppose I should pay more attention to my investments...

Frankly, I don't monitor my retirement accounts anywhere near as closely as I should.

Who has the time?

But I do try to look at them every quarter or so, to think about how they are doing and what I should change, if anything.

So I signed on and looked at my IRA, and was reading through the various positions.

And I noticed information for a company I'd never heard of!

How did a company I'd never heard of appear in my account? Did I fumble-finger some trade, months ago? Was I hacked? The "purchase history" for the new company showed three apparently legitimate trades, dating back to 2011.

Then I did some web surfing, and arrived at: Kimberly-Clark Announces Details for Completion of Kimberly-Clark Health Care Spin-Off

Kimberly-Clark Corporation (NYSE: KMB) today announced the record date, distribution date and distribution ratio for the previously announced tax-free spin-off of its health care business. The spin-off will form the new publicly traded company, Halyard Health, Inc. Kimberly-Clark also increased its 2014 share repurchase program to take into account expected proceeds as a result of the spin-off.

Kimberly-Clark shareholders will receive one share of Halyard Health common stock for every eight shares of Kimberly-Clark common stock held as of the close of trading on Oct. 23, 2014, the record date for the spin-off.

And, sure enough, the dates of the three trades are precisely the three occasions on which I purchased Kimberly-Clark stock in my IRA.

The computers had quietly taken care of it all.

But, really, I ought to pay more attention to my retirement accounts.

Maybe next year, he says...

Wednesday, December 17, 2014

In which I try to learn more about things I don't understand.

Building on this post of a month ago, and also closely related to Behind the Beautiful Forevers, today I came across a fascinating article on Quartz: The secret to the Uber economy is wealth inequality

These luxuries are not new. I took advantage of them long before Uber became a verb, before the world saw the first iPhone in 2007, even before the first submarine fibre-optic cable landed on our shores in 1997. In my hometown of Mumbai, we have had many of these conveniences for at least as long as we have had landlines—and some even earlier than that.

It did not take technology to spur the on-demand economy. It took masses of poor people.

It's a fascinating article, and not too long.

Well worth reading.

Tuesday, December 16, 2014

Those who do not remember history...

... or something like that.

Anyway, BIG STORM UPDATE: California Drought Situation Improves says:

The much anticipated big storm that hit California on December 10-12 lived up to its expectations with torrential rains, heavy snow in the Sierra, and high winds (although these were not as powerful as forecast). A small tornado (EF-0) even touched down in South Los Angeles Friday morning, December 12th. Rainfall in the San Francisco Bay Area was especially impressive with San Jose picking up 3.23” on December 11th, its 3rd greatest calendar day rainfall on record (POR since 1893). Venado, in Sonoma County, received 9.44” in 24 hours. What was most welcome, however, was that the heavy rain also drenched the southern portions of California including the Central Valley, erasing the seasonal precipitation deficits that were still in place just a week ago.

but also notes that:

Although neither of these changes are any guarantee that the winter will continue to be a wet one, they are at least a good sign. The RRR (‘Ridiculously Resilient Ridge’) that has been the hallmark of drought is nowhere in sight (for the time being). Of course, this is the way the season of 2012-2013 began when a very wet November and December came to an end by January when the RRR set up and didn’t let go for the following two years.

Oh, yes, we remember that well.

The story mentions Venado, a place I've never been. Mill Creek Road looks like it would be a fun drive on a nice summer day. I'll have to try that sometime...

Meanwhile, though, the state is still bone dry, so: let it snow, let it snow, let it snow.

Tuesday, December 9, 2014

Markov chain humor

On the heels of last year's git man page generator, we now have:

  • The Doom that Came to Puppet
    Posts generated by a Markov chain trained on the Puppet documentation and the assorted works of H. P. Lovecraft. Created by @branan using barrucadu/markov. Inspired by King James Programming.

Very, very geeky.

But quite humorous (assuming you know a little bit about Lovecraft and a little bit about Puppet).

Here's a nice example:

“As our guarded messages stated, we rested at midnight after our day of terror and bafflement—but not without a tentative plan for one or more specific features”

Monday, December 8, 2014

Indeed, you can't simply call for a Tow Truck at this point...

The team at Wired have been doing a good job covering some of the events in the current Volvo Ocean Race:

  • Grueling 39K-Mile Yacht Race Tests the Sanity of Cramped Crews
    The 2014-15 edition of the race, which takes place every three years, will cover 39,000 miles, hit six continents, and run from October to June. This is the first time it will be a “one-design” race: All entrants must use a specially designed boat—the $6 million Volvo 65—with the same exact specifications. The new carbon-fiber boats, designed by Farr Yacht Design in Annapolis, Md. specifically for the next two Volvo Ocean Races and assembled in different spots around the world, are strong and sturdy.
  • Why the World’s Best Sailors Still Hit Reefs in Open Water Races
    But even with precautions and top equipment, you don’t always know what’s around you. The charts that record things like reefs can be off by up to five miles at some points, says Will Oxley, the navigator for Team Alvimedica. The archipelago where Vestas Wind ran aground (16°32’00.0″S 59°32’00.0″E) isn’t the proverbial uncharted territory, but it’s not particularly well charted, either. Because it’s not on a shipping route, rarely hosts races, and is nearly uninhabited, there’s been little need for detailed mapping. “This is quite new territory,” Oxley says.
  • How to Repair a $6M Racing Yacht You Sailed Into a Reef
    The good news is that putting in fresh carbon fiber “is relatively easily done,” Hogoboom says. The companies that built the boats still have the molds for all the parts (the current design will be used for the 2017-2018 race), and they could start making replacements while the damaged boat is dragged back. Then it’s just a question of cutting out the damaged areas and bonding on the new bits. Nonetheless, it would require a big commitment: Four companies, each in a different country, contributed parts of the boat, and they would all probably need to provide replacement parts.

Saturday, December 6, 2014

Early December link clearance

If the rain is keeping you indoors, here's some stuff to read.

  • The Case for Slow Programming
    Venture-backed software development here in the San Francisco Bay area is on a fever-pitch fast-track. Money dynamics puts unnatural demands on a process that would be best left to the natural circadian rhythms of design evolution. Fast is not always better. In fact, slower sometimes actually means faster – when all is said and done.
  • The Myth of the Interchangeable Programmer: Can't We Just Offshore Him?
    The problem here is that the SMFs incorporate a number of flawed assumptions. The first of these assumptions is that programmers are fungible. The SMFs assume all programmers will contribute roughly the same amount to a project and that all programmers are interchangeable.
  • Immigration is about talent, not costs
    So you can be sure that all things being equal we would prefer to hire local talent for a position. But all engineers are not created equal, and great engineers are scarce worldwide, so when we find great engineers abroad we make them compelling offers and try to hire them.
  • Google Test Automation Conference
    All of the GTAC 2014 video recordings and slides are publicly available. You can watch them from the GTAC 2014 YouTube playlist or browse the talks below
  • Never Send a Human to do a Machine’s Job: How Facebook uses bots to manage tests
    Developers aren't the jerks, bots are!

    Filing tasks, sending email on failure isn't enough.

    All tests need owners, if we fail to blame a revision.

  • From Test Driven Development to Behavioral Driven Design
    Behavioral-driven development was introduced by Dan North with issues he continually came across in test-driven development. (2) He suggested that instead of simply writing tests, developers should think of specifying behaviors, which is how the users want the feature to behave. In BDD, you should always start with the features that are most important to the users. Through collaboration and continual feedback, the practice of knowing what is most important becomes clearer.
  • Too Big to Fail
    "The system is slow" is a poor bug report—in fact, it is useless. However, it is the one most often uttered in relation to distributed systems. Typically the first thing that users of the system notice is that the response time has increased and that the results they get from the system take far longer than normal. A distributed system needs to express, in some way, its local and remote service times so that the systems operators, such as the devops or systems administration teams, can track down the problem. Hot spots can be found through the periodic logging of the service request arrival and completion on each host. Such logging needs to be lightweight and not directed to a single host, which is a common mistake. When your system gets busy and the logging output starts taking out the servers, that's bad. Recording system level metrics, including CPU, memory and network utilization will also help in tracking down problems, as will the recording of network errors. If the underlying communications medium becomes overloaded, this may not show up on a single host, but will result in a distributed set of errors, with a small number at each node, which lead to chaotic effects over the whole system. Visibility leads to debuggability; you cannot have the latter without the former.
  • Exactly-Once Delivery May Not Be What You Want
    I think there are two lessons here for people building distributed systems. One is that end-to-end system semantics matter much more than the semantics of an individual building block, and sometimes what seems like a very desirable semantic for a building block may end up making the end-to-end problem harder. The other is that simple, practical, solutions like unique IDs can make really hard problems much easier, and allow us to build and ship real systems that work in predictable ways.
  • Aliens in the Valley: The complete history of Reddit, the Internet's front page
    This is the story of how a bootstrapped startup with a funny name and no initial ties to the tech scene outlasted better-funded competitors, survived founder drama, endured tensions with its parent company and later navigated life as a standalone business — all in order to build the front page of the Internet.
  • PostgreSQL’s Powerful New Join Type: LATERAL
    a LATERAL join is like a SQL foreach loop, in which PostgreSQL will iterate over each row in a result set and evaluate a subquery using that row as a parameter.
  • Steve Hetzler's "Touch Rate" Metric
    As an example of the use of these graphs, Steve analyzed the idea of MAID (Massive Array of Idle Drives). He used HGST MegaScale DC 4000.B SATA drives, and assumed that at any time 10% of them would be spun-up and the rest would be in standby. With random accesses to data objects, 9 out of 10 of them will encounter a 15sec spin-up delay, which sets the response time limit. Fully powering-down the drives as Facebook's cold storage does would save more power but increase the spin-up time to 20s.
  • Java for Everything
    If you and I have a contest to write a simple blogging system and you’re using (say) Python, you’ll have something interesting in 30 minutes using pickling and whatnot, and it’ll take me two days to build something with MySQL. Many language choices are based on trivial contests like these. But after two weeks of development, when we both have to add a feature, mine will take at most as long as yours, and I won’t be spending any time figuring out how to get my system to handle so many users, or tracking down why some obscure if clause breaks because you misspelled the name of a function, or figuring out what the heck this request parameter contains.
  • The Unreasonable Effectiveness of C
    C is the total package. It is the only language that's highly productive, extremely fast, has great tooling everywhere, a large community, a highly professional culture, and is truly honest about its tradeoffs.

    Other languages can get you to a working state faster, but in the long run, when performance and reliability are important, C will save you time and headaches.

  • The Last Homesteads of Wonder Valley, California
    Arid, undeveloped, and sparsely beautiful, Wonder Valley was deemed “useless” by the federal government, and so was portioned out for mostly recreational use at $10 to $20 an acre. The sole requirement was that the applicant "proved up" the land with a small house.
  • NPR’s Book Concierge: Our Guide To 2014’s Great Reads
    Use the filters below to explore some 250 titles NPR staff and critics loved this year.

Friday, December 5, 2014

Great article on benchmarking SSDs

Check out this great article: The SSD Endurance Experiment: Two freaking petabytes: The survivors soldier on to another really big number

The authors have been running a benchmark on a bunch of SSD drives for more than a year!

They're looking at reliability, manageability, and performance.

We benchmarked all the SSDs before we began our endurance experiment, and we've gathered more performance data after every 100TB of writes since. It's important to note that these tests are far from exhaustive. Our in-depth SSD reviews are a much better resource for comparative performance data. What we're looking for here is how each SSD's benchmark scores change as the writes add up.

Yes, you read that right: "every 100 TB".

They are forcing petabytes of data through their SSDs, and observing how they fare.

And they fare, in fact, quite well:

The results of our experiment do, however, point to some more general conclusions about SSDs as a whole. Although only two drives made it to 2PB, all six wrote hundreds of terabytes without issue, vastly exceeding their official endurance specifications. More importantly, the drives all survived far more writes than most users are likely to generate. Typical consumers shouldn't worry about exceeding the endurance of modern SSDs.

The whole article is pretty interesting.

The end of TNR

For about 15 years, from about 1985 to about 2000, I was not only a reliable subscriber but a devoted reader of The New Republic

I loved pretty much everything about the magazine: its style, its selection of topics, its ability to attract the best writers.

Most of all, I loved its fearlessness; it always seemed to me to be willing to ask the hardest questions and not to shy away from the answers.

I'm sure that I changed, but sometime around the turn of the millenium The New Republic changed even more. I drifted away, reluctantly, and finally stopped reading it entirely.

Now comes news of the final collapse, as covered in Slate: What We Lost With the Loss of the New Republic

I sort of found my own voice, as an adult, by reading TNR. Even though it's nothing like it once was, I'll miss it.

Wednesday, December 3, 2014

Post-L-Triptophan reading list

OMG there's so much to read.

It's amazing how much accumulates when you take a week off...

  • The QA Mindset
    My concern is that the absence of QA is the absence of a champion for aspects of software development that everyone agrees are important, but often no one is willing to own. Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective. Traditionally, they are known as the folks who break things, who find bugs, but QA’s role is far more important. It’s not that QA can discover what is wrong, they intimately understand what is right and they unfailingly strive to push the product in that direction.

    I believe these are humans you want in the building.

  • Gangnam Style Video Overflows YouTube Counter
    If you hover your mouse over the counter, it spins like a slot machine; if you hold the mouse there long enough it will show a negative number. But the negative number is not what I expected. Is there a bug in the Easter egg?
  • The real and complete story - Does Windows defragment your SSD?
    The short answer is, yes, Windows does sometimes defragment SSDs, yes, it's important to intelligently and appropriately defrag SSDs, and yes, Windows is smart about how it treats your SSD.
  • Future of Popular Coding Tool In Doubt After Public Split
    The split highlights the tensions that often exist between the corporate sponsor of an open source project and the many other coders and businesses who use it and help build it. Docker, the company behind a new approach to cloud computing that has exploded in popularity in the past year and half, is in a similar boat, with some community members complaining that the parent company has strayed from its original mission and one outfit going so far as to create a new rival for the project.
  • Two Farmers and Common Knowledge
    They needed some way to make their cycling workers reliable again, and that meant finding a technical solution to the problem.
  • Uber's Secret Weapon: The World
    But the international aspect is if anything even more important. This might not be obvious to people in San Francisco, who are spoiled with dozens of hopeful and well-funded startups, many of which are doing much the same thing that Uber is aspiring to. But leave the Bay Area, and the fears and frustrations of trying to get a cab start getting magnified – especially when you’re in a foreign country. The value of Uber is only partially in the service it provides; increasingly, it’s also in the global ubiquity of that service.
  • Introduction and Initial Plans
    Welcome to Quantum OS! We are working on developing an operating system based upon Linux which conforms to Google’s Material Design guidelines. The focus will be on creating a stable and easy-to-use operating system with a heavy emphasis on well-thought-out design.
  • God's Lonely Programmer
    TempleOS is more than an exercise in retro computing, or a hobbyist’s space for programming close to the bare metal. It’s the brainchild—perhaps the life’s work—of 44-year-old Terry Davis, the founder and sole employee of Trivial Solutions. For more than a decade Davis has worked on it; today, TempleOS is 121,176 lines of code
  • The Art of Insight in Science and Engineering
    To master complexity, we can organize it or discard it. The Art of Insight in Science and Engineering first teaches the tools for organizing complexity, then distinguishes the two paths for discarding complexity: with and without loss of information. Questions and problems throughout the text help readers master and apply these groups of tools. Armed with this three-part toolchest, and without complicated mathematics, readers can estimate the flight range of birds and planes and the strength of chemical bonds, understand the physics of pianos and xylophones, and explain why skies are blue and sunsets are red.
  • Out the Window
    A group of researchers that were immensely valuable according to Microsoft’s own metric just a couple of months before were thrown out to the hands of Microsoft’s competitors that were more than happy to oblige. Similarly, previously valued research projects were carelessly lost (quite possibly to be picked up by others). Excellence as defined by Microsoft did not protect you, impact did not protect you
  • Towards a more perfect link underline
    Overall, the underline is much thicker than Wichary’s ideal and sits too close to the text for my comfort. But most damningly to my mind, the underline does not change weight to adapt to the weight of the font, leading to unfortunate mismatches
  • Why Renewable Energy (Alone) Won't Full Solve the Problem
    I argue that all employees should be limited only by their ability rather than an absence of resources or an inability to argue convincingly for more. This is one of the most important yet least discussed advantages of cloud computing: taking away artificial resource limitations in support light-weight experimentation and rapid innovation. Making individual engineers and teams responsible to deliver more value for more resources consumed makes it possible encourage experimentation without fear that costs will rise without sufficient value being produced. And, because cloud computing is so inexpensive and comes without a long term commitment, a single engineer to do a trial run of a 1,000 core analysis to improve supply chain logistics without appreciable financial risk. If it works, keep doing it and reap the economic gain. If it doesn’t work, little was spent and it may have been a failed experiment but it was an inexpensive failed experiment. Economic systems are very powerful at driving innovation.
  • Best at Everything? It’s Closer to True
    It makes sense considering that the quarterback, Peter Williams, is studying, actually, to be a rocket scientist. Or that the senior linebacker Cameron Wagar, a mechanical engineering major, last week endured a review of a semester-long group project on Monday, a biology report due Wednesday and a biology exam Friday, just before the team bus left for Bangor, Me., where the undefeated Engineers faced Husson University in the first round of the N.C.A.A. Division III playoffs.
  • Some of these Things are not like the others
    What’s more important is to consider all the discrete moving parts, the small pieces loosely joined, and how they fit together: networked systems, objects, and actors, interacting with one another at a distance, over APIs, legislature, process. These are all interfaces for the city. They’re not all digital technology, either – they’re frequently transitions between state, the digital communicating with the human, or the physical, or the abstract, and vice versa.

    Cities are made of many different things, in many states, and digital technology’s role shouldn’t just be to bridge between all those states and the digital – but also to link things of one state to another.

  • How browsers get to know you in milliseconds
    When the user clicks on a link or enters a URL, the server returns a page with ad tags that contain JavaScript to invoke an auction with a particular exchange. The browser then issues a request to an exchange such as AppNexus, or to a supply-side platform (SSP) that then sends it to the exchange.
  • Nifty tech delivers ineffective crap at incredible speed!
    Hold on a minute. Online display ads are terribly ineffective, despite all the bleeding-edge technology being thrown at them?

    Close. But not despite. Because.

  • Unpacking privacy
    The problem is that if the medium is targetable, then the best strategy for an individual site is to do targeting, even if (because of the signaling value of its content) the site would do better in a system where no user could be targeted. When we stop thinking about privacy as a big, complicated, hard concept, and try to break out some kind of Minimum Viable Privacy, just enough to protect that "car intender" from site to site tracking, then ways out of the race to the bottom start to present themselves.
  • Consider Your Perspective: A Chart – Gold vs U.S. Debt
    There was a period where the two seemed to move together, and there was a much longer period where they did not. One can selectively choose start and endpoints that torture the data in an effort to further a narrative or confirm a bias, but when one does that, one should not be shocked when one’s conclusions turn out to be incorrect.
  • The Ship Breakers
    The ships are driven right up onto shoreline lots set aside for ship breaking, then attacked by hammer and blowtorch until all usable material has been stripped away to be sold or recycled.
  • Under London
    I'm a sucker for images of the human form stranded amidst the shadows of massive, dimensionally abstract spatial environments, so I thought I'd post these purely as eye candy.
  • Yerba Buena Island
    aids to navigation became more important and the unique octagonal lighthouse and the fog signal, which are still operational on the southern end of the island, were completed in 1875. The beautiful house just above them was constructed for the lighthouse keeper and now serves as the home of a Coast Guard admiral.
  • The Missing Men
    Why didn’t a Rolling Stone writer talk to the alleged perpetrators of a gang rape at the University of Virginia?
  • Ill Doctrine
    ill Doctrine is a video blog hosted by Jay Smooth, founder of New York's longest running hip-hop radio show, WBAI's Underground Railroad.

Tuesday, December 2, 2014

Behind the Beautiful Forevers: a very short review

As I mentioned the other day, I had some time to read recently, so I caught up on a few books that had been sitting on my Kindle for a while.

One, in particular, was Katherine Boo's Behind the Beautiful Forevers: Life, Death, and Hope in a Mumbai Undercity.

Beautiful Forevers is simply an astonishing book.

I thought I knew a little bit about India. Though I've never been there, I've worked closely with Indian colleagues (both India-based and U.S.-based) on many occasions, I've known both family and co-workers who have traveled extensively in India, and I've read about India over many years.

But Beautiful Forevers is a shocking eye-opener.

It's non-fiction that reads like the latest crime thriller you pick up from the newsstand rack.

It's investigative journalism of the first order, digging down below the surface and helping you truly understand what is happening and why.

And it's the heart-breaking, emotional, compelling story of some of the most interesting people you'll never meet.

The story that Boo tells is horrible, tragic, and shattering.

But it is also important, and timely, and necessary.

If you get a chance, take the time to read Beautiful Forevers.

The Martian: a very short review

I happened to do some traveling recently and had some time to read.

One of the books I read was Andy Weir's The Martian: A Novel.

The Martian is a book about survival.

On Mars.

There are many great books about survival, both fiction and non-fiction, and the topic is very popular in movies, as well (Gravity is a recent example; Castaway a slightly older one).

The Martian's great appeal in this genre is that, well, it occurs on Mars.

And the author is fascinated by the concept of extraterrestial exploration, and has done his research, and thus writes with both passion and substance.

It doesn't hurt that Weir clearly has an abundance of writing talent, and can spin a tale.

The book just races along. I'll often dawdle in a book, as I have so many other distractions available. But The Martian is a compelling thriller and I sped through it in barely 10 hours.

I hope Weir continues to write, and I hope his subsequent efforts are as good as his debut.

Wednesday, November 26, 2014

The bug it took three dozen engineers to fix

Don't miss this nicely presented story about a subtle but devastating performance bug in a database connection pool cache eviction policy choice which was finally solved by a task force of Facebook engineers: Solving The Mystery of Link Imbalance: a Metastable Failure State at Scale

Sunday, November 23, 2014

Carlson prevails again

A beautiful win for Carlson in game eleven and he retains the championship.

Shall I make a prediction? Next up is caruana.

Saturday, November 22, 2014

Turkey trot

It seems like we are at that time of the year when everybody is incredibly busy.

And there's so much to read!

  • If you only watch one technical video this year, make it James Hamilton: AWS Innovation at Scale
    This session, led by James Hamilton, VP and Distinguished Engineer, gives an insider view of some the innovations that help make the AWS cloud unique. He will show examples of AWS networking innovations from the interregional network backbone, through custom routers and networking protocol stack, all the way down to individual servers. He will show examples from AWS server hardware, storage, and power distribution and then, up the stack, in high scale streaming data processing. James will also dive into fundamental database work AWS is delivering to open up scaling and performance limits, reduce costs, and eliminate much of the administrative burden of managing databases.
  • Google to Quadruple Computer Science Prize Winnings to $1 Million
    The Turing Award had carried prize money of $250,000 and was jointly underwritten by Google and Intel since 2007. But Intel decided to step away as a funder, and Google stepped up and upped the ante.
  • Microsoft Releases Emergency Security Update
    “The attacker could forge a Kerberos Ticket and send that to the Kerberos KDC which claims the user is a domain administrator,” writes Chris Goettl, product manager with Shavlik. “From there the attacker can impersonate any domain accounts, add themselves to any group, install programs, view\change\delete date, or create any new accounts they wish.
  • Compiler Design in C
    Compiler Design in C is now, unfortunately, out of print. However, you can download a copy
  • Keeping Secrets
    the conference featured the work of a group from Stanford that had drawn the ire of the National Security Agency and the attention of the national press. The researchers in question were Martin Hellman, then an associate professor of electrical engineering, and his students Steve Pohlig, MS ’75, PhD ’78, and Ralph Merkle, PhD ’79.

    A year earlier, Hellman had published “New Directions in Cryptography” with his student Whitfield Diffie, Gr. ’78. The paper introduced the principles that now form the basis for all modern cryptography

  • Building a complete Tweet index
    In this post, we describe how we built a search service that efficiently indexes roughly half a trillion documents and serves queries with an average latency of under 100ms.
  • Why I'm not signing up for Google Contributor (or giving up on web advertising)
    People say all kinds of stuff. You have to watch what they do. What they do, offline, is enjoy high-value ad-supported content, with the ads. Why is the web so different? Why do people treat web ads more like email spam and less like offline ads? The faster we can figure out the ad blocking paradox, the faster we can move from annoying, low-value web ads to ads that pull their weight economically.
  • Your developers aren’t slow
    Feel like your team isn’t shipping fast enough? Chances are, your developers aren’t to blame.

    What’s really slowing down development?

    If it’s not your developers, what’s slowing down development? Here’s a hint: it’s your process.

  • Cache is the new RAM
    You know things are really desperate when “less painful than writing it yourself” is the main selling point.
  • Delayed Durability in SQL Server 2014
    With delayed durability, the transaction commit proceeds without the log block flush occurring – hence the act of making the transaction durable is delayed. Under delayed durability, log blocks are only flushed to disk when they reach their maximum size of 60KB. This means that transactions commit a lot faster, hold their locks for less time, and so Transactions/sec increases greatly (for this workload). You can also see that the Log Flushes/sec decreased greatly as well, as previously it was flushing lots of tiny log blocks and then changed to only flush maximum-sized log blocks.
  • Delayed Durability in SQL Server 2014
    Like many other additions in recent versions of SQL Server (*cough* Hekaton), this feature is NOT designed to improve every single workload – and as noted above, it can actually make some workloads worse. See this blog post by Simon Harvey for some other questions you should ask yourself about your workload to determine if it is feasible to sacrifice some durability to achieve better performance.
  • The Programmer's Price: Want to hire a coding superstar? Call the agent.
    Hiring computer engineers used to be the province of tech companies, but, these days, every business—from fashion to finance—is a tech company. City governments have apps, and the actress Jessica Alba is the co-founder of a startup worth almost a billion dollars. All of these enterprises need programmers. The venture capitalist Marc Andreessen told New York recently, “Our companies are dying for talent. They’re like lying on the beach gasping because they can’t get enough talented people in for these jobs.”
  • git-p4 - Import from and submit to Perforce repositories
    Create a new Git repository from an existing p4 repository using git p4 clone, giving it one or more p4 depot paths. Incorporate new commits from p4 changes with git p4 sync. The sync command is also used to include new branches from other p4 depot paths. Submit Git changes back to p4 using git p4 submit. The command git p4 rebase does a sync plus rebases the current branch onto the updated p4 remote branch.

Fun and Games

With two games to go, Carlsen still maintains his 1 point advantage over Anand. But two games left means nothing is yet decided.

  • Did champ fall asleep during chess title match game?
    The chess world is still buzzing over images posted to social media that appear to show world chess champion Magnus Carlsen of Norway actually falling asleep at the board during this week’s Game 8 of his world championship match with Indian challenger Viswanathan Anand of India in Sochi, Russia.
  • Cool Chess Photo Trivia: Name event, Year!
    Did both the players know back then that they would meet in Sochi! Can you name the chess event and the year?
  • Ten Years of World of Warcraft
    These days, so many more people have passed through the gates of Azeroth than ever played its antecedents that many don’t even know the deep wellspring sources from which it came. Most of the defining characteristics of WoW are from a long tradition that started around 1990. WoW represents the (perhaps final) evolution of the DikuMUD model.
  • The Unbalanced Design of Super Smash Brothers
    I think a large part of what enables this depth to be found in Smash Brothers is that the game isn't balanced. It's not a small roster of perfectly tuned characters. It's a big game with lots of characters that creates a huge and unexplored problem space. It's up to the players to explore the nooks and crannies and see what treasures they can find. I find that incredibly exciting and compelling.
  • Handmade Hero
    Handmade Hero is an ongoing project to create a complete, professional-quality game accompanied by videos that explain every single line of its source code.
  • Finally, a New Clue to Solve the CIA’s Mysterious Kryptos Sculpture
    Is a Clock a Clock?

    Sanborn told WIRED that he’s always been fascinated by Berlin’s many clocks but the Berlin Clock in particular has intrigued him the most. The clock, also known as the Berlin Uhr or Set Theory Clock, was designed in the 1970s by inventor and tinkerer Dieter Binninger. It displays the time through illuminated colored blocks rather than numbers and requires the viewer to calculate the time based on a complex scheme.

  • Boxer-Clad Coders Adorn Silicon Valley’s Billboard Boom
    Surging billboard demand has led to a jarring sight for those driving on San Francisco’s main highway these days -- a 50-foot picture of a pasty software engineer, lying provocatively on his side, showing a bit of chest hair and wearing only his underwear.

    “Find the hottest tech talent,” reads the billboard for technology jobs website Dice.com. The ad went up last month to tout new services for recruiting technology workers.

  • Lavish Perks Spawn New Job Category: At Tech Companies, Aim-to-Please Specialists Provide Yoga Classes, Jell-O Shots; ‘We’re Like Little Elves’
    “We are just providing basic standards,” says Ms. Nguyen, 40 years old, whose title is head of workplace. Free lunch, dinner, snacks and events like a Jell-O shot-making “studio night” are a big part of what it takes to keep Pinterest’s roughly 450 employees productive and happy, she adds.
  • Bandstand
    On the whole, if there were different line-ups of bands, the classic line-up of a band is shown. (Obviously, there might be times when what I consider a classic line-up is different to what other people might consider a classic line-up.) Stage positions are approximated.
  • Mango’s Tropical CafĂ© Owners Unveil Track Design for World’s Tallest Rollercoaster – The Skyscraper™
    The track design of Skyscraper is totally unique to Orlando and was created specifically to begin and end inside SKYPLEX. The $250 million entertainment complex comprises approximately 495,000 square feet located on 12 prime acres in the attractions corridor. SKYPLEX will feature the area’s largest indoor theme park complete with an upscale arcading, simulators, rock climbing, bars, and a host of rides and games, as well as additional opportunities for attractions, interactive venues, restaurants and retail.

Is it an omen? The squirrel that occupies our front yard stole the lemon off our lemon bush.

Interpret that!

Friday, November 21, 2014

Other people discuss things I don't understand

I should understand these things.

But I don't.

But I must try to.

  • Uber Executive Suggests Digging Up Dirt On Journalists
    Over dinner, he outlined the notion of spending “a million dollars” to hire four top opposition researchers and four journalists. That team could, he said, help Uber fight back against the press — they’d look into “your personal lives, your families,” and give the media a taste of its own medicine.

    Michael was particularly focused on one journalist, Sarah Lacy, the editor of the Silicon Valley website PandoDaily, a sometimes combative voice inside the industry. Lacy recently accused Uber of “sexism and misogyny.” She wrote that she was deleting her Uber app after BuzzFeed News reported that Uber appeared to be working with a French escort service. “I don’t know how many more signals we need that the company simply doesn’t respect us or prioritize our safety,” she wrote.

  • The Coming Era of Privacy Scandals: Why the Uber story is certain to recur
    The data revolution has happened; the toothpaste can’t be put back into the tube. It’s not just the NSA which has access to enormous amounts of personal data on us; it’s any number of companies big and small, which you may or may not have ever heard of. If that data can be subpoenaed, and most of it can be, then it can also be accessed without a subpoena by people within the firm, who might have nothing better to do during their lunch hour than look up celebrities, or friends, or enemies and see what they can find. Such behavior is ignoble, to be sure – but it is going to happen.
  • Uber's data makes a creepy point about the company
    A couple of years ago, there was an entry on the company's blog titled "Rides of Glory." The company examined its rider data, sorting it for anyone who took an Uber between 10 p.m. and 4 a.m. on a Friday or Saturday night. Then it looked at how many of those same people took another ride about four to six hours later – from at or near the previous nights' drop-off point.

    Yes, Uber can and does track one-night stands.

  • Some thoughts on App Based Car Services
    Cars are already becoming generic. And already we have a generation coming up that gives a much smaller damn about driving than did previous ones — at least in the U.S. All that aspirational stuff about independence and style doesn’t matter as much as it used to. How long before GM, Ford and Toyota start making special models just for Uber and Lyft drivers?
  • What to do about Uber
    So here's a modest suggestion: treat the city's roads like a traditional public utility, or a public resource like cell spectrum. Invite Uber, Lyft and other ride-sharing companies to tender for a license as the city's preferred ride-sharing service. Let them compete to show how they'll treat drivers and riders fairly, in their terms and conditions; and let them bid for the right to this future profit stream.
  • Nick Denton On How Gawker, Uber and Facebook Will Save Humanity
    Gawker Media’s sites have also been aggressive in calling out the many supposed sins of Uber and its CEO, Travis Kalanick. But Denton has praise for the car-summoning service — high praise indeed. “Uber may do more for the world than foreign aid workers in Mozambique because at some point some version of Uber will allow for more efficient use of resources and a better standard of living,” he says.
  • Stuck in the middle
    There is nothing surprising or wrong with the idea of a tech company investigating reporters. The idea that only reporters have the ability to publish is a 20th century idea. Now anyone who wants to speak can start a blog or a podcast and get up and speak.

Thursday, November 20, 2014

Drip drip drip

It's snowing in Truckee!.

And it rained about half an inch, maybe more, here in the Bay Area, today.

But oh-my-goodness is it dry: 203 drought maps reveal just how thirsty California has become.

More than 80% of California is in extreme drought, according to the U.S. Drought Monitor, and the state's condition isn't expected to improve in the near future.

I'm not sure I'd want to change places with Buffalo, New York, though: Jets-Bills Game Will Not Be Played In Buffalo Due To Massive Snowfall.

A league spokesman says a severe lake-effect snowstorm hitting Buffalo has led the NFL to relocate the Bills home game against the New York Jets on Sunday.

But hopefully we can at least continue to get some rain...

Wednesday, November 19, 2014

Things you can put into your roasted vegetables dish

For example, if you're making roasted vegetables for the company potluck.

  • Sweet Potatoes
  • Yams
  • Potatoes
  • Parsnips
  • Beets
  • Carrots
  • Celery
  • Leeks
  • Butternut Squash
  • Acorn Squash
  • Kabocha Squash

Clean; peel; chop.

Toss with olive oil, thyme, rosemary, salt and pepper.

Bake at 350 degrees for 45 minutes or so.

Attend company potluck.

Tuesday, November 18, 2014

Read, read, read

My youngest child is 23 years old!

Does this mean I am finally a grown-up?

  • Alexander Grothendieck 1928–2014
    So what can all this mean for us who work in what Grothendieck described as a “mansion” in which “the windows and blinds are all closed,” while he was one of those “whose spontaneous and joyful vocation it has been to be ceaseless building new mansions”? At least he did not call our dwelling a cave. However, in complexity theory we have it worse than Plato’s cave-prisoners in not merely missing the blinding world outside, but sensing its impact as a negative image in our present ignorance of lower bounds.
  • Andrew Odlyzko Strikes Again
    Last year I blogged about Andrew Odlyzko's perceptive analysis of the business of scholarly publishing. Now he's back with an invaluable, must-read analysis of the economics of the communication industry entitled Will smart pricing finally take off?.
  • A Worm's Mind In A Lego Body
    The nematode worm Caenorhabditis elegans (C. elegans) is tiny and only has 302 neurons. These have been completely mapped and the OpenWorm project is working to build a complete simulation of the worm in software. One of the founders of the OpenWorm project, Timothy Busbice, has taken the connectome and implemented an object oriented neuron program.
  • Keep Daly City Uncool
    Admittedly, the rumor that a proposed move to Daly City caused dissent among Reddit's ranks is met with a sense of relief around here where there's reason to fear that all those nearby gobs of money will ooze out of San Francisco and Atherton and engulf Val's, Joe's and our twin Targets like ravenous blob fueled by Bitcoin and fully-vested stock options.
  • Introducing “A Field Guide to the Distributed Development Stack”: Tools to develop massively distributed applications
    In addition to being a (hopefully) useful framework, the Guide is also meant to be a living resource. So, we’ve put the source on GitHub and invite you to contribute. If you feel like we’ve missed a tool (which we most certainly have, since new things are popping up every day) or a major theme, then fork the repo and send me a pull request. We’ll be keeping this document up to date and republishing it as we watch this trend continue to grow. We’ll use O’Reilly Atlas to pull in the contributions and periodically republish the guide.
  • Binary artifact management in Git
    Management of large binaries is still an unsolved problem in the Git community. There are effective alternatives and work-arounds but it’ll be interesting to see if anyone tries to solve the problem more systematically.
  • Beating Ebola Means Drinking, Last Thing Patient Wants to Do
    “We will halve the mortality by firstly just stopping anti-inflammatories and giving hydration, and really pushing it,” Mardel said. “I want every man and woman in Sierra Leone to know this. I want sports personalities to be talking about it. I want everybody to be talking about it.”
  • This Mod Is the Absolute Best Way to Play XCOM
    XCOM: Enemy Unknown is a great tactical game from Firaxis games. Its first expansion pack, Enemy Within, added so much amazing content that it made playing the game an entirely new experience. And now there’s a mod that’s so good it makes the expansion pack feel like a tired retread.

This time, the buzzing was not in my head

Reason number forty nine to have an unnecessary panic attack:

The other evening, I was going through my house, getting ready for bed, turning off lights, closing doors, making all my regular routine rounds.

I check the garage to make sure the light is off, but when I open the door, there is a strange sound.

It's sort of a buzzing sound, sort of a clicking sound, sort of a humming sound.

I wander around the garage for a while, trying to find the source of the sound: it's clearly coming from the car.

The car is switched off, but I wonder: did I perhaps leave the key in the ignition? leave a switch at a funny setting?

No.

The sound is coming from under the back of the car, so I get down on my tummy, shine my flashlight around, stick my arm in and try to touch various parts of the car to see if any of them are vibrating.

It's completely baffling.

So, I do the Only Sensible Thing: I go back in to the house, sit down at my computer, and search the Web.

Sure enough, there are dozens of forums filled with discussions of this behavior, and after clicking a few times, one of the messages sends me back out to the car, to look in the glove compartment, in the Owner's Manual, where, on page 9, I find:

Noise from under the vehicle.

NOTE: You may hear a noise from under the vehicle approximately 5 to 10 hours after the engine is turned off. However, this does not indicate a malfunction. This noise is caused by the operation of the fuel evaporation leakage checking system and is normal. The noise will stop after approximately 15 minutes.

And it did.

Sunday, November 16, 2014

Carlsen wins game 6!

After 6 rounds, it's Magnus Carlsen 3.5, Viswanathan Anand 2.5.

Superb chess, even though it's nearly all over my head. You can read lots of analysis on the web, including here.

Until I read the analysis, I didn't understand exactly where Anand lost, but I'm comforted that after I read the analysis, I understood it.

Me, I just love the fact that they both play 1. e4.

Saturday, November 15, 2014

Stuff I'm reading, mid-November edition

Today we once again came to the conclusion that the middle of November is the perfect time to go to Morgan Territory Park, and that Morgan Territory Park is the perfect place to be a dog.

  • IBM X-Force Researcher Finds Significant Vulnerability in Microsoft Windows
    Cutting to the chase, VBScript permits in-place resizing of arrays through the command “redim preserve.” This is where the vulnerability is.

    redim preserve arrayname( newsizeinelements )

    VBScript.dll contains a runtime evaluation method, CScriptRuntime::Run(VAR *), which farms out the SafeArray redimension task to OleAut32.dll with the SafeArrayRedim(…) function. Essentially, what happens is that fairly early on, SafeArrayRedim() will swap out the old array size (element count) with the resize request. However, there is a code path where, if an error occurs, the size is not reset before returning to the calling function, VBScript!CScriptRuntime::Run().

  • Microsoft Fixes Bug That Dates Back to Windows 95
    According to Microsoft, the bug resides in the Windows Object Linking and Embedding (OLE) technologies that debuted in the 1990s as a way for applications to share information with each other.
  • Unintended Consequences of the Information Economy
    The DoD can’t just dial up more innovation capacity by throwing money at the problem, like they did in WWII. Nor, in a free country, can the U. S. government just mandate for whom companies choose to work. Innovation capacity requires not only brilliant engineers, who are hard enough to come by, and who cannot be easily identified in the job market, but also a willingness to accept a lot of risk: to try and perhaps to fail, over and over. To old school 20th century managers, this looks a lot like waste, but in fact it’s a necessary part of the innovation process. The economics of conflict is changing just like the economics of everything else is changing.
  • Wil Wheaton: Anonymous trolls are destroying online games. Here’s how to stop them.
    It’s time to break this cycle—and to teach gamers that they can compete without being competitive, that they can win and lose without spewing racist, misogynist, homophobic bile at their fellow gamers. But doing so requires casting off the cloak of anonymity.
  • Social media is broken
    Anonymity can be very important for the marginalized, for whistleblowers, etc. But within their communities of trust they build reputation, including pseudonymous reputation. The real issue is feeling free of reputation, which equals feeling free of consequence. That is where bad behavior comes from.
  • Why I love databases
    To many of my colleagues and friends, the database is a magical black box system, too scary and complicated to understand. I wanted to change that.

    While talking about databases, the topic of distributed systems cannot be ignored. Most modern databases are distributed, either implicitly (distributed clustered databases) or externally (a single application connected to multiple databases via application-level sharding).

    This post is a confession of my love for databases and distributed systems. It is mostly targeted towards programmers like me, application developers who regularly interact with databases.

  • Climbing off the CTO ladder (Before We Fall Off)
    Spotify has recently started thinking about the non-management career track and we are working to see if there is a program we would like to adopt. We’ve had feedback from engineers and managers alike that we need to add a tech ladder; however, I am not convinced that adding a ladder fixes everything. The second ladder is a blunt instrument that can easily add some clarity to our growth ambitions, but it is not a complete solution. Part of management is optimizing for the productivity of the team, not our own work streams. This should not be taken lightly as it will have an effect on the organization’s culture. Understanding this may help invest in a technology ladder that works for the culture of your organization.
  • Optimizing Disk IO and Memory for Big Data Vector Analysis
    For “big data” datasets where the size of data is significantly larger than the size of memory, the most common bottleneck is disk bandwidth. Disk bandwidth of the highest-end disks remain on the order of hundreds of megabytes per second, while memory bandwidth is usually at least an order of magnitude faster. Furthermore, very little work is required of the CPU per record (just two predicate evaluations and a sum) --- database queries tend to be far less CPU-intensive than other domains (such as graphics rendering or scientific simulations). Hence, step 1 is often a bottleneck.
  • Intro to Distributed Hash Tables (DHTs)
    Lets assume that you want to get to some place, and you are not sure where it is. A good idea would be to ask someone how to get there. If you are very far from your destination, most likely the person you asked will give you a very vague description of how to get there. But it will get you starting in the correct direction.

    After you advance a while, you can ask somebody else. You will get another description, this time more a detailed one. You will then follow this description, until you get closer.

    Finally when you are really close, you will find someone that knows exactly where is that place you are looking for. Then your search will end.

  • Yelp Prison Review Faxbot
    Fletcher Bach and I recently discovered that there are reviews of prisons on Yelp. Some of these reviews are snarky one-liners -for example, one yelper describes Rikers Island as a “great island getaway right in my own backyard”. Other reviews appear to be honest first person accounts. Some people review what it’s like to visit the prison; others describe their experiences as inmates.
  • Android 5.0 Lollipop, thoroughly reviewed
    Android 5.0 Lollipop is at least the biggest update since Android 4.0, and it's probably the biggest Android release ever. The update brings a complete visual overhaul of every app, with a beautiful new design language called "Material Design." Animations are everywhere, and you'd be hard-pressed to find a single pixel from 4.4 that was carried over into 5.0—Google even revamped the fonts.
  • Google's New, Improved Android Will Deliver A Unified Design Language
    As Android lead designer Matias Duarte demoed it on stage, he explained that it moved with the physics of card stock, but also splash with your touch, like “ink rippling in a pond.” He clearly put it better than I can, though I’d add that Android‘s core UI has long been cleanly designed, but was always a bit cold. Material Design adds a bit of human warmth back to the equation.
  • Cooking the Alinea cookbook
    Allen Hemberger cooked his way through one of the most complex cookbooks out there, the Alinea cookbook. Aside from the chefs who work in the kitchen there, Hemberger's probably the only person to have made every single recipe. These recipes aren't easy; look at the last one he prepared...he even struggled to find the correct ingredients.

The Owens Valley dust bowl

Can plowing the soil actually REDUCE the dust that blows away?

And can it be possible to reduce the dust pollution in the Owens Valley while also INCREASING water deliveries to the Los Angeles area?

My mom sent me an absolutely fascinating story from the Los Angeles Times: New dust-busting method ends L.A.'s longtime feud with Owens Valley

The story describes a technique which certainly falls into the "seems too good to be true" category:

The new solution is relatively inexpensive and nearly waterless, DWP officials said. It involves using tractors to turn moist lake bed clay into furrows and basketball-sized clods of dirt. The clods will bottle up the dust for years before breaking down, at which point the process will be repeated.

The method was first tested in the early 1990s, then tabled out of concern the furrows and clods would disintegrate after a few rains. Two years ago, the DWP resurrected the idea and tested it on several acres of lake bed, but on a much larger scale, with furrows 2 to 3 feet deep. The results showed promise, provided the treated area has clay soil and flooding infrastructure in place.

Don't underestimate the stakes here: this is big business, and big money. And the other techniques that have been attempted over the previous quarter-century cost much, much more:

The new method will cost DWP customers about $1 million per square mile — three times less than shallow flooding. The cost of reducing dust with gravel, which has been applied to swaths of the lake bed, is about $25 million per square mile, officials said.

The utility has already spent $1.3 billion in accordance with a 1997 agreement to combat dust over a 40-square-mile area, reducing particle air pollution in the region by 90%.

This represents dramatic progress for the Owens Valley, which has been struggling with these issues for 100 years.

Perhaps more importantly, the Owens Valley is probably the most-watched, most-studied, most-analyzed, and most-fought-over area in the water battles that occupy the entire Western United States.

So what happens in the Owens Valley doesn't just affect the Owens Valley; it affects fully one third of the country.

For example, consider this discussion about water issues in southern Nevada: Las Vegas and the Groundwater Development Project, which has an entire chapter entitled "Remember Owens Valley".

Not much grows on the exposed lakebed, and that’s where the trouble starts. There’s dust, lots of it. Sweeping winds come roaring down the valley and create tremendous dust storms. At times there’s so much dust and the visibility is so poor the locals call it the “Keeler fog” after the small remnant of a town on what used to be the lake’s eastern shoreline. Recently, a pilot followed a dust plume from the lake-bed all the way into the Grand Canyon.

Nothing about this process has been easy. A year ago, ARID Journal covered the controversy in detail: Particulate Matters: Settling the Dust on the Owens Dry Lakebed.

For the first 50 years that it diverted Owens River water to Los Angeles, the LADWP denied that dust was a problem on the river’s former lakebed. When in 1976 scientists at China Lake Naval Weapons Center in Ridgecrest, California, photographed clouds carrying an estimated 40,000 metric tons of fine alkali grit billowing out of the Sierra into the neighboring Mojave Desert foothills of Kern County, the LADWP claimed that Inyo County had some of the best air in the country and that, “there has been no substantiation of adverse health effects of alkali dust.” In 1987 the U.S. Environmental Protection Agency deemed the severity of the fine-grain pollution issuing from Owens dry lake as the worst in the country, outside of forest fire smoke, as often as 24 days a year. LADWP was on record that the land impacted by its water exports was “such a small area we think it is insignificant.

Over decades of study, multiple techniques were attempted:

More than a dozen suppression methods were considered in the lead-up to the first mitigation projects, including covering the lakebed in used automobile tires, but only three were eventually approved for widespread use: gravel cover, plants, and shallow flooding. Gravel, at $33 million per square mile to install, was deemed prohibitively expensive. Plant cover, most of which had to be salt grass, cost $15 million per square mile to install, then it needed irrigating. By far the cheapest immediate fix for a water company was to install bubblers to provide shallow flooding. LADWP estimates that the up-front cost of this was more like $12.9 million per square mile.

And the battle went on in the courts:

When a new dust abatement notice for 2.93 additional square miles arrived that summer, Nichols called in the lawyers in what has proved a sustained assault on the 1998 dust deal. Schade and Great Basin also went to the courthouse, filing suit against LA for non-compliance on an outstanding order. By October 2012, over in federal court, LADWP was suing Schade’s department, naming him personally as a capricious and rogue regulator, and also naming the California Air Resources Board, the US Environmental Protection Agency, the California State Lands Commission, and the federal Bureau of Land Management as colluders.

So when research by an Orange County landscape architecture firm named Nuvis came up with a new idea, it must have seemed like a fantasy at first:

Those decisions include widespread use of an as yet un-validated waterless dust control method called “tillage,” which will have to be approved by regulators including Schade before the meandering furrows shown in the Nuvis schematics could be plowed into the lakebed. “We’re hoping that tillage, basically like farm tillage, will be approved,” says Adams. “It costs about 10% of what it costs to do flooding. It’s a huge savings for rate payers.”

So, let's hope this this new technique actually works.

And let's hope that agencies across the west are able to learn from this, and spend less time (and money) fighting in the courts, and more time (and money) figuring out how to use water effectively without destroying the land.

And let's all say a big thanks to Ted Schade.

Tuesday, November 11, 2014

Game on!

It's going to be a competition this time -- Anand won game 3, to even the score at 1.5 to 1.5.

Super!

Sunday, November 9, 2014

An early lead for Carlsen

After two games in the 2014 World Chess Championship, Magnus Carlsen has won game two.

In game one, Anand had white, and although it was hard-fought, Carlsen held the draw.

In game two, Carlsen had white, and won.

The details of the games, of course, are far above my ability to comprehend, but they are certainly beautiful to watch.

Wednesday, November 5, 2014

Carlsen Anand II

There's just three days until Carlsen-Anand begins.

The match will be held in Sochi, Russia: here's the official site.

The games are played at 3 PM Sochi time; according to WorldTimeBuddy, this will be 4 AM my time.

So I can wake up each day with the very best of chess!

Michael Aigner has a nice preview here.