OS X tarballs: a saga of extended attributes

Are you a Mac user? Creating a tarball? Beware!

Today I ran into a problem where a tar file that I created had a bunch of ._ files:

$ tar tvf ../bokeh-0.2.tgz
drwxr-xr-x 0 pwang staff 0 Oct 23 15:12 bokeh-0.2/
-rw-r--r-- 0 pwang staff 226 Oct 23 15:04 bokeh-0.2/._.gitattributes
-rw-r--r-- 0 pwang staff 31 Oct 23 15:04 bokeh-0.2/.gitattributes
-rw-r--r-- 0 pwang staff 226 Oct 23 15:04 bokeh-0.2/._.gitignore
-rw-r--r-- 0 pwang staff 1047 Oct 23 15:04 bokeh-0.2/.gitignore
-rwxr-xr-x 0 pwang staff 226 Oct 23 15:04 bokeh-0.2/._bokeh
drwxr-xr-x 0 pwang staff 0 Oct 23 15:04 bokeh-0.2/bokeh/
-rwxr-xr-x 0 pwang staff 226 Oct 23 15:04 bokeh-0.2/._bokeh-server
-rwxr-xr-x 0 pwang staff 1681 Oct 23 15:04 bokeh-0.2/bokeh-server
-rw-r--r-- 0 pwang staff 226 Oct 23 15:04 bokeh-0.2/._CHANGELOG
-rw-r--r-- 0 pwang staff 1388 Oct 23 15:04 bokeh-0.2/CHANGELOG
...

When I looked in the source directory which I was tarring up, I see no such files. What gives?

It turns out that OS X’s version of tar will automatically create these ._ versions for each file that has extended properties and attributes, because it does not want that information lost. Extended attributes are an aspect of the HFS+ filesystem which OS X uses. If you ever do an “ls -l” and see an “@” symbol near the permissions on your files, that indicates that it has extended attributes:

$ ls -l
total 128
-rw-r--r--@ 1 pwang staff 1388 Oct 23 15:04 CHANGELOG
-rw-r--r--@ 1 pwang staff 2587 Oct 23 15:04 QUICKSTART.md

To list these attributes, you do “ls -@”:

$ ls -@l
total 128
-rw-r--r--@ 1 pwang staff 1388 Oct 23 15:04 CHANGELOG
com.apple.quarantine 71
-rw-r--r--@ 1 pwang staff 2587 Oct 23 15:04 QUICKSTART.md
com.apple.quarantine 71

To strip these off, you use the xattr command:

$ xattr -d com.apple.quarantine CHANGELOG

To strip the attributes off of all files in a directory tree, use find:

$ find . -xattr -exec xattr -d com.apple.quarantine {} \;

Now you can create a tarball that won’t be offensive to your friends!

Compilers, Runtimes, and Users – Oh my!

I’ve been meaning to respond to this since last week, but was totally caught up with SciPy and a product launch. Better late than never.

This is a response to Alex Gaynor’s response  to my previous post, “Does the Compiler Know Best?”

I plan to respond to some of the more specific points from Alex in the comment stream of his blog post itself. However, I am writing a new post here to reiterate the broader picture of the forest which I think is being missed for the trees. Specifically: I wrote the blog post in response to the disturbing rise in comments of the form “Why shouldn’t PyPy be the official implementation of Python?” I think part of Alex’s (and other PyPy devs’) confusion or consternation at the post is due to my failure to more clearly identify the intended audience. The post was meant as a response to those voices in the peanut gallery of the blogo/twitter-sphere, and was not meant as a harangue against the PyPy community.

However, I do stand firmly by my core point: a huge part of Python’s success is due to how well its runtime plays with existing codebases and libraries, and it would be utterly foolish to abandon that trump card to join in the “who has the fastest VM” race.

This is wrong. Complete and utterly. PyPy is in fact not a change to Python at all, PyPy faithfully implements the Python language as described by the Python language reference, and as verified by the test suite.

There is certainly no argument that PyPy is a faithful implementation of the Python official language standard. It passes the test suites! However, do the test suites reflect the use cases out in the wild? Or do those not matter? On the basis of the many years of consulting I have done in applying Python to solving real problems at companies with dozens, hundreds, or even thousands of Python programmers, my perspective is that CPython-as-a-runtime is almost as important as Python-as-a-language. They have co-evolved and symbiotically contributed to each other’s success.

It is not my place, nor is it my intent, to tell people what they can and cannot implement. My one goal is to voice a dissenting opinion when I feel the ecosystem is presented with proposals which I think are either short-sighted or damaging. My previous post was in response to the clamor from those who have only been exposed to Python-the-language, and who only have visibility to those surface layer use cases.

Second, he writes, “What is the core precept of PyPy? It’s that “the compiler knows best”.” This too, is wrong. First, PyPy’s central thesis is, “any task repeatedly performed manually will be done incorrectly”, this is why we have things like automatic insertion of the garbage collector, in preference to CPython’s “reference counting everywhere”, and automatically generating the just in time compiler from the interpreter…

Interestingly enough, CPython’s reference-counting based garbage collection scheme is sometimes cited as one of the things that makes it integrate more nicely with external code. (It might not exactly be easy to track all the INCREFs and DECREFs, but the end result is more stable and easier to deal with.) And there is no problem with auto-generating the JIT compiler, except that (as far as I know) there is not a well-defined API into it, so that external code can interoperate with the compiler and the code it’s running.

It would appear that there is a one way funnel from a user’s Python source and the RPython interpreter through the PyPy machinery, to emit a JITting compiler at the other end. This is fine if the center and the bulk of all the action is in the Python source. However, for a huge number of Python users, this is simply not the case. For those users, having an opaque “Voila! Here’s some fast machine code!” compiler pipeline is not nearly as useful as a pipeline whose individual components they can control.

And that is the primary difference between a compiler and an interpreter. The interpreter has well-defined state that can be inspected and modified as it processes a program. A compiler has a single-minded goal of producing optimized code for a target language. Of course, the PyPy project has a compiler and an interpreter, but the generated runtime is not nearly as easy to integrate and embed as CPython. I will say it again: CPython-the-runtime is almost as important as Python-the-language in contributing to the success of the Python ecosystem.

Second, the PyPy developers would never argue that the compiler knows best, … having an intelligent compiler does not prohibit giving the user more control, in fact it’s a necessity! There are no pure-python hints that you can give to CPython to improve performance, but these can easily be added with PyPy’s JIT.

Quick quiz: How long has Python had static typing?
Answer: Since 1995, when Jim Hugunin & others wrote Numeric.

Numeric/Numarray/Numpy have been the longest-lived and most popular static typing system for Python, even though they are generally only used by people who wanted to statically type millions or billions of memory locations all at once. CPython made it easy to extend the interpreter so that variables and objects in the runtime were like icebergs, with a huge amount of sub-surface mass. The perspective of CPython-the-runtime has been one of “extensible & embeddable”.

Does it matter that these extensions were “impure”? Did that hurt or help the Python ecosystem?

At the end of the day, it comes down to what users want. In the last 5 years, since Python has successfully regained mindshare in the web development space that was lost to RoR, there have been a large number of relative new-comers to the ecosystem whose needs are much more focused on latency and I/O throughput, and for whom extensions modules are an afterthought. For these users, there is a single-minded focus on raw VM performance; the closest they will ever get to an extension module is maybe a DB driver or some XML/text processing library or an async library. It’s understandable that such people do not understand what all the fuss is, and why they might vocally push for PyPy to replace CPython as the reference implementation.

My experiences with Python and the users that I’ve been exposed to are a much different crowd. I don’t think they are any fewer in number; however, they are usually working at large companies and banks or military and government, and generally do not tweet and blog about their technology. I have sat in numerous corporate IT discussions where Python has been stacked up against Java, .Net, and the like – and in all of these things, I can assure you that the extensibility of the CPython interpreter (and by extension, the available of libraries like NumPy) have been major points in our favor. In these environments, Python does not exist in a vacuum, nor is it even at the top of the foodchain. Its adoption is usually due to how well it plays with others, both as an extensible and as an embeddable VM.

The reason I felt like I needed to write a “dissenting” blog post is because I know that for every comment or Reddit/HN comment, there are dozens or hundreds of other people who are watching the discussions and debate, and not able to (or willing to) comment, but who have internal corporate discussions about technology direction. If Python-the-community were to deprecate CPython-the-runtime, those internal discussions would head south very quickly.

My post was directed specifically at people who make comments about what Python should or should not do, without having good insight into what its user base (both individual and corporate) look like. The subtext of my post is that “there are voices in the Python ecosystem who understand traditional business computing needs”, and the audience of that subtext is all the lurkers and watchers on the sidelines who are constantly evaluating or defending their choice of language and technologies.

Belated thoughts on Facebook’s IPO

Paul Graham wrote a letter to various YCombinator companies in response to Facebook’s IPO.  In the discussion, he makes the comment:

Plus Mark himself is such a fearsomely effective person. And so young; he’s only a little older now than Larry and Sergey were when they started Google; so he still has the energy and mental flexibility that’s usually the only asset of founders just starting out.

I actually think this is backwards: The primary reason why Facebook can be successful is because it is organized such that it could realize the vision of a technical founder, who has a foot in tech and a foot in business.

You’re right that they haven’t tried to make money yet. I once raged to a friend at Google that Facebook’s use of ads displayed an utter lack of creativity or seriousness about building a business. (I also argued that Google should not try to tackle Facebook on its home turf, but if it really wanted to attack it head-on, it would need to build a destination site instead of merely “socializing” all their products. Sadly, that destination site turned out to be G+, and then they used it to socialize their products. Sigh.)

There are two Facebooks. One is an identity service, built on a noisy social graph accreted over the years, which Zuckerburg is hoping to make as fundamental as DNS for the modern non-anonymized web. The other is a micro-blogging and photo sharing site on top of that identity service. Both are going to be hard to make money with.

It is really difficult to directly monetize the core identity service. They can attempt to provide the service as a bona fide piece of Internet technology (e.g. opening up retail locations to verify accounts in-person, or by partnering with wireless providers that sell their eventual phone), but then governments and others will get into the act and require open API access to the user information and graph data. As soon as that happens, they’ve given out their crown jewels.

Any other mechanism for monetizing the identity service is tantamount to providing advertising on the back of your driver’s license, or making your social security card into a frequent-diner loyalty card. People won’t like it, privacy advocates and anti-corporate doomsayers will have a field day, and governments will start interfering with their core tech. (A good strategy for Google might be to work on a stealth, open-ID based “Plan B” to have in their back pocket, for the day if/when Facebook does shoot itself in this foot like this. They’ve certainly screwed the privacy pooch before.)

And as for the other half of Facebook, they are in a crowded, fickle space, and their offering there isn’t actually that great (and I say that as a user and a technologist). Twitter, Pinterest, and a host of small startups are very real threats. Mark paid $1B to keep Instagram from Twitter and maybe Google; how many more of those can he afford? Furthermore, Apple has enoughcash in the bank to buy two Facebooks, even at its massively inflated P/E, and its gaming platform is tied to a much broader, more sustainable group of customers than Zynga’s “whales”. I’ve personally spent over $100 on iOS games – and barely noticed that that was the case. It was easy, casual, natural, and I will probably continue to dump more money into the iOS ecosystem, because it works great. I’ve spent exactly $1.98 to buy two Zynga games, mostly to get rid of the annoying ads, and I’ve stopped playing both.

Lastly, Amazon, whose customer profiles include credit cards and addresses, and whose social information is tied in to actual purchasing intent, has yet to reveal the punchline in its Kindle strategy. Just as ‘selling books’ was not the ultimate purpose of the company, I am fairly certain that ‘reading books’ is not the ultimate purpose of the Kindle. One concept: A Kindle fire with a barcode scanner turns every single aisle in every brick & mortar store in the country (with 3G signal) into a showroom for Amazon. Add passive background RFID scanning as the user walks the aisles, and it’s a brave new world of retail.

TLDR: Mark is a smart and capable guy, now backed with fresh cash, but he’s got to apply some creative thinking to demonstrate that he can actually milk his cash cow. His worst case scenario is to become basically Verisign, and I have not seen them demonstrate any new thinking to suggest this will not be the case.

Does the compiler know best?

Ted Dziuba recently blogged about Python3’s Marketing Problem. I chimed in on the comment thread, but there was a deeper point that I felt is missed in the discussions about the GIL and PyPy and performance.  Lately I’ve seen more and more people expressing sentiments along the lines of:

I’m of the same mind, but think that instead of offering a GIL fix, the goodie should have been switching over to PyPy. That would have sold even more people on it than GIL removal, I think.

I know it is an unpopular opinion, but somebody’s got to say it: PyPy is an even more drastic change to the Python language than Python3. It’s not even a silver bullet for performance. I believe that its core principles are, in fact, antithetical to the very things that have brought Python its current success. This is not to say that it’s not an interesting project. But I really, really feel that there needs to be a visible counter to the meme that “PyPy is the future of Python performance”.

What is the core precept of PyPy? It’s that “the compiler knows best”. Whether it’s JIT hotspot optimization, or using STM to manage concurrency, the application writer, in principle, should not have to be bothered with mundane details like how the computer actually executes instructions, or which instructions it’s executing, or how memory is accessed. The compiler knows best.

Conversely, one of the core strengths of Python has been that it talks to everybody, because its inner workings are so simple. Not only is it used heavily by folks of all stripes to integrate legacy libraries, but it’s also very popular as an embedded scripting system in a great number of applications. It is starting to dominate on the backend and the front-end in the computer graphics industry, and hedge funds are starting to converge on it as the best language to layer on top of their low-level finance libraries.

If you doubt that transparency is a major feature, you simply have to look at the amount of hand-wringing that JVM folks do about “being hit by the GC” to understand that there, but by the grace of Guido, go we. If we have to give up
ease of embedding and interoperability, and visibility into what the running system is doing, for a little improvement in performance, then the cost is too steep.

It’s understandable that those who see Python as merely a runtime for some web app request handlers will have a singular fixation with “automagically” getting more performance (JIT) and concurrency (STM) from their runtime. I never thought I’d say this, but… for those things, just fucking use Node.js. Build a Python-to-JS cross compiler and use a runtime that was designed to be concurrent, sandboxed, lightweight, and has the full force of Google, Mozilla, Apple, and MSFT behind optimizing its performance across all hardware types. (It would not surprise me one bit if V8+NaCl finally became what the CLR/DLR could have been.) Armin and the PyPY team are incredibly talented, and I think Nick is probably right when he says that nobody has more insight and experience with optimizing Python execution than Armin.

But even Armin has essentially conceded that optimizing Python really requires optimization at a lower level, which is why PyPy is a meta-tracing JIT. However, PyPy has made the irreversible architectural decision that that level should be merely an opaque implementation detail; the compiler knows best.

An alternative view is that language runtimes should be layered, but always transparent.

Given the recent massive increase of commercial investment in LLVM, and the existence of tools in that ecosystem like DragonEgg, syntax really ceases to be a lock-in feature of a language. (Yes, I know that sounds counter-intuitive.) Instead, what matters more is a runtime’s ability to play nicely with others, and of course its stable of libraries which idiomatically use that runtime. Python could be that runtime. Its standard library could become the equivalent of a dynamic language libc.

Python gained popularity in its first decade because it was a non-write-only Perl, and it worked well with C. It exploded in popularity in its second decade because it was more portable than Java, and because the AMD-Intel led to spectacular improvements in CPU performance, so that an interpreted language was fast enough for most things. For Python to emerge from its third decade as the dynamic language of choice, its core developers and the wider developer community/family will have to make wise, pragmatic choices about what the core strengths of Python are, and what things are best left to others.

View in this light, stressing Unicode over mere performance is a very justifiable decision that will yield far-reaching, long term returns for the language. (FWIW, this is also why I keep trolling Guido about better DSL support in Python; “playing nicely with others” in a post-LLVM world means syntax interop, as well.)

The good news is that the python core developers have been consistently great at making pragmatic choices. One new challenge is that the blogosphere/twittersphere has a logic unto itself, and can lead to very distracting, low signal-to-noise ratio firestorms over nothing. (fib(), anyone?) Will Python survive the noise- and gossip-mill of the modern software bazaar? Only time will tell…

Tagged

A Sketch of the Future of (Mobile) Computing

I saw two interesting tech news items from this morning.

Google Begins Testing Its Augmented Reality Glasses
and
Motorola is Turning Android into a Desktop OS

In the future, what data/apps/preferences are not stored in the cloud and streamed to your device will be encapsulated in a small digital token that you keep with you and plug into any available local hardware. The idea of lugging around a laptop or even a phone that is a physical container for your data will be utterly outdated. Consider: Your iPhone contains about 32GB of storage. You can, today, go into a Best Buy and get a 32GB mini-SDHC card that is the size of your pinky nail. The SIM card of your phone is equal in size.

So, the only thing that distinguishes your phone from any other phone in the world can physically fit on something the size of a fingernail. The only challenges are software ones: apps would need to recognize a larger set of hardware than they currently do, but Apple, Google, and Microsoft all have their own strategic initiatives to tackle that challenge.

So instead of plugging a phone into a pad (like the motorola thing), you will plug a tiny data crystal into any computing device, and have your data, your apps, your contacts, your photos, etc. all right there.

The augmented reality glasses are a way for you to always have some of your data available to you, even if you are away from a larger computing devices. Like with my Looxcie, it will be life streaming your experiences to your data crystal over bluetooth, and will have a minimal voice-activated dialer/phone interface that uses whatever local network is available. If you are on a mobile cellular network, it will use the subscriber information off your data crystal to connect to the cellular provider; if you are on a Wifi, it will use standard internet. (FaceTime & iMessage already does this.)

The crux is that data has traditionally been confined/jailed in physical devices, but storage has gotten so cheap and bandwidth has become so pervasive that this no longer makes sense. So the real challenge is to deliver a software development platform (and ecosystem) that allows developers to target multiple devices, contexts, and usage environments. Apple wants to do this by unifying iOS and desktop, and having storage use iCloud. Google wants to use Android as the underlying mobile OS, but both Google and Microsoft are betting on HTML and the web as the application environment.

Javascript Refresher Cheat Sheet

Last week I dove back into Javascript after being away from it for a while, and I wrote up a “Javascript Refresher Cheat Sheet” on the basis of some of the books and web resources I read.

Watson is winning at buzzing, not Jeopardy

It’s been inspiring to watch IBM’s Watson kicking butt on Jeopardy, since I am a scientific programmer and understand the difficulty of the problem the Watson team is attempting to solve. However, I can’t help but notice that Watson seems to be having much better luck nailing the buzzer, compared to its human counterparts.

Years ago, a family friend ended up on Jeopardy, and after the experience, she commented that she had underestimated the importance of finessing the buzzer. A Google search for “jeopardy buzzer” turns up quite a few pages, including this very informative page entitled “How to Win on the Buzzer”, by Michael Dupee, a former Jeopardy contestant.

It’s not clear to me how Watson is notified that Alex Trebek has finished reading the clue, but it seems pretty clear to me that no matter how you do it, the computer clearly has an advantage. If, for instance, the computer is simply sent a signal that is wired in to the same system that the off-stage assistant uses to enable the human contestants’ buzzers, then the computer can instantly respond with almost zero latency as soon as it gets the signal. There is no way that a human can compete with that, because the humans that rely on the pin light to notify them of buzzer activation will always be late. Alternatively, those that try to “time” the assistant and guess when he feels Trebek has finished reading the clue will never have the microsecond accuracy that Watson has.

Alternatively, if a direct signal is not sent, but rather Watson is equipped with an audio sensor to process Trebek’s voice as he reads the clue, it’s very easy to write a simple optimization routine that quickly learns when the assistant activates the buzzers. Watson can expend a tiny, minuscule fraction of a single processor to this task, and still be orders of magnitude more accurate at timing than its human competitors.

The point is that if one were to replace Watson with a human being that is every bit as knowledgeable and capable as Watson, the human being would not fare nearly as well in competition, simply because his or her motor response cannot beat that of a specialized robot. So, while Watson’s ability to understand and solve open-ended Jeopardy clues is certainly impressive, the reason he is trouncing the humans seems to have more to do with robotics rather than reasoning.

Screw you, Yelp.

A few days ago I received an email from “Yelp HQ” informing me that my review of Case Handyman & Remodeling had been flagged, and after review, they decided that my inclusion of the links to the detailed remodeling entries on my blog were “promotional” and violated their terms of service. An excerpt from the email:

“I’m writing because your review of Case Remodeling of Austin has been flagged by the community, and after evaluation, our Support team has determined that the review violates our Terms of Service (http://www.yelp.com/static?p=tos). Because personal accounts cannot be used in any promotional manner, the links to your blog that you’ve included in your review are problematic and will need to be removed.”

Naturally, I sent a response:

Hi Miranda,

I’m curious what aspects of the Terms of Service have been violated? I re-read the Terms of Service, and the Content Guidelines, and could not really find anything that pertains to my review. I linked to the detailed remodel information at the end of my review because it is too much content to post in a Yelp review; it includes day-by-day breakdowns, a lot of detailed information about the kinds of dysfunctional communication and interaction I had with Case Handyman &
Remodeling, and many photos.

The “promotional content” part of the Yelp Content Guidelines seem to specifically address people who post links that promote their own businesses and such. I am hardly doing that, but rather am instead providing more information and context for other Yelp users. Again, I simply do not see how this violates the letter or the spirit of the Terms of Service.

Of course, just because Yelp emails you, doesn’t mean that you have the privilege of emailing them back:

From: “Yelp Team”
To: XXXXX@gmail.com
Thanks for emailing Yelp.
Unfortunately, you have reached an email address that is not in use.

And now I offer a fun little challenge: try to find the email address of someone at Yelp that you can write about this. No luck? Yeah. You have to use their web form. Fuck that. If you email me, threatening to remove content that I provided for free for your site, then it’s not really right to make me jump through hoops and fill out CAPTCHAs just to respond to you. In the spirit of making a good-faith effort, I even tried tweeting @Yelp, to no avail.

So, that’s it. That’s my last Yelp review. Others who want to use Case Handyman & Remodeling or deal with Ed Dudley and get screwed, you can thank the nice folks at Yelp HQ for removing my review that could have spared you pain.

You know what the real irony is? The real irony is that the person who flagged my review is probably someone associated with the business itself. Good job, Yelp!

For reference, here is my review in full. You can decide for yourself if it is “promotional” in nature about my blog:

I’m currently using CASE for remodeling about 800 sq ft of my house, including building a brand new roof, kitchen, master bedroom and bath, and deck. Thus far, the construction quality and timeliness of the project have been OK. Our construction manager José and his crew do a pretty good job, but some of the subcontractors have left a bit to be desired. On the plus side, they at least showed up on time and got the work done (for the most part).

The financial side of the project is a completely different story. We were initially led to believe that this would be a fixed-bid contract with potentially a few change orders if they discovered things that needed to be fixed along the way. Well, as it turns out, despite the fact that we demolished half of the house and rebuilt from the ground up, there was still a tremendous amount of unexpected work. We are currently a whopping 30% over the contract amount and at change order #23. (I could have bought myself a very sweet car with the overages that we’ve had.)

During the initial discussion phase of the project, Cliff Zoch (who was the remodeling consultant we worked with) indicated that CASE really preferred to do fixed-bid contracts and pooh-poohed the “industry standard” practice of lowballing the inital contract to earn business and making back profit margins on change orders. Well, based on the progression of our project thus far, that is exactly what is happening to us.

We still have about 3 weeks left on our remodel, and I will update this once it is complete. I am also blogging my remodel at http://pwang.wordpress.com/category/remodel/ and I have a fairly comprehensive overview of our project at http://pwang.wordpress.com/2009/05/01/project-overview/. I have posted many pictures and I detail the joys and travails of our ongoing project, including interactions with various subcontractors.

Contortions (and eventual success) with pydistutils.cfg

I finally upgraded to Snow Leopard (OS X 10.6) this past weekend, and the first order of business was to get Python configured the way I wanted.  I had previously been using a very custom install based on the old “Intel Mac Python 2.5″ notes that Robert Kern wrote up for ETS developers/users, and I had resolved to be more intentional and organized about how I managed the installation of Python packages on my new system.

So, I first installed EPD, the Enthought Python Distribution as a base. Then I created a ~/.pydistutils.cfg file with the contents as outlined in the Python docs on Installing Python Packages:

[install]
install-base=$HOME/Library/Python2.6
install-purelib=site-packages
install-platlib=plat-mac
install-scripts=scripts
install-data=data

I then tried to install Mercurial using the one-liner:


$ easy_install mercurial

And I was promptly greeted with the error:


error: install-base or install-platbase supplied, but installation scheme is incomplete

WTF?

Google turned up nothing of substance, save for a link to an old subversion commit of distutils/commands/install.py. Taking this as a sign, I opened up my local copy of the file and a brief code read revealed the source of the problem: I was missing the install-headers option. So, I added the line:

    install-headers=Include

And was greeted by a different error:

install_dir site-packages/
TEST FAILED: site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    site-packages/

and your PYTHONPATH environment variable currently contains:

    '/Users/pwang/Library/Python2.6/site-packages:/Users/pwang/Library/Python2.6/plat-mac:'

Well, this was most disheartening. I was, after all, following the Python docs, which seem to imply that install-purelib would be appended to install-base. The above error message suggests that this was not the case, so I went back to the distutils source, and more code reading and tracing seemed to confirm this. So, I added an explicit $base to all of the config lines in my pydistutils.cfg, with a final result that looked like this:

[install]
install-base=$HOME/Library/Python2.6
install-purelib=$base/site-packages
install-platlib=$base/plat-mac
install-headers=$base/Include
install-scripts=$base/scripts
install-data=$base/data

This, finally, seemed to work. easy_install mercurial worked great, and everything installed into the proper locations. One thing to note was that the $base variable in pydistutils.cfg needs to be lower case.

Hopefully this entry will turn up the next time someone searches for “install-base or install-platbase supplied, but installation scheme is incomplete” and they are spared having to dig through the distutils source.

How I Smoke a Brisket

Below is a detailed description of how I make a brisket on my Big Green Egg, including my dry rub recipe.  My early attempts at this years ago were plagued with difficulties, and the meat didn’t turn out great, but I have finally figured out how to consistently do it well.  The single most important thing about smoking meat with the Big Green Egg is learning how to correctly build the fire.  If your fire isn’t built right, you will have nothing but trouble.  I describe the key aspects of this process later.

But first, let’s start with prepping the brisket itself.

Continue reading

Follow

Get every new post delivered to your Inbox.