I love everything about living in San Diego, even after an El Nino winter. The one thing I absolutely cannot stand is traffic. It's not that my area (UCSD) is terribly congested, it's that the traffic lights have a perverse way of slowing you down for no reason at all.
It all starts with the people. San Diegans are your average Southern Californians, to whom being in a car is a way of life and the availability of distraction is a fundamental civic right. As a result, traffic moves in the strangest patterns, and people have the hardest time getting started after a light turns green.
The next thing that is unpleasant are the giant boulevards that make up most of the roadways here. Usually, that should speed up traffic, but since there are many malls and the weather is nice, there are a lot of pedestrians (good) that take up incredibly long to cross these inner city speedways.
Which gets us finally to the final piece in my traffic tragedy: the traffic lights. They were built some time in the 80s or 90s and have triggers that tell the light when someone drove up. The idea, back in the day, was that traffic was going to flow better if the lights turned only as needed. Sounds good, right? The light stays green for the main direction of traffic for the longest time, and only when someone needs to turn does the light follow suit. Similarly, as soon as the main direction of traffic is in a lull, the light turns and lets the "minority direction" pass.
That sounds plausible. Now imagine what happens when traffic is moderately heavy, like most of the day: car trickle in from all different directions at about constant rate. The enormous amount of time it takes to cross an intersection plus the slow response times of San Diegans conspire to creating extremely long cycles for the lights, which means there is pretty much always someone waiting for the light to turn.
Perversion comes in: you get to a light that is turning red. You slow down, you wait for the light to turn green again. Then you start, and you get to the next light. Of course, since you were waiting with everybody else at the light before, there is no traffic ahead of you. A lull. The next light interprets it as a lull in traffic and turns red. You get to the next light just in time for it to turn, and you wait at the next light, too.
The perversion continues: the same story repeats light after light, and to cover the short distance of UCSD campus, you take 12 minutes - three times as long as you would when the lights don't turn, or 8 minutes of waiting for 4 minutes of driving.
And that, dear folks, just drives me crazy.
2010-03-15
2010-03-08
YHIHF: The Open Society
Story from my youth: a family friend from Italy married a German lady. Italians tend to be lax about government, Germans punctilious. When time came for the yearly income tax declaration, the man prepared the forms at the very last second, as Italians are wont to do, and gave them to his German wife to submit the morning after.
Curious as she was, the German lady looked at the forms are shouted in disbelief, "Husband, that's not your real income! You make three times as much! You are a government employee, you can't declare to the government that you make less than the government pays you!"
I know what you are all thinking: what the heck was the guy telling his wife how much money he made if he didn't tell the government. But that's not the focus of the story. Instead, the husband looked at the German wife (whose strictness he may or may not have found slightly arousing) and told her, "Wife, if I declare more, then everybody in the tax office is going to ask why this one guy makes three times as much as everybody else in the office!"
It used to be indeed the case that information was something hard to come by. Not any more. Now, the government knows how much you make before you do, and information about you is available everywhere. For those of us that grew up in the past, this is a terrifying notion. In one of my previous jobs, I worked for a company that worried a lot about cyber-security, and I knew very well what kinds of threats there were and how you could exploit things.
This coming generation, though, the Millennials are going for a completely different approach. They are simply ignoring the risks. Sometimes they wake up with a bad headache, sometimes an old photo of their drinking under age, posted on Facebook a decade prior, surfaces to prevent them from getting a job as librarian. But all in all, they seem to be pretty happy the way they live.
Now, the thing is that the approach is not bad at all, not even really that insecure - it's just that you have to go all the way with information. You either keep it close or you open it all up, and the end result is the same. It's just in the netherworld of information openness that problems occur.
What do I mean? Currently, we have two economies that run in parallel and with a few exchange points: cash and electronic. Cash transactions are non-traceable (hence the information content is low). Electronic transactions are traceable. If all exchanges happened electronically, we wouldn't have to ever worry about theft, If something was taken from you, there would be a record of where that money went, who took it, and ways to get it back. It's only when you can turn electronic money into cash that problems arise, because nobody knows where cash goes.
That's not limited to money. Another example: GPS and the laws of the road. A "friend of mine" rode his motorcycle on I-5, from San Diego to San Francisco. Posted speed limit: 65. Minimum speed on the freeway: 90. This "friend of mine" was pretty much forced to drive at everybody's speed, since the freeway was congested and the greatest risk for a motorcycle is being hit from behind.
Now, with GPS in all cell phones, there is certainly the possibility and eventually the certainty that the government (in this case, Highway Patrol) will gain the right to track your speed. What happens then? Will the information be something we hold on to until we are forced to surrender it because of an accident?
Fact is, there is something deeply hypocritical about a law that everybody ignores. Posting a speed limit that is 25 mph lower than the average speed of traffic is a puzzling act. That we acquiesce to it is even more puzzling. We don't care, because it's not enforced.
Imagine what will happen, though, when law enforcement will gain access to your current location and speed: you will be automatically fined whenever you exceed the speed limit, which in current traffic is probably virtually everywhere and at all times, at least on California freeways outside of congested areas.
What about the lady whose drunken photo was found during a routine web search by her prospective employer? What about the man, just diagnosed with cancer, that was dropped by his health insurance because he hadn't declared a wart for which he had been treated? Are those urban myths?
It doesn't really matter: the essence is clear, information can be negative for you. Of course, the opposite is true, as well. Employers that discriminate (like in the case of the "drunken lady") will be noted for their bigotry. Health insurance providers that stiff the customers will be crucified.
Europe, always a little more backwards in time, has declared that it intends to combat information overload by making information private. I understand where Europe comes from, but America has always been an open society, and it's time to open it up even more. Where there have been circumstantial barriers, we should remove them as soon as they become visible.
Once the Highway Patrol notices that 90 mph is the norm on a stretch of highway without causing accidents, the legislature needs to notice. Once it is clear that a particular pattern is used for discriminating behavior, the legislature needs to act.
Curious as she was, the German lady looked at the forms are shouted in disbelief, "Husband, that's not your real income! You make three times as much! You are a government employee, you can't declare to the government that you make less than the government pays you!"
I know what you are all thinking: what the heck was the guy telling his wife how much money he made if he didn't tell the government. But that's not the focus of the story. Instead, the husband looked at the German wife (whose strictness he may or may not have found slightly arousing) and told her, "Wife, if I declare more, then everybody in the tax office is going to ask why this one guy makes three times as much as everybody else in the office!"
It used to be indeed the case that information was something hard to come by. Not any more. Now, the government knows how much you make before you do, and information about you is available everywhere. For those of us that grew up in the past, this is a terrifying notion. In one of my previous jobs, I worked for a company that worried a lot about cyber-security, and I knew very well what kinds of threats there were and how you could exploit things.
This coming generation, though, the Millennials are going for a completely different approach. They are simply ignoring the risks. Sometimes they wake up with a bad headache, sometimes an old photo of their drinking under age, posted on Facebook a decade prior, surfaces to prevent them from getting a job as librarian. But all in all, they seem to be pretty happy the way they live.
Now, the thing is that the approach is not bad at all, not even really that insecure - it's just that you have to go all the way with information. You either keep it close or you open it all up, and the end result is the same. It's just in the netherworld of information openness that problems occur.
What do I mean? Currently, we have two economies that run in parallel and with a few exchange points: cash and electronic. Cash transactions are non-traceable (hence the information content is low). Electronic transactions are traceable. If all exchanges happened electronically, we wouldn't have to ever worry about theft, If something was taken from you, there would be a record of where that money went, who took it, and ways to get it back. It's only when you can turn electronic money into cash that problems arise, because nobody knows where cash goes.
That's not limited to money. Another example: GPS and the laws of the road. A "friend of mine" rode his motorcycle on I-5, from San Diego to San Francisco. Posted speed limit: 65. Minimum speed on the freeway: 90. This "friend of mine" was pretty much forced to drive at everybody's speed, since the freeway was congested and the greatest risk for a motorcycle is being hit from behind.
Now, with GPS in all cell phones, there is certainly the possibility and eventually the certainty that the government (in this case, Highway Patrol) will gain the right to track your speed. What happens then? Will the information be something we hold on to until we are forced to surrender it because of an accident?
Fact is, there is something deeply hypocritical about a law that everybody ignores. Posting a speed limit that is 25 mph lower than the average speed of traffic is a puzzling act. That we acquiesce to it is even more puzzling. We don't care, because it's not enforced.
Imagine what will happen, though, when law enforcement will gain access to your current location and speed: you will be automatically fined whenever you exceed the speed limit, which in current traffic is probably virtually everywhere and at all times, at least on California freeways outside of congested areas.
What about the lady whose drunken photo was found during a routine web search by her prospective employer? What about the man, just diagnosed with cancer, that was dropped by his health insurance because he hadn't declared a wart for which he had been treated? Are those urban myths?
It doesn't really matter: the essence is clear, information can be negative for you. Of course, the opposite is true, as well. Employers that discriminate (like in the case of the "drunken lady") will be noted for their bigotry. Health insurance providers that stiff the customers will be crucified.
Europe, always a little more backwards in time, has declared that it intends to combat information overload by making information private. I understand where Europe comes from, but America has always been an open society, and it's time to open it up even more. Where there have been circumstantial barriers, we should remove them as soon as they become visible.
Once the Highway Patrol notices that 90 mph is the norm on a stretch of highway without causing accidents, the legislature needs to notice. Once it is clear that a particular pattern is used for discriminating behavior, the legislature needs to act.
2010-03-03
OpenServices.org
Well, the name is not that important - I use it just to conceptualize an idea. What the actual implementation should be named, well, that's up for grabs. The idea, though, is phenomenally good. I came up with it, after all. (Where is your sarcasmey when you need one?)
You know all about open source. One of the things you might know about open source that is not about open source is that most of the largest web infrastructure companies use open source software throughout their stack. Be it Yahoo!, Google, Amazon, Facebook, Twitter, and what you have: if they are big, they use open source.
(The only major exception, for obvious reasons, is MSN/Live/Bing.)
The sad thing is that these companies (with the notable exception of Google) rarely give something back to open source. In particular, not even Google releases the source of their main site software. Of course we understand that (sort of). But if there is a market for open source software, why not for open source services?
What is the basic idea of openservices.org? It would be a site or a series of federated sites that not only publish all the software they are written with, but also function in an open source fashion. That means that, while they are run by a group of dedicated and security/stability-conscious admins, the software that runs on them is the result of open collaboration.
You'll say, but we already have something like that! It's called Wikipedia! And I say, well yes, that's exactly what I am thinking of. But more than Wikipedia. Slay the dragon and bypass all the stupid restrictions and costs of the sites you know.
Let's go a step back. In Internet Theory, we classify four types of sites:
One of the strange anomalies of the Internet is the kinds of sites that can attract large amounts of paying customers. There are porn sites, of course, but there are more free alternatives available now, so that market is drying out.
There are betting sites. They derive their attractiveness mostly from the fact they or their content are banned from several jurisdictions (most notably the United States or part of them). As a result, the demand for such sites can only be met by illegal or semi-legal operations, which as usual create a black market.
There are gaming sites. Here the attractiveness comes from the quality of the product alone, and from the fact that the games become dull after a while. I am talking about sites like World of Warcraft. But the amount of effort that goes into a good online game is way ahead of the pure programming, as places like OpenSim show.
Then there is the last one: personals sites. The application is simple, almost painfully so. From a programmer's perspective, it's a collection of profiles and some way to match them. On many sites, the way to match is simply by searching using a form that might as well have been created in 1994. I would call that the lowest-hanging fruit.
How would OpenServices.org work? There would be four arms:
Now, imagine a site that works with a volunteer effort. Senior contributors get fancy titles and supervise (which also means, mentor) junior members. Coming out of, say, college, someone might already have several years of system administration experience, or creative design experience.
The only thing required for this to work is enough money for servers and bandwidth. Seems a win-win scenario for everybody. With the possible exception of match.com and eHarmony.
You know all about open source. One of the things you might know about open source that is not about open source is that most of the largest web infrastructure companies use open source software throughout their stack. Be it Yahoo!, Google, Amazon, Facebook, Twitter, and what you have: if they are big, they use open source.
(The only major exception, for obvious reasons, is MSN/Live/Bing.)
The sad thing is that these companies (with the notable exception of Google) rarely give something back to open source. In particular, not even Google releases the source of their main site software. Of course we understand that (sort of). But if there is a market for open source software, why not for open source services?
What is the basic idea of openservices.org? It would be a site or a series of federated sites that not only publish all the software they are written with, but also function in an open source fashion. That means that, while they are run by a group of dedicated and security/stability-conscious admins, the software that runs on them is the result of open collaboration.
You'll say, but we already have something like that! It's called Wikipedia! And I say, well yes, that's exactly what I am thinking of. But more than Wikipedia. Slay the dragon and bypass all the stupid restrictions and costs of the sites you know.
Let's go a step back. In Internet Theory, we classify four types of sites:
- Anonymous sites - sites that do not make use of the user's identity at all, like www.yahoo.com or www.google.com
- Registered sites - sites that allow for and possibly require a registration, but that do not store information other than user credentials and preferences from that site, like my.yahoo.com or igoogle.com
- Private sites - sites that store real world information about a person (instead of just a user), like mail.yahoo.com or gmail.com
- Restricted sites - private sites that are behind additional protections required by law, by industry standard, or by company policy, like credit card processing sites
One of the strange anomalies of the Internet is the kinds of sites that can attract large amounts of paying customers. There are porn sites, of course, but there are more free alternatives available now, so that market is drying out.
There are betting sites. They derive their attractiveness mostly from the fact they or their content are banned from several jurisdictions (most notably the United States or part of them). As a result, the demand for such sites can only be met by illegal or semi-legal operations, which as usual create a black market.
There are gaming sites. Here the attractiveness comes from the quality of the product alone, and from the fact that the games become dull after a while. I am talking about sites like World of Warcraft. But the amount of effort that goes into a good online game is way ahead of the pure programming, as places like OpenSim show.
Then there is the last one: personals sites. The application is simple, almost painfully so. From a programmer's perspective, it's a collection of profiles and some way to match them. On many sites, the way to match is simply by searching using a form that might as well have been created in 1994. I would call that the lowest-hanging fruit.
How would OpenServices.org work? There would be four arms:
- Software development - the creation, publication, and maintenance of all the software on the services
- Systems administration - the ownership of the servers and the installation of software updates
- Creative and project management - pushing through the next generation of features
- Quality - making sure that the new software works reliably
Now, imagine a site that works with a volunteer effort. Senior contributors get fancy titles and supervise (which also means, mentor) junior members. Coming out of, say, college, someone might already have several years of system administration experience, or creative design experience.
The only thing required for this to work is enough money for servers and bandwidth. Seems a win-win scenario for everybody. With the possible exception of match.com and eHarmony.
2010-03-02
Historica II: Google Sketchup
Leave it to Google to invent something incredibly useful. Sketchup is a site that Google devoted to 3D modeling of the earth. You go there to find models of buildings and whole cities, and you can contribute by creating your own models in 3D using the (Windows-only) Sketchup software.
The range of models available right now is fairly interesting, if not particularly large. There are models of things like Athens (and Vienna, Madrid, and a host of other cities), woodworking (furniture, etc.) and building pieces. It's all very good.
Google seems to have gotten into the business trying to get 3D models for its Google Earth application, which is the attempt at realistic rendering of the Earth in an application. It used to be the first application I used to get satellite imagery, until it was merged into Google Maps to give us the (infinitely useful) Satellite View.
Personally, I find the current state of the world a lot less interesting than the past. While Google seems to spend all this time and other people's energy to get a 3D model for use in one of its applications, the uses for Historica are much wider:
The range of models available right now is fairly interesting, if not particularly large. There are models of things like Athens (and Vienna, Madrid, and a host of other cities), woodworking (furniture, etc.) and building pieces. It's all very good.
Google seems to have gotten into the business trying to get 3D models for its Google Earth application, which is the attempt at realistic rendering of the Earth in an application. It used to be the first application I used to get satellite imagery, until it was merged into Google Maps to give us the (infinitely useful) Satellite View.
Personally, I find the current state of the world a lot less interesting than the past. While Google seems to spend all this time and other people's energy to get a 3D model for use in one of its applications, the uses for Historica are much wider:
- Virtual tours for tourists (or people that want to visit)
- Historical reconstruction (research)
- Research for writers
- Education/teaching
- Realistic game play
- Planning and urban development
Historica I: OpenSim
One of the projects I've been wanting to start on is a reconstruction of the world as it appeared at different points in time, a sort of historic view of what places like Chang'an, Athens, Rome, or even New York City looked like in the past.
The basic idea is that the information is all there, but it's invisible. We have it piecemeal - there is the outstandingly visual model of ancient Rome in the Museum of Roman Antiquity, there are models of ancient and modern cities, there are maps, there are views. But there is no consistent model.
Now, imagine you actually created a place like Wikipedia, where everybody interested can create a model of a city, of a building, and add the time coordinates when it existed. Then you'd be able to walk through an ancient city, or through New York in 1928, or watch San Francisco the way it looked like in 1989.
I had been working for months on this, trying to get different pieces of software to play together. The last thing I worked on was delta3d, a collection of open source projects that are made to play nice with each other. It has an interesting Python interface with which it's a real pleasure to play.
The real jump in interest, though, came when I found OpenSim. I am not sure how the project started, fact is it tries to be as compatible to SecondLife as can be. You can download the whole thing, compile it, and make it work. It's quite useless right now, since there aren't a lot of compelling applications, but it could do exactly what Historica needs.
Here is a link to the software:
http://opensimulator.org/
The basic idea is that the information is all there, but it's invisible. We have it piecemeal - there is the outstandingly visual model of ancient Rome in the Museum of Roman Antiquity, there are models of ancient and modern cities, there are maps, there are views. But there is no consistent model.
Now, imagine you actually created a place like Wikipedia, where everybody interested can create a model of a city, of a building, and add the time coordinates when it existed. Then you'd be able to walk through an ancient city, or through New York in 1928, or watch San Francisco the way it looked like in 1989.
I had been working for months on this, trying to get different pieces of software to play together. The last thing I worked on was delta3d, a collection of open source projects that are made to play nice with each other. It has an interesting Python interface with which it's a real pleasure to play.
The real jump in interest, though, came when I found OpenSim. I am not sure how the project started, fact is it tries to be as compatible to SecondLife as can be. You can download the whole thing, compile it, and make it work. It's quite useless right now, since there aren't a lot of compelling applications, but it could do exactly what Historica needs.
Here is a link to the software:
http://opensimulator.org/
2010-03-01
Cloud Computing Your Self
We are getting used more and more to applications in the cloud, and we are getting more and more comfortable storing even private data on servers over which we have absolutely no control. It's quite risky on one hand, but on the other it offers enormous convenience.
We use web sites for all sorts of services, and more often than not this is done with no significant risk to our identity. I have successfully changed my address at the DMV site, paid the registration to my car, gotten a copy of my birth certificate online (from Italy, no less), and filed taxes. It's really amazing.
One thing that I find strangely missing, though, is a virtualization of my self. Of course, I don't mean the flesh thing that is hacking at the keyboard right now. I mean my preferences and personal data.
I am a computer junkie. I have more computing devices at home right now than you'll find at the nearest Fry's, and each one of them has to be configured over and over again. Pretty much any computer I get is immediately repartitioned, Kubuntu installed on it, and then I start the laborious process of moving my user preferences on it.
I went through a ton of different options, and ended up linking all the configuration files I need into a directory. The directory then is under version control, and to create the new environment I check out that directory and run a linking script in it.
That's great, but there is no provision for changes in format. When I switched from Pidgin to Kopete, for instance, I lost all my log files. Same would be true switching from one music player (Amarok) to another (Rhythmbox). Then there are the different formats for address books, etc.
Strangely, though, the one issue that most people seem to agree upon is the need to synchronize bookmarks. While the companies and projects that sync calendars and address books do both a crappy job and are fairly rare, the projects that offer automatic syncing of bookmarks are plentiful and outstandingly stable.
It's been a while, now, that these projects have started including online storage. The first one I've used was XMarks, a Firefox extension that has been meanwhile ported to other browsers, as well. XMarks allows you to synchronize your bookmarks and (optionally) your browsing data (including stored passwords) to the XMarks server. The data is protected using a passphrase (and one hopes the XMarks folks do as they claim and encrypt the data on their servers and do not store the passphrase).
I recently switched from XMarks to Weave, the corresponding Mozilla project. Like all Mozilla projects, it's open source, which means (a) I can read the code and determine whether it does as I think it should, and (b) I can run my own copy of the server software if I don't like the idea of Mozilla having access to my personal data.
Now, I recently got a new computer - a "gaming" laptop that I use to run intensive compilation runs. Well, I installed Firefox, installed Weave, entered my user info and - voilá! - the new Firefox looked like the old one on this computer. Miracles.
Of course you think, now, that it's something that never happens to you. Well, you are wrong: your personal info is something you typically enter into every new phone that you buy. As a matter of fact, I know people that stick with old phones just because they don't want to go through the pain of moving their phone book over.
Some phones (like Windows Mobile or the iPhone) make the transition easier. The cheaper phones typically don't. But there is absolutely no reason for it, and it should go without saying that your entire virtual persona is available online for you to download whenever you get a new phone.
Why is that so important? Because if you have no barrier to switching phones, you will be more likely to buy a phone that suits your needs of the moment. You could rent a phone more easily if you knew your information is on it, you would switch phones more easily if there was no pain involved. You would stop looking at your phone as a treasure trove and investment and more as what it is: a communication device.
We use web sites for all sorts of services, and more often than not this is done with no significant risk to our identity. I have successfully changed my address at the DMV site, paid the registration to my car, gotten a copy of my birth certificate online (from Italy, no less), and filed taxes. It's really amazing.
One thing that I find strangely missing, though, is a virtualization of my self. Of course, I don't mean the flesh thing that is hacking at the keyboard right now. I mean my preferences and personal data.
I am a computer junkie. I have more computing devices at home right now than you'll find at the nearest Fry's, and each one of them has to be configured over and over again. Pretty much any computer I get is immediately repartitioned, Kubuntu installed on it, and then I start the laborious process of moving my user preferences on it.
I went through a ton of different options, and ended up linking all the configuration files I need into a directory. The directory then is under version control, and to create the new environment I check out that directory and run a linking script in it.
That's great, but there is no provision for changes in format. When I switched from Pidgin to Kopete, for instance, I lost all my log files. Same would be true switching from one music player (Amarok) to another (Rhythmbox). Then there are the different formats for address books, etc.
Strangely, though, the one issue that most people seem to agree upon is the need to synchronize bookmarks. While the companies and projects that sync calendars and address books do both a crappy job and are fairly rare, the projects that offer automatic syncing of bookmarks are plentiful and outstandingly stable.
It's been a while, now, that these projects have started including online storage. The first one I've used was XMarks, a Firefox extension that has been meanwhile ported to other browsers, as well. XMarks allows you to synchronize your bookmarks and (optionally) your browsing data (including stored passwords) to the XMarks server. The data is protected using a passphrase (and one hopes the XMarks folks do as they claim and encrypt the data on their servers and do not store the passphrase).
I recently switched from XMarks to Weave, the corresponding Mozilla project. Like all Mozilla projects, it's open source, which means (a) I can read the code and determine whether it does as I think it should, and (b) I can run my own copy of the server software if I don't like the idea of Mozilla having access to my personal data.
Now, I recently got a new computer - a "gaming" laptop that I use to run intensive compilation runs. Well, I installed Firefox, installed Weave, entered my user info and - voilá! - the new Firefox looked like the old one on this computer. Miracles.
Of course you think, now, that it's something that never happens to you. Well, you are wrong: your personal info is something you typically enter into every new phone that you buy. As a matter of fact, I know people that stick with old phones just because they don't want to go through the pain of moving their phone book over.
Some phones (like Windows Mobile or the iPhone) make the transition easier. The cheaper phones typically don't. But there is absolutely no reason for it, and it should go without saying that your entire virtual persona is available online for you to download whenever you get a new phone.
Why is that so important? Because if you have no barrier to switching phones, you will be more likely to buy a phone that suits your needs of the moment. You could rent a phone more easily if you knew your information is on it, you would switch phones more easily if there was no pain involved. You would stop looking at your phone as a treasure trove and investment and more as what it is: a communication device.
2010-02-22
Automounting - SSH Host or Loopback Devices
Modern Linux systems have an extensive set of automounters for all sorts of devices that are inserted into your computer. Last I checked, when I plugged in my helmet camera, Linux was merrily talking with it despite the fact it was a recent model (the miracles of UMS implementations).
I used to be really dependent on autofs for this kind of thing. Back in the day, if you wanted your floppy disk to magically appear, you had to mount it either explicitly (ick), implicitly (fstab), or magically. You'd type "ls /plug/floppy" and some magic would go out and fish the correct mount options.
The magic is still handy for all the cases when you have a file somewhere, but your computer doesn't know it. In my case, that's mostly when a file is on a remote server or on an image file.
To solve the problem, there is autofs. That's a tool that magically knows where to put things because you (or someone else for you) told it to.
The autofs paradigm has two major items:
Another, more immediately useful example, is the mounting of .iso files. When you download a Linux (or Debian, or whatever) CD/DVD, it comes as a single file you have to burn to a physical medium. Now, it would be great if you could just mount that file. Wait, you can! Not only can you do that, you can automagically do that.
So, this is what this article is all about. We will see how to use a remote server and an ISO file just like regular directories - and nothing is standing in the way of more and more interesting additions to our file systems.
[Setup]
autofs is well integrated intu Ubuntu: just type the trusted
So, what happened? Absolutely nothing. By default, the autofs installation does nothing at all. But it has a few tricks in store.
All the action you care about happens in the /etc directory. There, you'll find a messy set of files called auto.[something]. In the current package, there are the following:
Of the examples, auto.misc is the simpler one, so we'll look at it first. It is composed of lines made up of three sections, just like auto.master. The sections are different in order (don't ask), but they are logically similar:
Here you see that autofs was born to mount network shares, since anything that isn't a straight NFS share (you don't care what that is) requires special handling.
The other file, auto.smb, is a lot more complicated. First of all, it doesn't have any lines like auto.misc. Instead, it is a script you can run on the command line. What it does (if you feel inclined to read the source, follow along) is to use the command "smbclient" to query the network and find shares to mount. Once it found one that matches your desired key, it "mounts" it.
It actually doesn't do the mounting itself - it simply returns a line like the ones in auto.misc (minus the key, since that's a given), thus telling the automounter what to do.
[SSH Mount]
So, now that we looked at the general setup, let's look more closely at mounting SSH servers. From now on, I'll use the convention that the user dude on the machine kde is trying to access a server named marco.example.org using the account guy.
The autofs setup is really simple. We need a file, say auto.ssh, that contains the automounter lines:
Ok, there is such a thing as a filesystem on ssh. We need to install that if we want to access the functionality. That's done easily:
sudo apt-get install sshfs
Now we have the fstype installed, and the weird syntax for the location is explained. The backslash ('\') before the pound sign took me hours to figure out, so I am expecting friendly comments and PayPal donation for the time I saved you.
What about the options? fstype is fuse, which is short for Filesystem in User land (you don't need to know right now - but it's a wonderful project). Reconnect and allow_others are for ease of use. The first one reconnects if the network goes down, the second one allows others that have access to the mount point to access your ssh share (you may want to rethink it on multi-user systems - but if you want ot make sure that your mount is accessible by daemons behind your software, you better set it).
The user and group ID are those of the connecting user, dude. You can find both very easily by typing id on the command line (they are the first two numbers on the result). The first user and group created on a Kubuntu machine have the ids 1000, so that's fairly common.
[Problems]
Yeah. It won't work like that. You have to do some extra work for the setup. (Again, donations appreciated...). First, we need to be able to connect to the server using SSH:
The next problem is a very immediate security problem. When automount is running, it is using the root account. So it's the root account that need to be able to access the remote server. But the root account doesn't have the credentials, dude has them. What to do?
Well, the easiest thing to do is to give the credentials to root. I mean, if you are root, you have access to the credentials, anyway, so it's quite pointless to hide them. Easiest way to do that is by creating a link between dude's credentials and root's. On a single-user system you could link the entire SSH directory:
Next thing is that autofs can't ask for a password or passphrase directly. We could either use an SSH agent (which I won't cover because it's complicated) or we can use a private key without a passphrase (which is ugly because it's totally insecure).
Now, I can't stress this enough: using a private key without a passphrase is seriously dangerous: it allows anyone with access to the machine with the private key to access all the machines that allow access with it. This setup is mostly for people that, like me, prefer SSH over SMB at home and whose main laptop contains all the valuable information. The idea here is that if someone compromises my laptop, the worst thing already happened. That they can access the backup server from there, not a big deal.
Ok, now that you have given root a passphrase-less key and made it a default key, or loaded an ssh-agent with which the automounter communicates, we are ready to go. Well, we have first to tell the automounter where to put the SSH servers. For that purpose, we add a single line to the file auto.master:
[Loop]
If you are a real geek, you probably have a ton of files that can be mounted as drives. You might have the .iso files you burnt to install Linux, you might have the virtual hard drive of a virtual machine (like VirtualBox or VMWare, or User-Mode-Linux for the courageous).
Mounting those drives is easy, but a real nuisance. Essentially, you type:
[The Solution]
What about if you get rid of the haphazard and use a script instead? Why a script, you ask. Well, the problem is that while you may know where the file is, unless you are particularly orderly, your computer won't know where to look. So, instead, we use the utility locate to find the file we need and then mount it to a well-known point.
Assume you want to mount the kubuntu ISO file. First, use locate to find it:
locate kubuntu
You get back a list of files that all match the name kubuntu. If you add the option -b, then you get only files who actual file name matches kubuntu (no files contained in directories named kubuntu. Still, we might have a ton of files.
What do we do? We look them all up and determine whether we can mount them. To do that, we use the tool file, which gives us a guess as to the content of the file. If it is a line that contains the words filesystem data, we know we can mount it and just need to pass the correct filesystem type to mount.
What we will do is prioritize the file names by length first. The shorter file name that matches our string is a better match. Then we look at numbers and select the one with the lower number, if the length is the same. Actually, you can decide to resolve conflicts whichever way you want. You can also decide not to resolve a conflict, and to return an error, in which case the mount fails (I don't like that behavior, even though it might be better, because I am doing this just out of convenience, after all!)
I wrote a Tcl script to do the job for me. I also loaded all the ISO images of CD-Rs I own onto my trusted server, backup, which I access (you guessed it) using SSH. Autofs is configured for loop on backup, and for ssh on this computer. So when I need to find a particular picture from one of the CDs I burnt, I simply type:
[More Fun]
Autofs is best friends with fuse, the file system in user space. Since fuse development is tons easier and less dangerous than development of file systems for the kernel, there are tons of available options.
If you don't believe me, just type in apt-cache search fuse into a terminal window, and look how many of the hits you can get for a file system. There are exciting things like flickrfs (which mounts your flickr photos), unionfs (which joins two directories together, for instance if you have no room left in one...), glusterfs (for clustered servers), and for everybody annoyed enough with UpPeRcase on UNIX, ciopfs (case insensitive on purpose).
[Update: if you want the Tcl script or a Python version of the same, please let me know.]
I used to be really dependent on autofs for this kind of thing. Back in the day, if you wanted your floppy disk to magically appear, you had to mount it either explicitly (ick), implicitly (fstab), or magically. You'd type "ls /plug/floppy" and some magic would go out and fish the correct mount options.
The magic is still handy for all the cases when you have a file somewhere, but your computer doesn't know it. In my case, that's mostly when a file is on a remote server or on an image file.
To solve the problem, there is autofs. That's a tool that magically knows where to put things because you (or someone else for you) told it to.
The autofs paradigm has two major items:
- mount points - which are directories that autofs watches carefully for access
- maps - which are the magic things that tell autofs how to map things
scp user@marco.example.org:/home/user/example.txt .Wouldn't it be much better for a hundred different reasons if you could just copy the file, say like this:
cp /net/marco.example.org/home/user/example.txt .Why is that better? Well, so far it isn't much. But if you could access the file like that, you could convince your editor to edit the file in place, saving it right where it is, without having to copy it locally. You could, for instance, just add the file path of your build server, and suddenly you run software remotely.
Another, more immediately useful example, is the mounting of .iso files. When you download a Linux (or Debian, or whatever) CD/DVD, it comes as a single file you have to burn to a physical medium. Now, it would be great if you could just mount that file. Wait, you can! Not only can you do that, you can automagically do that.
So, this is what this article is all about. We will see how to use a remote server and an ISO file just like regular directories - and nothing is standing in the way of more and more interesting additions to our file systems.
[Setup]
autofs is well integrated intu Ubuntu: just type the trusted
sudo apt-get install autofsand you are on your way. The dependencies are downloaded, the files installed, the autofs daemon started, and you are ready to go.
So, what happened? Absolutely nothing. By default, the autofs installation does nothing at all. But it has a few tricks in store.
All the action you care about happens in the /etc directory. There, you'll find a messy set of files called auto.[something]. In the current package, there are the following:
- auto.master - the main file
- auto.misc - an example with different hard-coded devices
- auto.smb - another example for SMB (Windows) shared drives
/some/path /etc/auto.something --options/some/path is a random directory you choose that henceforth is where autofs is going to put your shares. Whenever you try to access a subdirectory of /some/path that doesn't already exist, autofs will try to mount it looking up the file /etc/auto.something for a hint at what to do. The --options are, well, optional and help determining the behavior of the automounter. --timeout, for instance, is very common: it tells the automounter to drop a directory if it hasn't been used in a set amount of time.
Of the examples, auto.misc is the simpler one, so we'll look at it first. It is composed of lines made up of three sections, just like auto.master. The sections are different in order (don't ask), but they are logically similar:
key options locationThe "key" argument here is not a path but a name - it's the name of the subdirectory you are trying to access. Say you went with the default in auto.master and assigned the mount /misc to auto.misc, and there is a key "floppy" in there, then whenever you try to access the directory /misc/floppy, the automounter would look at the line starting with "floppy" to decide what to mount (the location) and how (the options).
Here you see that autofs was born to mount network shares, since anything that isn't a straight NFS share (you don't care what that is) requires special handling.
The other file, auto.smb, is a lot more complicated. First of all, it doesn't have any lines like auto.misc. Instead, it is a script you can run on the command line. What it does (if you feel inclined to read the source, follow along) is to use the command "smbclient" to query the network and find shares to mount. Once it found one that matches your desired key, it "mounts" it.
It actually doesn't do the mounting itself - it simply returns a line like the ones in auto.misc (minus the key, since that's a given), thus telling the automounter what to do.
[SSH Mount]
So, now that we looked at the general setup, let's look more closely at mounting SSH servers. From now on, I'll use the convention that the user dude on the machine kde is trying to access a server named marco.example.org using the account guy.
The autofs setup is really simple. We need a file, say auto.ssh, that contains the automounter lines:
marco.example.org -fstype=fuse,allow_other,reconnect,uid=1000,gid=1000 sshfs\#guy@marco.example.org:/Wait, wait, wait!!! What does that all mean? First, you have the key, which we already knew about. Then we have options, which we need to explain and chance. Finally, we have the weirdest thing as location.
Ok, there is such a thing as a filesystem on ssh. We need to install that if we want to access the functionality. That's done easily:
sudo apt-get install sshfs
Now we have the fstype installed, and the weird syntax for the location is explained. The backslash ('\') before the pound sign took me hours to figure out, so I am expecting friendly comments and PayPal donation for the time I saved you.
What about the options? fstype is fuse, which is short for Filesystem in User land (you don't need to know right now - but it's a wonderful project). Reconnect and allow_others are for ease of use. The first one reconnects if the network goes down, the second one allows others that have access to the mount point to access your ssh share (you may want to rethink it on multi-user systems - but if you want ot make sure that your mount is accessible by daemons behind your software, you better set it).
The user and group ID are those of the connecting user, dude. You can find both very easily by typing id on the command line (they are the first two numbers on the result). The first user and group created on a Kubuntu machine have the ids 1000, so that's fairly common.
[Problems]
Yeah. It won't work like that. You have to do some extra work for the setup. (Again, donations appreciated...). First, we need to be able to connect to the server using SSH:
ssh guy@marco.example.orgFirst time we do that, we are asked whether we'd like to accept the key. That's very important (insert security blabla), but in our case, it's mostly a nuisance. You have to connect manually the first time, because otherwise autofs is confused by the reply. So do that.
The next problem is a very immediate security problem. When automount is running, it is using the root account. So it's the root account that need to be able to access the remote server. But the root account doesn't have the credentials, dude has them. What to do?
Well, the easiest thing to do is to give the credentials to root. I mean, if you are root, you have access to the credentials, anyway, so it's quite pointless to hide them. Easiest way to do that is by creating a link between dude's credentials and root's. On a single-user system you could link the entire SSH directory:
sudo ln -s ~dude/.ssh ~root(Assuming root doesn't have a .ssh directory yet.)
Next thing is that autofs can't ask for a password or passphrase directly. We could either use an SSH agent (which I won't cover because it's complicated) or we can use a private key without a passphrase (which is ugly because it's totally insecure).
Now, I can't stress this enough: using a private key without a passphrase is seriously dangerous: it allows anyone with access to the machine with the private key to access all the machines that allow access with it. This setup is mostly for people that, like me, prefer SSH over SMB at home and whose main laptop contains all the valuable information. The idea here is that if someone compromises my laptop, the worst thing already happened. That they can access the backup server from there, not a big deal.
Ok, now that you have given root a passphrase-less key and made it a default key, or loaded an ssh-agent with which the automounter communicates, we are ready to go. Well, we have first to tell the automounter where to put the SSH servers. For that purpose, we add a single line to the file auto.master:
/share/ssh auto.sshThat means that from now on, our servers are going to be mounted under /share/ssh (make sure the directory exists, is owned by root, and has the permissions 755). Restart the autofs daemon with sudo /etc/init.d/autofs restart, and there you go! Now try:
ls /share/ssh/marco.example.organd you will get the directory listing for the root directory of that server. You will be able to see, modify, and create files exactly like the user guy on that server could.
[Loop]
If you are a real geek, you probably have a ton of files that can be mounted as drives. You might have the .iso files you burnt to install Linux, you might have the virtual hard drive of a virtual machine (like VirtualBox or VMWare, or User-Mode-Linux for the courageous).
Mounting those drives is easy, but a real nuisance. Essentially, you type:
mount -o loop filename mountpointand then you can access the file as if you were in the mount point. Say, you have the Karmic CD stored as /home/dude/Downloads/kubuntu-9.10-desktop-i386.iso. To put it onto the directory /share/iso/kubuntu-9.10 you would type:
mount -o loop /home/dude/Downloads/kubuntu-9.10-desktop-i386.iso /share/iso/kubuntu-9.10Of course, the directory usually doesn't exist, so you just mount it as /mnt, and you manage until you need a different version and unmount the first one. It gets messy after a short while.
[The Solution]
What about if you get rid of the haphazard and use a script instead? Why a script, you ask. Well, the problem is that while you may know where the file is, unless you are particularly orderly, your computer won't know where to look. So, instead, we use the utility locate to find the file we need and then mount it to a well-known point.
Assume you want to mount the kubuntu ISO file. First, use locate to find it:
locate kubuntu
You get back a list of files that all match the name kubuntu. If you add the option -b, then you get only files who actual file name matches kubuntu (no files contained in directories named kubuntu. Still, we might have a ton of files.
What do we do? We look them all up and determine whether we can mount them. To do that, we use the tool file, which gives us a guess as to the content of the file. If it is a line that contains the words filesystem data, we know we can mount it and just need to pass the correct filesystem type to mount.
What we will do is prioritize the file names by length first. The shorter file name that matches our string is a better match. Then we look at numbers and select the one with the lower number, if the length is the same. Actually, you can decide to resolve conflicts whichever way you want. You can also decide not to resolve a conflict, and to return an error, in which case the mount fails (I don't like that behavior, even though it might be better, because I am doing this just out of convenience, after all!)
I wrote a Tcl script to do the job for me. I also loaded all the ISO images of CD-Rs I own onto my trusted server, backup, which I access (you guessed it) using SSH. Autofs is configured for loop on backup, and for ssh on this computer. So when I need to find a particular picture from one of the CDs I burnt, I simply type:
and off I go. I don't need to get to the server backup and do the mount manually, and I don't have to care what the name of the ISO file is and where I stored it (in the backup tree or in the live image tree). I don't have to invoke ssh, don't have to worry about scp, and don't have to figure out what to cache on my machine before I fire up my image manipulation program.gwenview /share/ssh/backup/loop/pics-2003
[More Fun]
Autofs is best friends with fuse, the file system in user space. Since fuse development is tons easier and less dangerous than development of file systems for the kernel, there are tons of available options.
If you don't believe me, just type in apt-cache search fuse into a terminal window, and look how many of the hits you can get for a file system. There are exciting things like flickrfs (which mounts your flickr photos), unionfs (which joins two directories together, for instance if you have no room left in one...), glusterfs (for clustered servers), and for everybody annoyed enough with UpPeRcase on UNIX, ciopfs (case insensitive on purpose).
[Update: if you want the Tcl script or a Python version of the same, please let me know.]
Subscribe to:
Posts (Atom)
