I’ve been milling over this post about big data in the IT world for quite a while. It basically says that given large (and growing) data sets, companies that didn’t previously need data researchers suddenly need people to help them use “big data.” Everyone company is a data company. In effect we have an ironic counter example to the effect of automation on the need for labor.

These would be “data managers” have their work cut out for them. Data has value, sure, but unlike software which has value as long as someone knows how to use it,¹ poorly utilized data is just as good as no data. Data researchers need to be able to talk to developers and help figure out what data is worth collecting and what data isn’t. Organizations need someone to determine what data has real value, if only to solve a storage-related problem. Perhaps more importantly data managers would need to guide data usage both technically (in terms of algorithms, and software design) and in terms of being able to understand the abilities and shortfalls of data sets.

There’s a long history of IT specialist positions: database developers, systems administrators, quality assurance engineers, release engineering, and software testing. Typically we divide this between developers and operations folks, but even the development/operations division is something of a misnomer. There are merits to generalism and specialization, but as projects grow, specialization makes sense and data may just be another specialty in a long tradition of software development and IT organization.

Speicailization also makes a lot of sense in the context of data, where having a lot of unusable data adds no value and can potentially subtract value from an organization.

A Step Back

There are two very fundamental points that I’ve left undefined: what “data” am I talking about and what kinds of skills differentiate “data specialists” from other kinds of technicians.

What are big data?

Big data sets are, to my mind, large collections of data, GIS/map based information, “crowd sourced” information, and data that is automatically collected through the course of normal internet activity. Big data is enabled by increasingly powerful databases and the ubiquity of the computing power, which lets developers process data on large scales. For examples: the aggregate data from foursquare and other similar services, comprehensive records of user activity within websites and applications, service monitoring data and records, audit trails of activity on shared file systems, transaction data from credit cards and customers, tracking data from marketing campaigns.

With so much activity online, it’s easier for software developers and users (which is basically everyone, directly or otherwise) to create and collect a really large collection of data regarding otherwise trivial events. Mobile devices and linkable accounts (OpenID, and other single sign-on systems) simplify this process. The thought and hope is all this data equals value and in many circumstances it does. Sometimes, it probably just makes things more complicated.

Data Specialists

Obviously every programmer is a kind of “data specialist” and the last seven or eight years of the Internet has done everything to make every programmer a data specialist. What the Internet hasn’t done is give programers a sense of basic human factors knowledge, or a background in fundamental quantitative psychology and sociology. Software development groups need people who know what kinds of questions data can and cannot answer regardless of what kind or how much data is present.

Data managers, thus would be one of those posistions that sits between/with technical staff and business staff, and perhaps I’m partial to work in this kind of space, because this is very much my Chance. But there’s a lot of work in bridging this divide, and a great deal of value to be realized in this space. And it’s not like there’s a shortage of really bright people who know a lot about data and social science who would be a great asset to pretty much any development team.

Big Data Beyond Software Development

The part of this post that I’ve been struggling over for a long time is the mirror of what I’ve been talking about thus far. In short, do recent advancements in data processing and storage (NoSQL, Map Reduce, etc.) that have primarily transpired amonst startups, technology incubators, and other “Industry” sources have the potential to help acdemic research? Are there examples of academics using data collected from the usage habits of websites to draw conclusions about media interaction, reading habits, cultural particpation/formation? If nothing else are sociologists keeping up with “new/big data” developents? And perhaps most importantly, does the prospect of being able to access and process large and expansive datasets have any affect on the way social scientists work? Hopefully someone who knows more about this than I do will offer answers!

The Debate

Computer programmers want data to be as structured as possible. If you don’t give users a lot of room to do unpredictable things, it’s easier to write software that does cool things. Users on the other hand, want (or think that they want) total control over data and the ability to do whatever they want.

The problem is they don’t. Most digital collateral, even the content stored in unstructured formats, is pretty structured. While people may want freedom, they don’t use it, and in many cases users go through a lot of effort to recreate structure within unstructured forms.

Definitions

Structured data are data that is stored and represented in a tabular form or as some sort of hierarchical tree that is easily parsed by computers. By contrast, unstructured data, are things like files that have data and where all of the content is organized manually in the file and written to durable storage manually.

The astute among you will recognize that there’s an intermediate category, where largely unstructured data is stored in a database. This happens a lot in content management systems, in mobile device applications, and in a lot of note taking and project management applications. There’s also a parallel semi-structured form, where people organize their writing, notes, content in a regular and structured manner even though the tools they’re using don’t require it. They’d probably argue that this was “best practice,” rather than “semi-structured” data, but it probably counts.

The Impact

The less structured content or data is the less computer programs are able to do with the data, and the more people have to work to make the data useful for them. So while we as users want freedom, that freedom doesn’t get us very far and we don’t really use it even when we have it. Relatedly, I think we could read the crux of the technological shift in Web 2.0 as a move toward more structured forms, and the “mash up” as the celebration of a new “structured data.”

The lines around “semi-structured” data are fuzzy. The trick is probably to figure out how to give people just enough freedom so that they don’t feel encumbered by the requirements of the form, but so much freedom that the software developers are unable to do really smart things behind the scene. That’s going to be difficult to figure out how to implement, and I think the general theme of this progress is “people can handle and developers should err on the side of stricture.”

Current Directions

Software like org-mode and twiki are attempts to leverage structure within unstructured forms, and although the buzz around enterprise content management (ECM) has started to die down, there is a huge collection of software that attempts to impose some sort of order on the chaos of unstructured documents and information. ECM falls short probably because it’s not structured enough: it mandates a small amount of structure (categories, some meta-data, perhaps validation and workflow,) which doesn’t provide significant benefit relative to the amount of time it takes to add content to these repositories.

There will be more applications that bridge the structure boundary, and begin to allow users to work with more structured data in a productive sort of way.

On a potentially orthogonal note, I’m working on cooking up a proposal for a LaTeX-based build system for non-technical document production that might demonstrate--at least hypothetically--how much structure can help people do awesome things with technology. I’m calling it “A LaTeX Build System.”

I’d love to hear what you think, either about this “structure question,” or about the LaTeX build system!

Most of the sites I visit these days are: Wikipedia, Facebook, sites written by people I’ve known online since the late 1990s, people who I met online around 2004, and a few sites that I’ve learned about through real life connections, open source, and science fiction writing. That’s about it, it sounds like a lot, and it is, but the collection is pretty static.

As I was writing about my nascent list of technical writing links, I realized that while I’ve been harping on the idea of manually curated links and digital resources for for a single archives for a couple of years now, I’ve not really thought about the use or merits of manually curated links to the internet writ large.

After all you can find anything you need with Google. Right?

I mostly assumed that if I could get people to curate their own content, “browsing” would become more effective, and maybe Google technology could adapt to the evolving social practice?

Though the inner workings of Google are opaque, we know that Google understands the Web by following and indexing the links that we create between pages. If we don’t link, Google doesn’t learn. Worse, if we let software create all the links between pages, then Google starts to break.

Put another way: the real intelligence of Google’s index isn’t the speed and optimization of a huge amount of data--that’s a cumbersome engineering problem--but rather our intelligence derived from the links we make. As our linking patterns change, as all roads begin to lead back to Wikipedia, as everyone tries to “game” Google (at least a little) the pages that inevitably float to the top are pages that are built to be indexed the best.

And Google becomes a self-fulfilling prophecy because whatever page we do find out about and create links to are the links that we’ve found using Google. We’ve spent a lot time thinking about what happens if google becomes evil, to think about what happens to us as Google stops providing new and useful information. We’ve spent considerably less work considering what happens when Google becomes useless.

Security, of technological resources, isn’t a technological problem. The security of technological resources and information is a problem with people.

There.

That’s not a very ground breaking conclusion, but I think that the effects of what this might mean for people doing security¹ may be more startling.

Beyond a basic standard of “writing and using quality software” and following sane administration practices, the way to resolve security issues is to fix the way people use and understand the implications of their use.

There are tools that help control user behavior to greater or lesser degrees. Things like permissions control, management, auditing, and encryption, but they’re just tools: they don’t solve the human problems and the policy/practice issues that are the core of best security practice. Teaching people how their technology works, what’s possible and what’s not possible, and finally how to control their own data and resources is the key to increasing and providing security services to everyone.

I think of this as the “free software solution,” because it draws on the strengths and methods of free software to shape and enhance people’s user experience and to improve the possible security of the public network as a whole. One of the things that has always drawn me to free software, and one of its least understood properties, deals with the power of source code to create an environment that facilitates education and inquiry. People who regularly use free software, I’d bet, have a better understanding of how technology works than people who don’t, and it’s not because free software users have to deal with less polished software (not terribly true), but has something to do with a different relationship between creators and users of software. I think it would be interesting to take this model and apply it to the “security problem.”

With luck, teaching more people to think about security processes will mean that users will generally understand:

• how encryption works, and be more amenable to managing their own cryptography identities and infrastructure. (PGP and SSH)
• how networking works on a basic level to be able to configure, set, and review network security. (LAN Administration, NetFilter)
• how passwords are stored and used, and what makes strong passwords that are easy to remember and difficult to break.
• how to control and consolidate identity systems to minimize social engineering vulnerabilities. (OpenID, OAuth, etc.)

There’s a lot of pretty basic knowledge that I think most people don’t have. At the same time, I think it’s safe to say that most of the difficult engineering questions have been solved regarding security, there’s a bunch of tooling and infrastructure on the part of various services that would make better security practices easier to maintain (i.e. better PGP support in mail clients). In the mean time….

Stay smart.

“Do eye glasses make us all cyborgs?” Someone asked me a few days ago.

I was annoyed more than anything.

Of course they do. Corrective lenses are a non-biological technology that shape our experience of the world and of our bodies. By this logic, pretty much every tool developed as a product of “technology” (applied science; otherwise known as tinkering with stuff,) renders us cyborgs.

I like the notion that cyborgism is the rule and not the exception in the course of human history, but it makes the conversation about the cyborg moment more banal. A more banal cyborg moment makes it harder to think about the parts that I think are most interesting: the internet, distributed collaboration, free software and open source, and the impact of technology on literature and reading/writing.

As a retort, I said something like, “Perhaps, but if you accept that eye glasses create cyborg beings, then you’d have to accept that shoes also create cyborgs. And the effects of shoes are much more interesting.”

Shoes affect how far people can walk, the speed of independent locomotion, they prevent all sorts of awful injuries, and probably lengthen the lifespan as a result. Shoes probably also change our feet and make us dependent upon wearing shoes, and more prone to certain kinds of injuries when barefoot. Fascinating stuff.

All other things being equal, I’m going to stick to internet and the cyborgs resulting from the encounter of humans and that technology.

By eliminating the artificial scarcity of software, open source software forces businesses and technology developers to think differently about their business models. There are a few ways that people have traditionally built businesses around open free and open source software. There are pros and cons to every business model, but to review the basic ideas are:

• Using open source software as a core and building a thin layer of proprietary technology on top of the open source core. Sometimes this works well enough (e.g. SugarCRM, OS X,) and sometimes this doesn’t seem to work as well (e.g. MySQL, etc.)
• Selling services around open source software. This includes support contracts, training services, and infrastructure provisioning. Enterprises and other organizations and projects need expertise to make technology work, and the fact that open source doesn’t bundle licensing fees with support contracts doesn’t make the support (and other services) less useful or needed for open source.
• Custom development services. Often open source projects provide a pretty framework for a technology, but require some level of customization to fit the needs and requirements of the “business case.” The work can be a bit uneven, as with all consulting, but the need a service are both quit real. While the custom code may end up back in the upstream, sometimes this doesn’t quite happen for a number of reasons. Custom development obviously overlaps with service and thin-proprietarization, but is distinct: it’s not a it doesn’t revolve around selling proprietary software, and it doesn’t involve user support or systems administration. These distinctions can get blurry in some cases.

In truth, when you consider how proprietary software actually convey value, it’s really the same basic idea as the three models above. There’s just this minor mystification around software licenses, but other than that, the business of selling software and services around software doesn’t vary that much.

James Governor of Red Monk suggests a fourth option: Packaging technology.

The packaging model is likely just an extension of the “services” model, but it draws attention to the ways that companies can create real value not just by providing services and not just by providing a layer of customization, but by spending time attending to the whole experience, rather than the base technology. It also draws some attention to the notion that reputation matters.

I suppose it makes sense: when businesses (and end users) pay for proprietary software, while the exchange is nominally “money” for “license” usage rights, in reality there are services and other sources of value. Thus it is incumbent upon open source developers and users to find all of the real sources of value that can be conveyed in the exchange of money for software, and find ways to support themselves and the software. How hard can it be?

I use a mail client called mutt. The quality of this software may largely explain my opinion this post on the continued relevance of email

I think mutt warrants a bit of extra attention for two reasons. First, because I think there are enough people out there who don’t use mutt who could and perhaps should, and I’d like to do a little encouraging; and second,like all fundamentally wonderful pieces of software, mutt can teach us something important about what makes technology great and pleasurable to use.

Working with any new kind of software is always a challenge. It is unfortunate that “features” and “functions” are the currency by which we judge software. Which is unfair to both the technology and ourselves, as the utility and quality of these features/functions depends on a number of subjective/individual factors. That said, with regards to mutt, my list is as follows:

• Mutt is agnostic on the editor question. I suspect the fact that I could use any text editor I wanted to write email was probably my original reason for switching to mutt in the first place. It’s amazing what a sane editing experience can do for the overall experience of writing emails.
• Support for PGP/GPG encryption. Signing and encrypting emails with PGP is probably only a minor advantage, and of limited actual utility, but I think it’s important and valuable to have this capability in your email client. After all, the success of PGP depends on a crowd effect: if it’s easy, sign all your email and hope that others will join you. Mutt makes this easy, which is a good thing indeed.
• Mutt operates independently of mail transmission protocols, which are universally flawed. In many ways, by not including support for mail transmission, mutt is more useful and more flexible than it would be if it was designed to handle mail transmission. Having said that, recent versions of mutt have internal support for IMAP/POP/SMUT. Not that I’d use it or recommend that you do use it and I suspect most mutt users don’t either.
• Mutt operates independently of mail storage format: you can maintain complete control over your mail data, and store email pretty much however you like. While this may be a burden to some, I’m somewhat controlling when it comes to data storage and preservation, and I think email archives are incredibly important. And I’m a weirdo about email storage.
• Mutt’s “sidebar patch” isn’t even a part of the core of the software, but it’s absolutely crucial to my experience of the software. Basically it gives you a heads-up-display of your mailboxes and tells you at a glance: if there are new messages and how many messages (new, flagged, read) are in an mailbox. While it eats into some screen real estate, it’s generally unused screen space and it’s more than worth the expenditure of pixels.
• Mutt runs on console and can be compiled on pretty much any contemporary UNIX-like system. Chances are there are packages for most operating system. So I feel pretty confident that I’ll pretty much be able to use mutt no matter what kind of system I end up using. Also console apps generally run pretty well in screen, which makes them accessible (and persistent) across the internet.

Onward and Upward!

This is a post about mobile technology in an unconventional sense. I think I’m probably an extreme “mobile” technology user: I ride a lot commuter rail and use my laptop extensively on the train. Then, I work on a laptop all day. In the evening, I often do at least a little additional work, again on the same laptop. There is, after all, always writing (like this post!) to fill any remaining free time.

I’m not a terribly typical mobile user. My main “mobile device” is a little ThinkPad (and sometimes a larger ThinkPad,) running Linux and a lot of Lisp (emacs and otherwise.) It’s not ideal for every situation: there are times when I just can’t bare to open the laptop again or it’s unfeasible (and there’s always the Kindle for times like those.) Most of the time it works well.

It’s hard to omit discussion of the “tablet” and the iPad. For me, the fact that tablets are not general purpose computers is a huge deterrent. This is probably not the case for everyone, though there are lots of shades to this debate. I think the more interesting question is not “do people need general purpose tablets?” and more “how will more ubiquitous embeded-type systems effect the way people will approach ‘general purpose’ computing environments” from here on out? Honestly, this in computing practice has already happened, but I think it will continue to pose important questions for users and developers as it continues.

The struggle, for me, revolves less around the question “how do I work remotely?” and more around “how do I also work when I’m at a desk?” The adjustment can be hard: For a while, I was so used to working on the train, and in random chairs, that I had a hard time focusing if the computer wasn’t actually on my lap. Bad ergonomics is only the start of this.

The current solution is to set up desks and workstations that use the same laptops and systems so that I’m not perpetually switching between fixed computers and mobile computers. I’m also keen for these desks to have their own appeal: bigger monitors, nice keyboards, and easy to attach power cords. I’ve also attempted to tie together all of the “I’d like to switch between laptop-mode and desk-mode,” functions (e.g. network connection, monitor attachment, window layout) into easy to trigger operations, so I can get started more quickly. Nice. Seamless. Efficient.

The lessons: There are many ways to maintain technical (cyborg) coherence despite/during geographical movement and sometimes that technology isn’t particularly cutting edge. Sometimes the best way to break yourself of a habit you don’t like is to play a game with yourself where you establish a more attractive option. Finally, a very small change or automation can be enough to take something difficult and make it much easier or something unpleasantly and make it workable.

I hate computer mice. A lot.

The closest I’ve gotten to liking a pointing device is an acquiescence to the TrackPoint on the laptops I use. That’s the little red dot in the middle of the ThinkPad keyboards. My problem with computer mice is the context switch between “typing-mode” and “mousing-mode.” Moving between the modes is jarring and inefficient. I’ve been using StumpWM and other similar window managers for years now and as my need for the mouse decreases my irritation with needing to use a mouse increases.

I’ve been struggling for a few months with a bit of a problem: Several months ago I got a new bigger ThinkPad, a T510, while it has my beloved TrackPoint there is also a TouchPad. After years of only using laptops with the red dot, this was very disconcerting. How did I keep from triggering the touchpad with my wrists? Couldn’t I just turn the damn thing off?

I did, and everyone who tried to use the computer after that was dismayed, and I didn’t care. Except, I found out that, apparently, disabling the touchpad also disables all non-TrackPoint pointers. So when I plugged the laptop into the docking station, the external mouse didn’t work.

Blast.

The solution to disabling and enabling the mouse on the fly, that follows isn’t as pretty as I’d like, but it works.

UPDATE: Turns out that my original procedure only appears to work. I’ve made the following modification to the toggle-mouse script, using a stock xorg.conf file.

File: /usr/bin/toggle-mouse

#!/bin/sh

TOUCHPAD=`xinput list | egrep “TouchPad” | sed -r ’s/.id=([0-9]).*$/1/’`

if [ [xinput list-props “$TOUCHPAD” | egrep -o “[0-9]$” | head -n1]{.title-ref} -eq 0 ]; then xinput set-prop “$TOUCHPAD” “Device Enabled” 1 elif [ [xinput list-props “$TOUCHPAD” | egrep -o “[0-9]$” | head -n1]{.title-ref} -eq 1 ]; then xinput set-prop “$TOUCHPAD” “Device Enabled” 0 else xmpp-notify “Your mouse is probably screwed up somehow” fi

Test the output of “xinput list | egrep "TouchPad" | sed -r 's/.*id=([0-9]*).*$/\1/'”, and inspect “xinput list” to make sure that the value of $TOUCHPAD is the xorg id of the touchpad (or other device) that you want to disable.

I’d actually recommend not putting this in /usr/bin/, but just so long as it’s in the path. Then run toggle-mouse at the command line. You may need to run this as root-suid, for your system system is configured. Tweak the TOUCHPAD variable as needed.

If you have a better solution, I would be terribly interested in hearing about it.

I’m always interested in how the lessons that people learn in IT trickle down to other kinds of work and problems. This is one of the reasons that I¹ am so interested in what developers are interested in: if you want to know what’s happening in the technology space, it’s best to start at the top of the food chain. For this reason this article from IBM, which addresses the use of IT/Data Center management tools outside of the data center was incredibly interesting for me.

When you think about it, it makes sense. IT involves a lot of physical assets, even more virtual assets, and when projects and systems grow big enough, it can be easy to lose track of what you have, much less what state it’s in at any given time. Generalized, this is a prevalent issue in many kinds of complex systems.

As an aside, I’m a little interested when software that provides asset management and monitoring features, will scale down to the personal level. That’ll be interesting too. There are the beginnings of this kind of thing (e.g. iTunes, and git-annex) but only the beginnings.

I’m left with the following questions:

• Obviously moving from managing and monitoring networked devices to managing and monitoring infrastructure objects like water filtration systems, storm water drainage, the electrical grid, snow removal, etc. presents a serious challenge for the developers of these tools, and this adaptation will likely improve the tools. I’m more interested in how cities improve in this equation. And not simply with regards to operating efficiencies. What do we learn from all this hard data on cities?
• Will cities actually be able to become more efficient, or will they need to expand to include another layer of management management, that nullifies the advances. There are also concerns about additional efficiency increasing the “carrying capacity of cities,” into unsustainable levels.
• Can the conclusions from automated city-wide reporting lead to advancements in quality of service, if we’re better at determining defective practices and equipment. In this vein, how cities share data between them will also be quite interesting.

I’d love to hear from you!