Google Summer Of Code Mentor Summit

So all i can say is that the Mentor Summit is pure awesome. A quick blog post using this qwerty keyboard won't cover it. (Damn you qwerty). There'll be a full write up in the future. Here are some impressions.

After all the issues we had with Fedora being an umbrella organisation for other projects, Toshio and i held a session about being such an org. Most notably was having KDE, ASF, and OSGEO present to share their experiences. They have had similar issues on a much larger scale.

I was surprised to see Mario Behling from LXDE here. We had a few good chats about doing deployments of LXDE on the XO 1.0 in Afghanistan and Vietnam, and using Freifunk to deliver internet. Hosting a school server in the "cloud" anyone?

Its nice and warm here, cant wait to go home to the Netherlands.

I'll write more when i have access to a real keyboard.

Doing Denyhosts a bit better

At my $dayjob we have been using denyhosts to help protect our systems from potential bad guys and botnets trying to break into our systems. Apparently the latest craze has been trying to get rooted unix systems incorporated into botnets using some combination of iframes, nginx, and the usual array of normal hacking methods. One attack vector is to bruteforce ssh, which isn't new, but it's been an increasing threat. As proof of the increasing sophistication of attackers, yesterday i noticed that we had an incoming attack using common Dutch firstnames. The goal, obviously is to find a legitimate user with a weak password.

There are a few ways to protect yourself against this vector, some easier than others. Using a mix will help you further as well. The usual logic, such as making the usernames not obvious, enforcing strong password policies and all the usual niceties that manage to start flamewars so easily. You can also make sure to disable remote login as root with password authentication or block root completely unless you're already logged in. One step further is to enforce sudo only access.

One step we also take is using denyhosts to blacklist the IP addresses that are clearly showing behavior matching that of a hacker. Examples of this behavior is logging in with a non existant username, logging in with an obviously bad username such as 'www-data' or 'oracle', logging in as more than one or two users, too many bad password attempts, logging as root and so on. Denyhosts handles the heuristics for us nicely but we wanted to take this a step further. The admins before me put together a nice script where one server could convert the output from denyhosts, pass it along to our puppet configuration which would in turn deploy a blacklist across the entire network. This would in turn do two things, it would protect servers who have not yet been touched by a certain attacker, and denyhosts running on each machine would provide an initial level of blockage.

In this process, we discovered a few problems with how denyhosts works in terms of network performance. One problem we had is that processing the output from denyhosts can be pretty heavy on a system and can involved a huge number of DNS queries. When i tried to tweak script to block more things, it overloaded the system, took out our LDAP server, users couldn't authenticate, and denyhosts started systematically blocking our own network. While certain critical paths were whitelisted, we still ended up with a screwed up network for a few hours, and had one hell of a morning. One local best practice we have now is to run processes like these on a seperate VM dedicated to the task. The further we can offload the processing from critical network space, the lest likely we'll overload the network. The VMs are also running on a hypervisor that's dedicated to running our failover services that are only activated when the primary network goes down.

The second problem we had is that denyhosts will put hostnames as well as IP addresses in /etc/hosts.deny. We realized that network traffic was just a bit laggier than normal once we had the setup running for a bit. Ultimately doing both a firewall level blacklist and then running a nearly identical blacklist in /etc/hosts.deny will reduce traffic more than is necessary. Instead, i cleaned up the /etc/hosts.deny from what denyhosts did, and in the future we'll have to look at another low performance hitting method of having more computers protected faster.

One idea i have for the future is to make the system more event based rather than cron job based. When denyhosts decides it needs to block a hostname or server, it calls something that sends a message across the network to the server that will do all the processing, sanitizing, checking to make sure the host is not on a whitelist, and so on, so that computers can respond right away, but still protect the whole network. Realistically, what we should be doing is just blacklisting the entire intarwebz except for our external webservers, and having our users work from a VPN. But that's a story for another day. All in all it was a fun exercise in how to think practically about the actual load we're deploying to our servers before we do it.

Emotions Rising

Dear Lazyweb,

Could someone please explain why people seem to get more emotional about what software handles their sound than they do about which distro they use or even which DE they use?

Thanks in advance.

FUDCon Live: Call for volunteers

I've typed this paragraph three or four times tonight in different versions, so i'll just pass you all a link that summarizes it better than i ever could, even if i did. The basic idea is that i want to make it easier to get people not present at FUDCon involved. I want to see better live documentation of the upcoming FUDCon in Toronto. As an extra bonus, if this can be pulled off correctly, people can sit in more than one session at a time. It will make multitaskers happy, and make it easier to decide if you're really sitting in the room you want to be sitting in. I'm looking for a few good men or women to volunteer to help out.

I'm looking to document the entire event via IRC like we did at FAD EMEA. Each room will get a separate IRC channel, and if we can get it coordinated, there'll be a zodbot in each one to take notes. This will let people take advantage of zodbot features in person, provide more community at each session, make the entire process more accessible, and who knows what other creative ideas could come out of it.

In short, read this: https://fedoraproject.org/wiki/FUDCon:Toronto_2009#Documentation

Please sign up to volunteer! We need quite a few people to guarantee complete coverage.

(If you have a better idea how to do this, comments and improvements are always welcome :-) )

Moving Day

Yesterday i took the day off from work in order to finish moving to a new flat across town. It's with great sadness that i won't be able to tell people i live in the Nude anymore. If i ever feel like being in the Nude though, it's not far away at all. It's almost literally across the street even.


I can honestly say i came out of the Nude on bike though.


Credits to Björn Lammers who took all the photos.

You can see more on Flickr and Facebook

I Had a Rant Brewing

I had a pretty nasty headache all day, which led to a huge nasty ugly rant brewing. There's been alot going on that's just been bugging me badly about the community and communities which was just ending in some really bad nastiness. There's also been alot of certain kinds of whining in various channels that has been getting on my nerves. All in all, it's a good thing i was busy at work, and i never got the chance to sit and write it down, because the last thing the Fedora Planet needs is more fire and brimstone. A few minutes ago, Tatica announced in #fedora-ambassadors that she's reconsidered and will start posting her photoblog on the Fedora Planet again. Suddenly, the fire and brimstone seemed to melt away.

Lately there have been alot of issues with a few individuals in the greater community making sexist remarks with an insane back and forth about what is right or not in a community. I'll let the journalists and pundits slag that battle out, because on one is paying me enough for my opinions (anymore). It reaches a point where we cross a line in the community, though, and that's when we don't just make of colour comments, but we start excluding community members. This is a point where we're not discussing what's right or not, but we're actively hurting people.

Sometimes you have to have thick skin to trudge through all the muck there is in the extended community. But it's not all that easy for everyone. I'm not going to get into the details about an issue i don't know all the details about. What i do know is this: some persons' responses to one of our community members extracurricular activities convinced her to stop showing us her art. In my point of view, everything everyone does inside and outside of Fedora is their art. Even if you get paid for it. Your life revolves around it some way, and it's a massive multi participant creative project. But it's also the sort project that can include a bewilderingly wide array of people. When i say this, i don't mean every person who likes to stay up until ungodly hours of the morning, puts up black curtains to sleep during the day, and wears socks with sandals, yet living in different time zones. When i say wide, i mean from all walks of lifes, in all professions, with all sorts of hobbies. That's what makes up our community. Stopping people from displaying their art is counter-productive to what we fundamentally do. Think about that the next time, before you are about to push send on that flame.

It gets worse though. I can't for the life of me remember when i was reading this, but it was at least 4 or 5 years ago, when i was reading through the Linux HOWTOs for fun. There was one document on encouraging more women into the community. I don't really want to single out women with these comments, because i think there's a lesson for working with all kinds of people here, but here it goes. The point was that these new people we want joining our community aren't just going to chug ramen noodles every night. They are going to try new foods, and talk about it, with each other, and in public. Working with them means accepting and appreciating that these conversations, be it off topic, are going to occur in the community. The correct approach is to encourage people to tell you what they are doing in life. Who knows, it might even lead you to trying new things once or twice too.

In my mind, though, there isn't a good history here at all. When i first joined the community, i remember there was alot of debate about multiple languages on the Fedora Planet. I know that people were not made to feel welcome just because they blogged in a different language. While there were some good suggestions on how to filter what some people saw as noise, the whole conversation really seemed just as counter-productive. It seemed to me that the planet the represented our community as a whole. This is it, and people were expressing their art. Making rules about how to do this wouldn't help. It seems like everytime we have an identity crisis though, i feel like we're missing that crucial detail.

(I would think that the people involved in this latest incident are not the same as in other debates, but again, i don't know the details.)

That notwithstanding, i still have enough material for a rant of a slightly different nature, but at least the dark mood has calmed down quite a bit since there was some marginal amount of daylight out. At this point i'm just glad to see that we have a community that encourages our own to keep on doing what they do best. I'm also really happy to see Tatica post her photoblog to the planet too. It means alot when the community is free to talk about other things besides just Fedora. It's part of what makes it art.

Fedora at the Dutch HUG

Unfortunately for myself, i've managed to land another speaking engagement, which means in between moving this weekend, i have to sit down and prepare a presentation. This coming Monday, the 12 of October, i'll be speaking at the Dutch HUG to talk about Fedora a bit. I will spend a bit of time discussing how to use Haskell in Fedora, what the status of things are, and how to build your own packages for Fedora. I'll explain how to use cabal2spec, a brief explanation of spec files, and a bit about the packaging process in Fedora itself. I will also cover what you'll need to do to run Haskell programs on enterprise distributions that are also RPM based. You won't need much Fedora experience to understand the presentation, though i'm assuming that you are already a Haskell programmer.

The presentation will be given at the monthly Dutch HUG meeting, which will be held at the Utrecht University Library in the Uithof. The presentations will start around 19.00 (7.00 PM for the Americans).